Share Query Cache feature

[SergeiPavlov]

Setting DomainConfiguration.ShareQueryCacheOverNodes = true enables to share compiled queries between nodes

[alex-kulakov]

No tests for this? This is really important. In case if cached query reached wrong node it would be a huge problem, we need to be 100% sure that this is not happening.

[SergeiPavlov]

No tests for this? This is really important. In case if cached query reached wrong node it would be a huge problem, we need to be 100% sure that this is not happening.

I have little knowledge how to write multi-node DO test. But we are using this feature many months in production.

[alex-kulakov]

I have little knowledge how to write multi-node DO test.

You can ask for help. I'll guide and help you, not a big deal.

But we are using this feature many months in production.

No offence but it'd rather choose proper test than "it works in our environment". I don't know your environment, I don't know how you use it. As I said I'll help you with tests.

[SergeiPavlov]

No offence but it'd rather choose proper test than "it works in our environment". I don't know your environment, I don't know how you use it. As I said I'll help you with tests.

Thank you. I hope I will have time to make tests. But now it is not high priority task

[alex-kulakov]

I don't like how upper-level types like TypeInfo and StorageNode got into SQL level. SQLDOM doesn't have to know about domains, sessions, typeinfos, storage nodes, etc. It is basically stand-alone, having just an SqlDriver, it should be allowed to extract schemas and catalogs, create and compile any SQL query supported by SQLDOM.

There is got to be a way to create an equivalent of the query created by DO by using SQLDOM and certain SqlDriver only, in this case it is impossible.

[alex-kulakov]

For example this is completely inappropriate.

[alex-kulakov]

This is also not ok, you have a configuration that wraps additional info for compilation. But no upper-level types should be used there.

[alex-kulakov]

I'm also thinking of having caches within nodes in cases when DomainConfiguration.ShareQueryCacheOverNodes = false but I'm weighting the outcome of this. Now number of items in query cache is multiplied by number of nodes, it should affect performance on thousands of nodes and thousands of queries (btw, how big is your cache size?).

Logically the idea of separated caches fits the concept attachable/detachable node - once node is disposed the resources it occupied should be released too.

[SergeiPavlov]

There is got to be a way to create an equivalent of the query created by DO by using SQLDOM and certain SqlDriver only, in this case it is impossible.

I agree this is some kind of hack. But it just works, that is why I decided to use it.

[alex-kulakov]

Can you list the parts of query you replace with placeholders or something similar to make query to be cacheable?

[SergeiPavlov]

Can you list the parts of query you replace with placeholders or something similar to make query to be cacheable?

• schema
• TypeId

[alex-kulakov]

Long time ago I did TypeId as parameter of query, but Alexander Ustinov said that there is something wrong with my implementation if applied to your case so i didn't merge it. Now I have a second thought that maybe it will somehow fit to this concept, ShareQueryCacheOverNodes requires basically two features combined - TypeIdAsParameter and ShareSchemaOverNodes, doesn't it?

[SergeiPavlov]

requires basically two features combined - TypeIdAsParameter and ShareSchemaOverNodes, doesn't it?

Looks like, yes
If you are talkimg about your year-old implementation of sharedCache-like feature, I considered to use, but was not suitable for our case (does not remember why already)

[alex-kulakov]

I have improved TypeId As Parameter PR (#147), It is based on QueryParameterBinding now which is much better.

[alex-kulakov]

Additionally, I was inspired by your idea and changed schema actualization, here's the PR (#289). I think we could combine these two. PRs (#147 and #289) and just introduce condition like

var domainConfig = domain.Configuration;
this.queryKey = domainConfig.ShareSchemaOverNodes && domainConfig.PreferTypeIdsAsQueryParameters
  ? queryKey
  : new Pair<object, string>(queryKey, session.StorageNodeId);

in this PR.

[SergeiPavlov]

just introduce condition like

done

[alex-kulakov]

Take a look at the implementation in master-shared-cache. It has tests for the feature as well. I need it to be tested on various database configs and then it will be ready for pull request

[alex-kulakov]

The PR is ready - #290