Use BitFaster.Caching LRU cache for QueryCache; Refactor common method implementations of ICache

[SergeiPavlov]

@botinko
made following research of LRU Cache performance:

I checked the performance of LRU caches. What has been tested:

• Original LRU cache from DataObjects
• FastLru cache from https://github.com/bitfaster/BitFaster.Caching with ConcurrentDictionary
• FastLru cache from https://github.com/bitfaster/BitFaster.Caching with https://github.com/VSadov/NonBlocking dictionary

There were also 3 types of load

• a rather large cache (20,000), a large number of keys (1,000,000), distribution of key frequencies according to the Zipf law. This type of load is likely to be close to reality after the node id is no longer part of the key and the cache size is increased.

• small cache (256), a large number of keys (1,000,000), distribution of key frequencies according to the Zipf law. This type of load will be if the node id is no longer part of the key, but the cache size is left at the default.

• small cache (256), a large number of keys (1,000,000), distribution of keys is uniform. We have this type of load now (due to the fact that the node id is part of the key and the number of nodes is large)

I also compared the performance in net core 3.1 and 5.

So, there are (332) 18 tests. Each test is executed in 1,2,3...16 threads to have a better understanding of how it works concurrently.
Tested on

AMD Ryzen 4900HS 3.0GHz (~4.0Ghz boost), 1 CPU, 16 logical and 8 physical cores
Microsoft Windows 10 Pro 10.0.19042

Results:

Runtime=.NET Core 3.1.9 
lruSize 20000 itemsNumber 1000000 high hit rate by Zipf law
======== GetOrAdd FastConcurrentLru ConcurrentDictionary object->object 1M Ops/sec:
28.10; 26.17; 25.01; 21.00; 18.24; 17.12; 16.26; 15.65; 15.38; 15.21; 14.67; 14.97; 14.83; 14.57; 15.48; 14.45;
======== GetOrAdd FastConcurrentLru NonBlockingDictionary object->object 1M Ops/sec:
26.46; 26.40; 26.01; 23.79; 22.79; 21.51; 20.47; 19.77; 19.45; 19.25; 18.90; 18.50; 17.83; 17.81; 17.71; 17.14;
======== GetOrAdd DataObjectsLRU Pair object->object 1M Ops/sec:
14.03; 6.802; 4.362; 3.928; 3.490; 3.242; 3.020; 2.824; 2.768; 2.694; 2.362; 2.358; 2.298; 2.298; 2.242; 2.270;
lruSize 256 itemsNumber 1000000 high hit rate by Zipf
======== GetOrAdd FastConcurrentLru ConcurrentDictionary object->object 1M Ops/sec:
27.53; 25.40; 19.87; 17.39; 15.58; 14.95; 14.63; 14.22; 13.32; 13.09; 12.56; 12.39; 12.24; 12.04; 11.92; 11.88;
======== GetOrAdd FastConcurrentLru NonBlockingDictionary object->object 1M Ops/sec:
26.61; 23.55; 22.36; 19.35; 17.46; 16.37; 15.45; 12.76; 15.66; 14.77; 14.60; 13.88; 12.98; 12.52; 11.93; 12.63;
======== GetOrAdd DataObjectsLRU Pair object->object 1M Ops/sec:
14.03; 7.314; 4.574; 3.900; 3.504; 3.364; 3.210; 3.006; 3.012; 2.962; 2.912; 2.876; 2.852; 2.832; 2.812; 2.804;
lruSize 256 itemsNumber 1000000 low hit rate - random
======== GetOrAdd FastConcurrentLru ConcurrentDictionary object->object 1M Ops/sec:
27.61; 26.10; 20.52; 17.73; 16.00; 13.76; 12.47; 11.56; 11.30; 10.53; 9.020; 10.00; 9.510; 8.944; 8.850; 8.580;
======== GetOrAdd FastConcurrentLru NonBlockingDictionary object->object 1M Ops/sec:
26.95; 22.20; 18.46; 17.40; 16.85; 15.01; 14.07; 13.36; 12.93; 12.24; 12.03; 12.51; 14.36; 13.33; 12.72; 11.33;
======== GetOrAdd DataObjectsLRU Pair object->object 1M Ops/sec:
14.14; 6.582; 3.802; 3.298; 2.912; 2.650; 2.448; 2.282; 2.144; 2.022; 1.958; 1.898; 1.852; 1.818; 1.780; 1.766;

Runtime=.NET Core 5.0
lruSize 20000 itemsNumber 1000000 high hit rate by Zipf law
======== GetOrAdd FastConcurrentLru ConcurrentDictionary object->object 1M Ops/sec:
39.26; 37.89; 32.75; 25.05; 21.94; 20.29; 17.63; 16.84; 16.59; 15.64; 15.93; 14.95; 15.21; 15.17; 15.57; 15.36;
======== GetOrAdd FastConcurrentLru NonBlockingDictionary object->object 1M Ops/sec:
30.77; 28.92; 28.36; 25.40; 23.92; 23.13; 22.26; 21.79; 19.49; 20.96; 19.32; 18.87; 19.03; 18.17; 17.78; 17.31;
======== GetOrAdd DataObjectsLRU Pair object->object 1M Ops/sec:
17.34; 7.296; 5.158; 4.224; 3.478; 3.466; 3.292; 3.246; 3.040; 2.808; 2.696; 2.822; 2.730; 2.728; 2.766; 2.752;
lruSize 256 itemsNumber 1000000 high hit rate by Zipf
======== GetOrAdd FastConcurrentLru ConcurrentDictionary object->object 1M Ops/sec:
38.73; 32.86; 23.78; 16.72; 14.52; 12.66; 13.55; 15.68; 14.59; 14.16; 13.28; 12.19; 11.82; 12.11; 11.73; 11.41;
======== GetOrAdd FastConcurrentLru NonBlockingDictionary object->object 1M Ops/sec:
29.72; 22.32; 21.66; 22.13; 17.68; 17.25; 16.90; 14.05; 17.03; 15.03; 15.07; 14.49; 13.86; 13.12; 13.28; 14.66;
======== GetOrAdd DataObjectsLRU Pair object->object 1M Ops/sec:
17.53; 6.960; 3.878; 3.584; 3.394; 3.288; 3.208; 3.110; 3.082; 3.092; 3.060; 3.028; 2.886; 2.976; 3.056; 3.082;
lruSize 256 itemsNumber 1000000 low hit rate - random
======== GetOrAdd FastConcurrentLru ConcurrentDictionary object->object 1M Ops/sec:
38.69; 28.10; 22.08; 18.07; 14.25; 15.93; 14.68; 12.48; 11.90; 11.29; 10.44; 11.34; 11.32; 11.13; 10.79; 10.51;
======== GetOrAdd FastConcurrentLru NonBlockingDictionary object->object 1M Ops/sec:
31.30; 26.92; 20.32; 15.70; 14.90; 12.24; 11.13; 14.16; 15.70; 15.54; 15.28; 16.11; 14.12; 13.78; 13.61; 13.27;
======== GetOrAdd DataObjectsLRU Pair object->object 1M Ops/sec:
17.28; 7.182; 3.688; 3.054; 2.720; 2.606; 2.500; 2.406; 2.334; 2.366; 2.330; 2.258; 2.262; 2.272; 2.238; 2.204;

Conclusions:

1. .net 5 improves performance dramatically
2. DataObjectsLRU slows down even on two threads. Also, I checked the Performance profile, and see very high lock contention with periodic lock-convoys
3. About NonBlockingDictionary. It's clearly faster with a large number of threads (4 and more) and parallels well.
   But in 1-3 thread ConcurrentDictionary is better.

Thus I recommend BitFaster.Caching with default ConcurrentDictionary as a DataObjectsLRU replacement as QueryCache.
This replacement will be useful even without removing node id from the query key. This will solve the problem of convoy lock in Query Cache. See the difference:

lruSize 256 itemsNumber 1000000 low hit rate - random
======== GetOrAdd FastConcurrentLru ConcurrentDictionary object->object 1M Ops/sec:
38.69; 28.10; 22.08; 18.07; 14.25; 15.93; 14.68; 12.48; 11.90; 11.29; 10.44; 11.34; 11.32; 11.13; 10.79; 10.51;
======== GetOrAdd DataObjectsLRU Pair object->object 1M Ops/sec:
17.28; 7.182; 3.688; 3.054; 2.720; 2.606; 2.500; 2.406; 2.334; 2.366; 2.330; 2.258; 2.262; 2.272; 2.238; 2.204;

[alex-kulakov]

Good enough