Graph benchmarks

[andrii0lomakin]

Graph benchmarks results clearly show that we need to run them periodically so we propose following format of running of benchmarks.

Benchmarks schedule:
-For full in memory tests (100% of data in cache) - run them every night on bare metal server. All other tasks should be switched off.
-For disc backed cache (only 20 % of data in load cache, but in future even less , I think 5% is good number) run every weekend.

What those tests do ? They compare perfomance of latest released version and version which is built from develop. If results are lower more than on X% (see below) if we compare them with released version - tests should fail.

What results are gathered:

1. Median (not average) of operations per second - tests fail if result is below than 10%.
2. 99 percentile of operations per second - tests fails if result is below 15%.
3. 100 percentile of operations per second - tests fail if result is below 30% , because we measure system jitter there too..
4. Time is needed to execute all test - tests fails if result is below 15%.

What tests we are going to run:
-Load of graph data from at least 2 real data sets.
-Batch load of graph data
-Short path performance
-In future we will add YCSB tests too.

[smolinari]

Yes! The benchmark testing will increase ODB's quality immensely. The other gain from this is you'll be able to quickly see if any supposed performance boosting changes actually work as expected or not. It will also raise everyone's confidence too. Good decision!

Scott

[andrii0lomakin]

@smolinari creating of tests is not enough, measuring everything right is also important. :-)

[smolinari]

@Laa - Absolutely Andrey! The benchmarks must be meaningful, in order to be effective and helpful.

Scott

[andrii0lomakin]

duplication