extended test mode for benchmarks

[samir-nasibli]

• extended test mode by adding benchmark mode

[densmirn]

Seems we need to add opportunity to run the performance tests under multiple threads by setting environment variables LOAD_PREV_RESULTS and NUMBA_NUM_THREADS before runs. Moreover we need to install extra Python modules openpyxl and xlrd to dump results to excel.

[Vyacheslav-Smirnov]

Seems we need to add opportunity to run the performance tests under multiple threads by setting environment variables LOAD_PREV_RESULTS and NUMBA_NUM_THREADS before runs. Moreover we need to install extra Python modules openpyxl and xlrd to dump results to excel.

@densmirn ,
We expect to run benchmarks internally for now so these variables can be set there.
By the way - what is the purpose of LOAD_PREV_RESULTS and what is should point to?
openpyxl is already added to test env for benchmarks. Do we need to add xlrd also?

[dmitrii-zagornyi]

@densmirn @Vyacheslav-Smirnov
I strongly recommend to use csv files instead of excel files, adding dependencies for openpyxl, xlrd.

[Vyacheslav-Smirnov]

@densmirn , Can we make benchmarks to output results in csv instead of excel?
This can move us from additional dependencies

[samir-nasibli]

@densmirn I have the same question: what is the purpose of LOAD_PREV_RESULTS and what is should point to? It seems that currently excel files are overwriting without aggregation of recent results

[samir-nasibli]

@densmirn @Vyacheslav-Smirnov @dmitrii-zagornyi aggregating results in csv is much preferable in the task of integration benchmarks into CI, isn't it? also this would avoid additional dependencies

[densmirn]

Seems we need to add opportunity to run the performance tests under multiple threads by setting environment variables LOAD_PREV_RESULTS and NUMBA_NUM_THREADS before runs. Moreover we need to install extra Python modules openpyxl and xlrd to dump results to excel.

@densmirn ,
We expect to run benchmarks internally for now so these variables can be set there.
By the way - what is the purpose of LOAD_PREV_RESULTS and what is should point to?
openpyxl is already added to test env for benchmarks. Do we need to add xlrd also?

LOAD_PREV_RESULTS = 1 makes to load previous result. It can be useful in case when we run tests on different number of threads where results of each run are dumped to the file. E.g. after below runs we get table with all the results.

export LOAD_PREV_RESULTS=1
export NUMBA_THREADING_LAYER=omp (or tbb)
export NUMBA_NUM_THREADS=1
python -m sdc.runtests sdc.tests.tests_perf
export NUMBA_NUM_THREADS=2
python -m sdc.runtests sdc.tests.tests_perf 
export NUMBA_NUM_THREADS=4
python -m sdc.runtests sdc.tests.tests_perf

Maybe it makes sense to remove the file before the next series of run.

[densmirn]

@densmirn @Vyacheslav-Smirnov @dmitrii-zagornyi aggregating results in csv is much preferable in the task of integration benchmarks into CI, isn't it? also this would avoid additional dependencies

In case if we want to get CSV results we need to wait #347.

[densmirn]

I vote for waiting for #347 to apply these changes over this PR.

[Vyacheslav-Smirnov]

@densmirn , @PokhodenkoSA ,
Then I think result's dump to html should be implemented in #347
And in current PR we can revert changes with html and merge it.

[densmirn]

Please resolve merge conflicts.