Search benchmarks are single-threaded and suffer from coordinated omission

[jasontedor]

The search benchmarks executes queries in a single-threaded fashion that suffers from coordinated omission. This hides a variety of problems that could impact search latency:

• lower allocation rates than real-world scenarios leading to less interruption from garbage collection pauses
• lower context switch rates than real-world scenarios
• no lock contention within the Elasticsearch codebase
• coordinated omission hides true search latency under load

[jasontedor]

Relates elastic/elasticsearch#16802

[danielmitterdorfer]

I implemented a proof-of-concept script outside of Rally based on the (experimental) elasticsearch-py-async library. It can schedule requests at a fixed rate and is not prone to coordinated omission anymore.

Integrating this into Rally has a few implications though and I have to chat with Honza first about that. One issue is the status of elasticsearch-py-async itself (experimental) and that it relies on a different version of the elasticsearch-py client library. As we use elasticsearch-py currently for all sorts of tasks, we would also need to change the internal connection to the metrics store, bulk indexing and a few other places.