Avoid O(N^2) in VALUES with ordinals grouping

[dnhatn]

Using the VALUES aggregator with ordinals grouping led to accidental quadratic complexity. Queries like FROM .. | STATS ... VALUES(field) ... BY keyword-field are affected by this performance issue. This change caches a sorted structure - previously used to fix a similar O(N^2) problem when emitting the output block - during the merging phase of the OrdinalGroupingOperator.

[elasticsearchmachine]

Hi @dnhatn, I've created a changelog YAML for you.

[dnhatn]

I updated the benchmark to simulate the ordinal grouping operator. With the main branch, I could not complete the benchmark with 1,000,000 groups, but with the fix, it took 1625ms. The benchmark results are below.

Before:
Benchmark                      (dataType)  (groups)  (numOrdinalMerges)  Mode  Cnt      Score   Error  Units
ValuesAggregatorBenchmark.run    BytesRef         1                   0  avgt    2      3.680          ms/op
ValuesAggregatorBenchmark.run    BytesRef         1                   1  avgt    2      3.632          ms/op
ValuesAggregatorBenchmark.run    BytesRef      1000                   0  avgt    2      2.515          ms/op
ValuesAggregatorBenchmark.run    BytesRef      1000                   1  avgt    2      9.397          ms/op
ValuesAggregatorBenchmark.run    BytesRef    200000                   0  avgt    2    148.966          ms/op
ValuesAggregatorBenchmark.run    BytesRef    200000                   1  avgt    2  90055.908          ms/op
ValuesAggregatorBenchmark.run         int         1                   0  avgt    2      0.494          ms/op
ValuesAggregatorBenchmark.run         int         1                   1  avgt    2      0.488          ms/op
ValuesAggregatorBenchmark.run         int      1000                   0  avgt    2      2.788          ms/op
ValuesAggregatorBenchmark.run         int      1000                   1  avgt    2      8.232          ms/op
ValuesAggregatorBenchmark.run         int    200000                   0  avgt    2    198.020          ms/op
ValuesAggregatorBenchmark.run         int    200000                   1  avgt    2  70918.020          ms/op
ValuesAggregatorBenchmark.run        long         1                   0  avgt    2      0.862          ms/op
ValuesAggregatorBenchmark.run        long         1                   1  avgt    2      0.873          ms/op
ValuesAggregatorBenchmark.run        long      1000                   0  avgt    2      4.212          ms/op
ValuesAggregatorBenchmark.run        long      1000                   1  avgt    2     10.450          ms/op
ValuesAggregatorBenchmark.run        long    200000                   0  avgt    2    257.926          ms/op
ValuesAggregatorBenchmark.run        long    200000                   1  avgt    2  75686.076          ms/op



After:
Benchmark                      (dataType)  (groups)  (numOrdinalMerges)  Mode  Cnt     Score   Error  Units
ValuesAggregatorBenchmark.run    BytesRef         1                   0  avgt    2     3.909          ms/op
ValuesAggregatorBenchmark.run    BytesRef         1                   1  avgt    2     3.951          ms/op
ValuesAggregatorBenchmark.run    BytesRef      1000                   0  avgt    2     2.635          ms/op
ValuesAggregatorBenchmark.run    BytesRef      1000                   1  avgt    2     2.703          ms/op
ValuesAggregatorBenchmark.run    BytesRef   1000000                   0  avgt    2  1519.385          ms/op
ValuesAggregatorBenchmark.run    BytesRef   1000000                   1  avgt    2  1623.915          ms/op
ValuesAggregatorBenchmark.run         int         1                   0  avgt    2     0.601          ms/op
ValuesAggregatorBenchmark.run         int         1                   1  avgt    2     0.613          ms/op
ValuesAggregatorBenchmark.run         int      1000                   0  avgt    2     2.504          ms/op
ValuesAggregatorBenchmark.run         int      1000                   1  avgt    2     2.591          ms/op
ValuesAggregatorBenchmark.run         int   1000000                   0  avgt    2  1396.017          ms/op
ValuesAggregatorBenchmark.run         int   1000000                   1  avgt    2  1441.373          ms/op
ValuesAggregatorBenchmark.run        long         1                   0  avgt    2     0.598          ms/op
ValuesAggregatorBenchmark.run        long         1                   1  avgt    2     0.597          ms/op
ValuesAggregatorBenchmark.run        long      1000                   0  avgt    2     2.397          ms/op
ValuesAggregatorBenchmark.run        long      1000                   1  avgt    2     2.510          ms/op
ValuesAggregatorBenchmark.run        long   1000000                   0  avgt    2  1538.923          ms/op
ValuesAggregatorBenchmark.run        long   1000000                   1  avgt    2  1625.971          ms/op

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)