Add a track for date_histogram

[nik9000]

Adds a track to nyc taxis to test date_histogram performance.

[nik9000]

This creates test data for elastic/elasticsearch#55559

[nik9000]

I pushed an update fixing some names, fixing some target intervals, and adding auto_date_histogram runs to test elastic/elasticsearch#56384.

[ebadyano]

^^Sorry I didn't see the update, I will rerun with new target intervals.
@nik9000 Thank you for the pull request!
I ran it on low-mem env with 1G heap. Results are bellow.

From latency vs service time it looks like target-interval/throughout for date_histogram_all_calendar_interval_with_tz is too high. Might want to set it lower. Also is it expected that these queries take so long? Service time for a requests with tz is 21457.2ms and 50911.7ms.. And just to confirm this is just to add the challenge, but without intention to run it during nightlies?

|---------------------------------------------------------------:|---------------------------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                                              |     243.836 |    min |
|             Min cumulative indexing time across primary shards |                                              |     243.836 |    min |
|          Median cumulative indexing time across primary shards |                                              |     243.836 |    min |
|             Max cumulative indexing time across primary shards |                                              |     243.836 |    min |
|            Cumulative indexing throttle time of primary shards |                                              |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                                              |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                                              |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                                              |           0 |    min |
|                        Cumulative merge time of primary shards |                                              |     81.6045 |    min |
|                       Cumulative merge count of primary shards |                                              |         242 |        |
|                Min cumulative merge time across primary shards |                                              |     81.6045 |    min |
|             Median cumulative merge time across primary shards |                                              |     81.6045 |    min |
|                Max cumulative merge time across primary shards |                                              |     81.6045 |    min |
|               Cumulative merge throttle time of primary shards |                                              |     20.9533 |    min |
|       Min cumulative merge throttle time across primary shards |                                              |     20.9533 |    min |
|    Median cumulative merge throttle time across primary shards |                                              |     20.9533 |    min |
|       Max cumulative merge throttle time across primary shards |                                              |     20.9533 |    min |
|                      Cumulative refresh time of primary shards |                                              |     1.46643 |    min |
|                     Cumulative refresh count of primary shards |                                              |         109 |        |
|              Min cumulative refresh time across primary shards |                                              |     1.46643 |    min |
|           Median cumulative refresh time across primary shards |                                              |     1.46643 |    min |
|              Max cumulative refresh time across primary shards |                                              |     1.46643 |    min |
|                        Cumulative flush time of primary shards |                                              |      2.4369 |    min |
|                       Cumulative flush count of primary shards |                                              |          31 |        |
|                Min cumulative flush time across primary shards |                                              |      2.4369 |    min |
|             Median cumulative flush time across primary shards |                                              |      2.4369 |    min |
|                Max cumulative flush time across primary shards |                                              |      2.4369 |    min |
|                                             Total Young Gen GC |                                              |     165.957 |      s |
|                                               Total Old Gen GC |                                              |           0 |      s |
|                                                     Store size |                                              |     24.7192 |     GB |
|                                                  Translog size |                                              | 5.12227e-08 |     GB |
|                                         Heap used for segments |                                              |     0.18935 |     MB |
|                                       Heap used for doc values |                                              |   0.0353203 |     MB |
|                                            Heap used for terms |                                              |   0.0381622 |     MB |
|                                            Heap used for norms |                                              |           0 |     MB |
|                                           Heap used for points |                                              |           0 |     MB |
|                                    Heap used for stored fields |                                              |    0.115868 |     MB |
|                                                  Segment count |                                              |          29 |        |
|                                                 Min Throughput |                                        index |     73472.1 | docs/s |
|                                              Median Throughput |                                        index |     76889.9 | docs/s |
|                                                 Max Throughput |                                        index |     85246.6 | docs/s |
|                                        50th percentile latency |                                        index |     945.991 |     ms |
|                                        90th percentile latency |                                        index |     1594.91 |     ms |
|                                        99th percentile latency |                                        index |     2826.31 |     ms |
|                                      99.9th percentile latency |                                        index |     7500.94 |     ms |
|                                     99.99th percentile latency |                                        index |     8858.36 |     ms |
|                                       100th percentile latency |                                        index |     9405.79 |     ms |
|                                   50th percentile service time |                                        index |      946.01 |     ms |
|                                   90th percentile service time |                                        index |     1594.92 |     ms |
|                                   99th percentile service time |                                        index |     2826.31 |     ms |
|                                 99.9th percentile service time |                                        index |     7500.94 |     ms |
|                                99.99th percentile service time |                                        index |     8858.36 |     ms |
|                                  100th percentile service time |                                        index |     9405.79 |     ms |
|                                50th percentile processing time |                                        index |     951.832 |     ms |
|                                90th percentile processing time |                                        index |      1601.5 |     ms |
|                                99th percentile processing time |                                        index |     2832.13 |     ms |
|                              99.9th percentile processing time |                                        index |     7504.16 |     ms |
|                             99.99th percentile processing time |                                        index |     8862.41 |     ms |
|                               100th percentile processing time |                                        index |     9409.59 |     ms |
|                                                     error rate |                                        index |           0 |      % |
|                                                 Min Throughput |         date_histogram_all_calendar_interval |         0.1 |  ops/s |
|                                              Median Throughput |         date_histogram_all_calendar_interval |         0.1 |  ops/s |
|                                                 Max Throughput |         date_histogram_all_calendar_interval |         0.1 |  ops/s |
|                                        50th percentile latency |         date_histogram_all_calendar_interval |     6009.08 |     ms |
|                                        90th percentile latency |         date_histogram_all_calendar_interval |     6087.82 |     ms |
|                                       100th percentile latency |         date_histogram_all_calendar_interval |     6181.38 |     ms |
|                                   50th percentile service time |         date_histogram_all_calendar_interval |     6003.86 |     ms |
|                                   90th percentile service time |         date_histogram_all_calendar_interval |     6082.56 |     ms |
|                                  100th percentile service time |         date_histogram_all_calendar_interval |     6175.74 |     ms |
|                                50th percentile processing time |         date_histogram_all_calendar_interval |     6005.18 |     ms |
|                                90th percentile processing time |         date_histogram_all_calendar_interval |     6085.43 |     ms |
|                               100th percentile processing time |         date_histogram_all_calendar_interval |     6177.74 |     ms |
|                                                     error rate |         date_histogram_all_calendar_interval |           0 |      % |
|                                                 Min Throughput | date_histogram_all_calendar_interval_with_tz |        0.05 |  ops/s |
|                                              Median Throughput | date_histogram_all_calendar_interval_with_tz |        0.05 |  ops/s |
|                                                 Max Throughput | date_histogram_all_calendar_interval_with_tz |        0.05 |  ops/s |
|                                        50th percentile latency | date_histogram_all_calendar_interval_with_tz |      345082 |     ms |
|                                        90th percentile latency | date_histogram_all_calendar_interval_with_tz |      532417 |     ms |
|                                       100th percentile latency | date_histogram_all_calendar_interval_with_tz |      579111 |     ms |
|                                   50th percentile service time | date_histogram_all_calendar_interval_with_tz |     21457.2 |     ms |
|                                   90th percentile service time | date_histogram_all_calendar_interval_with_tz |       21607 |     ms |
|                                  100th percentile service time | date_histogram_all_calendar_interval_with_tz |       21656 |     ms |
|                                50th percentile processing time | date_histogram_all_calendar_interval_with_tz |     21460.1 |     ms |
|                                90th percentile processing time | date_histogram_all_calendar_interval_with_tz |     21609.9 |     ms |
|                               100th percentile processing time | date_histogram_all_calendar_interval_with_tz |     21658.8 |     ms |
|                                                     error rate | date_histogram_all_calendar_interval_with_tz |           0 |      % |
|                                                 Min Throughput |            date_histogram_all_fixed_interval |         0.1 |  ops/s |
|                                              Median Throughput |            date_histogram_all_fixed_interval |         0.1 |  ops/s |
|                                                 Max Throughput |            date_histogram_all_fixed_interval |         0.1 |  ops/s |
|                                        50th percentile latency |            date_histogram_all_fixed_interval |     5627.85 |     ms |
|                                        90th percentile latency |            date_histogram_all_fixed_interval |     5699.22 |     ms |
|                                       100th percentile latency |            date_histogram_all_fixed_interval |     5784.11 |     ms |
|                                   50th percentile service time |            date_histogram_all_fixed_interval |     5622.39 |     ms |
|                                   90th percentile service time |            date_histogram_all_fixed_interval |        5693 |     ms |
|                                  100th percentile service time |            date_histogram_all_fixed_interval |        5778 |     ms |
|                                50th percentile processing time |            date_histogram_all_fixed_interval |     5623.93 |     ms |
|                                90th percentile processing time |            date_histogram_all_fixed_interval |     5694.03 |     ms |
|                               100th percentile processing time |            date_histogram_all_fixed_interval |     5779.71 |     ms |
|                                                     error rate |            date_histogram_all_fixed_interval |           0 |      % |
|                                                 Min Throughput |    date_histogram_all_fixed_interval_with_tz |        0.01 |  ops/s |
|                                              Median Throughput |    date_histogram_all_fixed_interval_with_tz |        0.01 |  ops/s |
|                                                 Max Throughput |    date_histogram_all_fixed_interval_with_tz |        0.01 |  ops/s |
|                                        50th percentile latency |    date_histogram_all_fixed_interval_with_tz |     50930.9 |     ms |
|                                        90th percentile latency |    date_histogram_all_fixed_interval_with_tz |     51055.4 |     ms |
|                                       100th percentile latency |    date_histogram_all_fixed_interval_with_tz |       51508 |     ms |
|                                   50th percentile service time |    date_histogram_all_fixed_interval_with_tz |     50910.5 |     ms |
|                                   90th percentile service time |    date_histogram_all_fixed_interval_with_tz |     51034.7 |     ms |
|                                  100th percentile service time |    date_histogram_all_fixed_interval_with_tz |     51487.6 |     ms |
|                                50th percentile processing time |    date_histogram_all_fixed_interval_with_tz |     50911.7 |     ms |
|                                90th percentile processing time |    date_histogram_all_fixed_interval_with_tz |     51035.6 |     ms |
|                               100th percentile processing time |    date_histogram_all_fixed_interval_with_tz |     51488.5 |     ms |
|                                                     error rate |    date_histogram_all_fixed_interval_with_tz |           0 |      % |

[ebadyano]

^^Sorry I didn't see the changes to target intervals, I will rerun with new target intervals.

[nik9000]

And just to confirm this is just to add the challenge, but without intention to run it during nightlies?

Correct! I hear tell that we don't really have room in the nightlies. I'll be doing a fair bit of work on agg performance in the next few weeks and mostly plan to run them myself because of the lack of room.

Also is it expected that these queries take so long?

That depends on the version of Elasticsearch you are using! If you have master as of this weekend then the service times should be much more similar between tz and not-tz. ~30% or ~40% different.

[ebadyano]

I re-run with the latest elasticsearch and the latest update. Latency and service time look reasonable. Thank you!

rally --skip-update --target-host="192.168.14.13:39200" --track-path=~/rally-track2/rally-tracks/nyc_taxis   --challenge="date-histogram" --car="defaults,g1gc" --client-options="timeout:240"  --runtime-jdk="bundled" --on-error="abort" --revision=latest

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Preparing for race ...
[INFO] Racing on track [nyc_taxis], challenge [date-histogram] and car ['defaults', 'g1gc'] with version [8.0.0-SNAPSHOT].

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
RunnRunning index                                                                  [100% done]
Running refresh-after-index                                                    [100% done]
Running date_histogram_calendar_interval                                       [100% done]
Running date_histogram_calendar_interval_with_tz                               [100% done]
Running date_histogram_fixed_interval                                          [100% done]
Running date_histogram_fixed_interval_with_tz                                  [100% done]
Running auto_date_histogram                                                    [100% done]
Running auto_date_histogram_with_tz                                            [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |                                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-----------------------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                                          |     235.162 |    min |
|             Min cumulative indexing time across primary shards |                                          |     235.162 |    min |
|          Median cumulative indexing time across primary shards |                                          |     235.162 |    min |
|             Max cumulative indexing time across primary shards |                                          |     235.162 |    min |
|            Cumulative indexing throttle time of primary shards |                                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                                          |           0 |    min |
|                        Cumulative merge time of primary shards |                                          |     82.6051 |    min |
|                       Cumulative merge count of primary shards |                                          |         245 |        |
|                Min cumulative merge time across primary shards |                                          |     82.6051 |    min |
|             Median cumulative merge time across primary shards |                                          |     82.6051 |    min |
|                Max cumulative merge time across primary shards |                                          |     82.6051 |    min |
|               Cumulative merge throttle time of primary shards |                                          |     24.4588 |    min |
|       Min cumulative merge throttle time across primary shards |                                          |     24.4588 |    min |
|    Median cumulative merge throttle time across primary shards |                                          |     24.4588 |    min |
|       Max cumulative merge throttle time across primary shards |                                          |     24.4588 |    min |
|                      Cumulative refresh time of primary shards |                                          |     1.62738 |    min |
|                     Cumulative refresh count of primary shards |                                          |         114 |        |
|              Min cumulative refresh time across primary shards |                                          |     1.62738 |    min |
|           Median cumulative refresh time across primary shards |                                          |     1.62738 |    min |
|              Max cumulative refresh time across primary shards |                                          |     1.62738 |    min |
|                        Cumulative flush time of primary shards |                                          |      2.9649 |    min |
|                       Cumulative flush count of primary shards |                                          |          30 |        |
|                Min cumulative flush time across primary shards |                                          |      2.9649 |    min |
|             Median cumulative flush time across primary shards |                                          |      2.9649 |    min |
|                Max cumulative flush time across primary shards |                                          |      2.9649 |    min |
|                                             Total Young Gen GC |                                          |     132.137 |      s |
|                                               Total Old Gen GC |                                          |           0 |      s |
|                                                     Store size |                                          |      24.773 |     GB |
|                                                  Translog size |                                          | 5.12227e-08 |     GB |
|                                         Heap used for segments |                                          |     0.19788 |     MB |
|                                       Heap used for doc values |                                          |   0.0366631 |     MB |
|                                            Heap used for terms |                                          |   0.0434418 |     MB |
|                                            Heap used for norms |                                          |           0 |     MB |
|                                           Heap used for points |                                          |           0 |     MB |
|                                    Heap used for stored fields |                                          |    0.117775 |     MB |
|                                                  Segment count |                                          |          33 |        |
|                                                 Min Throughput |                                    index |     67866.8 | docs/s |
|                                              Median Throughput |                                    index |     71041.3 | docs/s |
|                                                 Max Throughput |                                    index |     86547.1 | docs/s |
|                                        50th percentile latency |                                    index |     968.275 |     ms |
|                                        90th percentile latency |                                    index |     1744.52 |     ms |
|                                        99th percentile latency |                                    index |     4514.29 |     ms |
|                                      99.9th percentile latency |                                    index |     15326.7 |     ms |
|                                     99.99th percentile latency |                                    index |     17877.6 |     ms |
|                                       100th percentile latency |                                    index |     17883.7 |     ms |
|                                   50th percentile service time |                                    index |     968.355 |     ms |
|                                   90th percentile service time |                                    index |     1744.53 |     ms |
|                                   99th percentile service time |                                    index |     4520.08 |     ms |
|                                 99.9th percentile service time |                                    index |     15326.7 |     ms |
|                                99.99th percentile service time |                                    index |     17877.6 |     ms |
|                                  100th percentile service time |                                    index |     17883.7 |     ms |
|                                50th percentile processing time |                                    index |     974.173 |     ms |
|                                90th percentile processing time |                                    index |     1751.25 |     ms |
|                                99th percentile processing time |                                    index |      4523.8 |     ms |
|                              99.9th percentile processing time |                                    index |     15330.1 |     ms |
|                             99.99th percentile processing time |                                    index |     17880.9 |     ms |
|                               100th percentile processing time |                                    index |     17889.5 |     ms |
|                                                     error rate |                                    index |           0 |      % |
|                                                 Min Throughput |         date_histogram_calendar_interval |         0.1 |  ops/s |
|                                              Median Throughput |         date_histogram_calendar_interval |         0.1 |  ops/s |
|                                                 Max Throughput |         date_histogram_calendar_interval |         0.1 |  ops/s |
|                                        50th percentile latency |         date_histogram_calendar_interval |     6287.17 |     ms |
|                                        90th percentile latency |         date_histogram_calendar_interval |     6345.52 |     ms |
|                                       100th percentile latency |         date_histogram_calendar_interval |      6381.7 |     ms |
|                                   50th percentile service time |         date_histogram_calendar_interval |     6282.02 |     ms |
|                                   90th percentile service time |         date_histogram_calendar_interval |     6339.04 |     ms |
|                                  100th percentile service time |         date_histogram_calendar_interval |     6376.73 |     ms |
|                                50th percentile processing time |         date_histogram_calendar_interval |      6282.7 |     ms |
|                                90th percentile processing time |         date_histogram_calendar_interval |     6340.08 |     ms |
|                               100th percentile processing time |         date_histogram_calendar_interval |     6377.59 |     ms |
|                                                     error rate |         date_histogram_calendar_interval |           0 |      % |
|                                                 Min Throughput | date_histogram_calendar_interval_with_tz |        0.07 |  ops/s |
|                                              Median Throughput | date_histogram_calendar_interval_with_tz |        0.07 |  ops/s |
|                                                 Max Throughput | date_histogram_calendar_interval_with_tz |        0.07 |  ops/s |
|                                        50th percentile latency | date_histogram_calendar_interval_with_tz |     8216.95 |     ms |
|                                        90th percentile latency | date_histogram_calendar_interval_with_tz |     8298.43 |     ms |
|                                       100th percentile latency | date_histogram_calendar_interval_with_tz |     8756.23 |     ms |
|                                   50th percentile service time | date_histogram_calendar_interval_with_tz |     8208.87 |     ms |
|                                   90th percentile service time | date_histogram_calendar_interval_with_tz |     8290.23 |     ms |
|                                  100th percentile service time | date_histogram_calendar_interval_with_tz |      8747.4 |     ms |
|                                50th percentile processing time | date_histogram_calendar_interval_with_tz |      8210.8 |     ms |
|                                90th percentile processing time | date_histogram_calendar_interval_with_tz |     8293.12 |     ms |
|                               100th percentile processing time | date_histogram_calendar_interval_with_tz |     8748.26 |     ms |
|                                                     error rate | date_histogram_calendar_interval_with_tz |           0 |      % |
|                                                 Min Throughput |            date_histogram_fixed_interval |         0.1 |  ops/s |
|                                              Median Throughput |            date_histogram_fixed_interval |         0.1 |  ops/s |
|                                                 Max Throughput |            date_histogram_fixed_interval |         0.1 |  ops/s |
|                                        50th percentile latency |            date_histogram_fixed_interval |     5768.59 |     ms |
|                                        90th percentile latency |            date_histogram_fixed_interval |     5816.92 |     ms |
|                                       100th percentile latency |            date_histogram_fixed_interval |     6065.91 |     ms |
|                                   50th percentile service time |            date_histogram_fixed_interval |     5762.37 |     ms |
|                                   90th percentile service time |            date_histogram_fixed_interval |     5811.26 |     ms |
|                                  100th percentile service time |            date_histogram_fixed_interval |      6059.9 |     ms |
|                                50th percentile processing time |            date_histogram_fixed_interval |     5763.63 |     ms |
|                                90th percentile processing time |            date_histogram_fixed_interval |     5813.07 |     ms |
|                               100th percentile processing time |            date_histogram_fixed_interval |     6061.54 |     ms |
|                                                     error rate |            date_histogram_fixed_interval |           0 |      % |
|                                                 Min Throughput |    date_histogram_fixed_interval_with_tz |        0.07 |  ops/s |
|                                              Median Throughput |    date_histogram_fixed_interval_with_tz |        0.07 |  ops/s |
|                                                 Max Throughput |    date_histogram_fixed_interval_with_tz |        0.07 |  ops/s |
|                                        50th percentile latency |    date_histogram_fixed_interval_with_tz |     8087.81 |     ms |
|                                        90th percentile latency |    date_histogram_fixed_interval_with_tz |     8164.79 |     ms |
|                                       100th percentile latency |    date_histogram_fixed_interval_with_tz |     8216.71 |     ms |
|                                   50th percentile service time |    date_histogram_fixed_interval_with_tz |     8079.69 |     ms |
|                                   90th percentile service time |    date_histogram_fixed_interval_with_tz |     8156.35 |     ms |
|                                  100th percentile service time |    date_histogram_fixed_interval_with_tz |     8208.07 |     ms |
|                                50th percentile processing time |    date_histogram_fixed_interval_with_tz |     8080.61 |     ms |
|                                90th percentile processing time |    date_histogram_fixed_interval_with_tz |     8157.24 |     ms |
|                               100th percentile processing time |    date_histogram_fixed_interval_with_tz |     8208.96 |     ms |
|                                                     error rate |    date_histogram_fixed_interval_with_tz |           0 |      % |
|                                                 Min Throughput |                      auto_date_histogram |        0.07 |  ops/s |
|                                              Median Throughput |                      auto_date_histogram |        0.07 |  ops/s |
|                                                 Max Throughput |                      auto_date_histogram |        0.07 |  ops/s |
|                                        50th percentile latency |                      auto_date_histogram |     6204.94 |     ms |
|                                        90th percentile latency |                      auto_date_histogram |     6309.86 |     ms |
|                                       100th percentile latency |                      auto_date_histogram |     6369.88 |     ms |
|                                   50th percentile service time |                      auto_date_histogram |     6194.12 |     ms |
|                                   90th percentile service time |                      auto_date_histogram |     6302.21 |     ms |
|                                  100th percentile service time |                      auto_date_histogram |     6359.45 |     ms |
|                                50th percentile processing time |                      auto_date_histogram |     6194.92 |     ms |
|                                90th percentile processing time |                      auto_date_histogram |     6303.01 |     ms |
|                               100th percentile processing time |                      auto_date_histogram |     6360.27 |     ms |
|                                                     error rate |                      auto_date_histogram |           0 |      % |
|                                                 Min Throughput |              auto_date_histogram_with_tz |        0.06 |  ops/s |
|                                              Median Throughput |              auto_date_histogram_with_tz |        0.06 |  ops/s |
|                                                 Max Throughput |              auto_date_histogram_with_tz |        0.06 |  ops/s |
|                                        50th percentile latency |              auto_date_histogram_with_tz |     8645.49 |     ms |
|                                        90th percentile latency |              auto_date_histogram_with_tz |     8738.74 |     ms |
|                                       100th percentile latency |              auto_date_histogram_with_tz |     9112.54 |     ms |
|                                   50th percentile service time |              auto_date_histogram_with_tz |     8634.99 |     ms |
|                                   90th percentile service time |              auto_date_histogram_with_tz |     8728.55 |     ms |
|                                  100th percentile service time |              auto_date_histogram_with_tz |     9102.37 |     ms |
|                                50th percentile processing time |              auto_date_histogram_with_tz |     8635.79 |     ms |
|                                90th percentile processing time |              auto_date_histogram_with_tz |     8729.37 |     ms |
|                               100th percentile processing time |              auto_date_histogram_with_tz |     9103.21 |     ms |
|                                                     error rate |              auto_date_histogram_with_tz |           0 |      % |```

[ebadyano]

LGTM! Thank you!

[nik9000]

Thanks for reviewing @ebadyano !