Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix G1 GC default IHOP #46169

Merged
merged 1 commit into from Sep 23, 2019
Merged

Conversation

henningandersen
Copy link
Contributor

@henningandersen henningandersen commented Aug 30, 2019

G1 GC were setup to use an InitiatingHeapOccupancyPercent of 75. This
could leave used memory at a very high level for an extended duration,
triggering the real memory circuit breaker even at low activity levels.
The value is a threshold for old generation usage relative to total heap
size and thus it should leave room for the new generation. Default in
G1 is to allow up to 60 percent for new generation and this could mean that the
threshold was effectively at 135% heap usage. GC would still kick in of course and
eventually enough mixed collections would take place such that adaptive adjustment
of IHOP kicks in.

The JVM has adaptive setting of the IHOP, but this does not kick in
until it has sampled a few collections. A newly started, relatively
quiet server with primarily new generation activity could thus
experience heap above 95% frequently for a duration.

The changes here are two-fold:

  1. Use 30% default for IHOP (the JVM default of 45 could still mean
    105% heap usage threshold and did not fully ensure not to hit the
    circuit breaker with low activity)
  2. Set G1ReservePercent=25. This is used by the adaptive IHOP mechanism,
    meaning old/mixed GC should kick in no later than at 75% heap. This
    ensures IHOP stays compatible with the real memory circuit breaker also
    after being adjusted by adaptive IHOP.

G1 GC were setup to use an `InitiatingHeapOccupancyPercent` of 75. This
could leave used memory at a very high level for an extended duration,
triggering the real memory circuit breaker even at low activity levels.
The value is a threshold for old generation usage relative to total heap
size and thus it should leave room for the new generation. Default in
G1 is to allow up to 60 percent and this could mean that the threshold
was effectively at 135% heap usage.

The JVM has adaptive setting of the IHOP, but this does not kick in
until it has sampled a few collections. A newly started, relatively
quiet server with primarily new generation activity could thus
experience heap above 95% frequently for a duration.

The changes here are two-fold:
1. Use 30% default for IHOP (the JVM default of 45 could still mean
105% heap usage threshold and did not fully ensure not to hit the
circuit breaker with low activity)
2. Set G1ReservePercent=25. This is used by the adaptive IHOP mechanism,
meaning old/mixed GC should kick in no later than at 75% heap. This
ensures IHOP stays compatible with the real memory circuit breaker also
after being adjusted by adaptive IHOP.
@henningandersen henningandersen added >enhancement :Core/Infra/Circuit Breakers v8.0.0 v7.5.0 labels Aug 30, 2019
@elasticmachine
Copy link
Collaborator

@elasticmachine elasticmachine commented Aug 30, 2019

@henningandersen
Copy link
Contributor Author

@henningandersen henningandersen commented Sep 5, 2019

Following esrally command results in circuit breaking more than half the time:

esrally --distribution-version=7.3.0 --challenge=append-no-conflicts-index-only --track-params="bulk_size:10,bulk_indexing_clients:1,ingest_percentage:2" --on-error=abort --car="defaults,g1gc"

Using default G1 settings (which has IHOP 45) still results in circuit breaker failures around 1/3 of the runs.

Using the changed JVM settings from this PR does not result in any circuit breaking.

Copy link
Member

@jasontedor jasontedor left a comment

LGTM.

@henningandersen
Copy link
Contributor Author

@henningandersen henningandersen commented Sep 6, 2019

To validate that performance does not degrade with this change, I ran the geonames append-no-conflicts track with original G1 settings (IHOP 75) and new settings from this PR (on my local machine). This was run twice. Following are pair-wise comparisons of the two runs (baseline is the original, contender is the new settings). The total run time of the test runs were:

baseline 1: 4894 s
baseline 2: 4873 s
contender 1: 4967 s
contender 2: 4847 s

The java version used were:

openjdk version "12" 2019-03-19
|                                                        Metric |                           Task |   Baseline |   Contender |     Diff |    Unit |
|--------------------------------------------------------------:|-------------------------------:|-----------:|------------:|---------:|--------:|
|                    Cumulative indexing time of primary shards |                                |    53.7075 |     49.3474 | -4.36015 |     min |
|             Min cumulative indexing time across primary shard |                                |    10.5107 |     9.74062 | -0.77013 |     min |
|          Median cumulative indexing time across primary shard |                                |    10.7407 |     9.86627 | -0.87442 |     min |
|             Max cumulative indexing time across primary shard |                                |    10.8875 |      9.9731 | -0.91437 |     min |
|           Cumulative indexing throttle time of primary shards |                                |          0 |           0 |        0 |     min |
|    Min cumulative indexing throttle time across primary shard |                                |          0 |           0 |        0 |     min |
| Median cumulative indexing throttle time across primary shard |                                |          0 |           0 |        0 |     min |
|    Max cumulative indexing throttle time across primary shard |                                |          0 |           0 |        0 |     min |
|                       Cumulative merge time of primary shards |                                |    23.4692 |     21.4138 | -2.05542 |     min |
|                      Cumulative merge count of primary shards |                                |         65 |          64 |       -1 |         |
|                Min cumulative merge time across primary shard |                                |    3.59612 |      3.0981 | -0.49802 |     min |
|             Median cumulative merge time across primary shard |                                |     5.0119 |     4.55842 | -0.45348 |     min |
|                Max cumulative merge time across primary shard |                                |    5.16957 |     4.63455 | -0.53502 |     min |
|              Cumulative merge throttle time of primary shards |                                |    3.33188 |     3.45387 |  0.12198 |     min |
|       Min cumulative merge throttle time across primary shard |                                |    0.40695 |      0.2302 | -0.17675 |     min |
|    Median cumulative merge throttle time across primary shard |                                |    0.65935 |      0.8339 |  0.17455 |     min |
|       Max cumulative merge throttle time across primary shard |                                |   0.840517 |    0.859567 |  0.01905 |     min |
|                     Cumulative refresh time of primary shards |                                |    7.98197 |     7.11847 |  -0.8635 |     min |
|                    Cumulative refresh count of primary shards |                                |        276 |         278 |        2 |         |
|              Min cumulative refresh time across primary shard |                                |    1.52805 |     1.40523 | -0.12282 |     min |
|           Median cumulative refresh time across primary shard |                                |     1.5819 |     1.41908 | -0.16282 |     min |
|              Max cumulative refresh time across primary shard |                                |    1.66415 |     1.44357 | -0.22058 |     min |
|                       Cumulative flush time of primary shards |                                |   0.315167 |    0.331867 |   0.0167 |     min |
|                      Cumulative flush count of primary shards |                                |         10 |          10 |        0 |         |
|                Min cumulative flush time across primary shard |                                |  0.0419333 |   0.0500833 |  0.00815 |     min |
|             Median cumulative flush time across primary shard |                                |  0.0703667 |   0.0677167 | -0.00265 |     min |
|                Max cumulative flush time across primary shard |                                |     0.0755 |   0.0796333 |  0.00413 |     min |
|                                            Total Young Gen GC |                                |     28.164 |      32.478 |    4.314 |       s |
|                                              Total Old Gen GC |                                |          0 |           0 |        0 |       s |
|                                                    Store size |                                |    3.06079 |     3.05187 | -0.00893 |      GB |
|                                                 Translog size |                                |    2.79351 |      2.7924 | -0.00111 |      GB |
|                                                    Index size |                                |    5.85432 |     5.84428 | -0.01003 |      GB |
|                                                 Total written |                                |    16.1713 |     16.1622 | -0.00909 |      GB |
|                                        Heap used for segments |                                |    4.56532 |     4.62933 |    0.064 |      MB |
|                                      Heap used for doc values |                                |  0.0703049 |   0.0740623 |  0.00376 |      MB |
|                                           Heap used for terms |                                |    3.38734 |     3.43461 |  0.04727 |      MB |
|                                           Heap used for norms |                                |  0.0654297 |   0.0752563 |  0.00983 |      MB |
|                                          Heap used for points |                                |   0.255676 |    0.252288 | -0.00339 |      MB |
|                                   Heap used for stored fields |                                |   0.786568 |    0.793106 |  0.00654 |      MB |
|                                                 Segment count |                                |         86 |          99 |       13 |         |
|                                                Min Throughput |                   index-append |    24238.3 |     26776.8 |  2538.43 |  docs/s |
|                                             Median Throughput |                   index-append |    25738.4 |     27546.7 |  1808.23 |  docs/s |
|                                                Max Throughput |                   index-append |    26190.8 |     28091.9 |  1901.09 |  docs/s |
|                                       50th percentile latency |                   index-append |    1257.18 |     1190.07 | -67.1092 |      ms |
|                                       90th percentile latency |                   index-append |    2386.15 |     2275.46 | -110.687 |      ms |
|                                       99th percentile latency |                   index-append |    3764.18 |     3358.96 | -405.218 |      ms |
|                                     99.9th percentile latency |                   index-append |    4289.57 |      5050.8 |  761.227 |      ms |
|                                      100th percentile latency |                   index-append |    5033.61 |     6254.59 |  1220.99 |      ms |
|                                  50th percentile service time |                   index-append |    1257.18 |     1190.07 | -67.1092 |      ms |
|                                  90th percentile service time |                   index-append |    2386.15 |     2275.46 | -110.687 |      ms |
|                                  99th percentile service time |                   index-append |    3764.18 |     3358.96 | -405.218 |      ms |
|                                99.9th percentile service time |                   index-append |    4289.57 |      5050.8 |  761.227 |      ms |
|                                 100th percentile service time |                   index-append |    5033.61 |     6254.59 |  1220.99 |      ms |
|                                                    error rate |                   index-append |          0 |           0 |        0 |       % |
|                                                Min Throughput |                    index-stats |    88.6366 |     87.2373 | -1.39931 |   ops/s |
|                                             Median Throughput |                    index-stats |    90.0284 |     89.9065 | -0.12196 |   ops/s |
|                                                Max Throughput |                    index-stats |    90.0719 |     89.9651 | -0.10677 |   ops/s |
|                                       50th percentile latency |                    index-stats |     7.8976 |     9.69223 |  1.79462 |      ms |
|                                       90th percentile latency |                    index-stats |    30.0101 |     82.4034 |  52.3933 |      ms |
|                                       99th percentile latency |                    index-stats |    134.775 |     329.001 |  194.226 |      ms |
|                                     99.9th percentile latency |                    index-stats |    149.184 |     340.958 |  191.774 |      ms |
|                                      100th percentile latency |                    index-stats |    154.332 |     342.075 |  187.743 |      ms |
|                                  50th percentile service time |                    index-stats |    7.29423 |     8.88469 |  1.59046 |      ms |
|                                  90th percentile service time |                    index-stats |    12.9509 |     10.6884 | -2.26249 |      ms |
|                                  99th percentile service time |                    index-stats |    17.6599 |     17.7253 |  0.06542 |      ms |
|                                99.9th percentile service time |                    index-stats |    51.1144 |     54.2444 |     3.13 |      ms |
|                                 100th percentile service time |                    index-stats |    78.9285 |      323.51 |  244.582 |      ms |
|                                                    error rate |                    index-stats |          0 |           0 |        0 |       % |
|                                                Min Throughput |                     node-stats |    89.9886 |     89.9973 |  0.00863 |   ops/s |
|                                             Median Throughput |                     node-stats |    90.0448 |     90.0268 | -0.01797 |   ops/s |
|                                                Max Throughput |                     node-stats |    90.1369 |     90.1334 | -0.00351 |   ops/s |
|                                       50th percentile latency |                     node-stats |    6.37825 |     7.21304 |  0.83479 |      ms |
|                                       90th percentile latency |                     node-stats |    11.6056 |     11.2865 | -0.31908 |      ms |
|                                       99th percentile latency |                     node-stats |    143.766 |     77.1808 | -66.5856 |      ms |
|                                     99.9th percentile latency |                     node-stats |    173.721 |     108.397 | -65.3244 |      ms |
|                                      100th percentile latency |                     node-stats |      174.5 |     113.752 |  -60.748 |      ms |
|                                  50th percentile service time |                     node-stats |    5.58809 |     6.78797 |  1.19988 |      ms |
|                                  90th percentile service time |                     node-stats |    8.85045 |     10.0326 |  1.18215 |      ms |
|                                  99th percentile service time |                     node-stats |    13.4348 |     13.3504 | -0.08439 |      ms |
|                                99.9th percentile service time |                     node-stats |    28.4861 |     57.8926 |  29.4065 |      ms |
|                                 100th percentile service time |                     node-stats |    172.245 |     113.662 | -58.5827 |      ms |
|                                                    error rate |                     node-stats |          0 |           0 |        0 |       % |
|                                                Min Throughput |                        default |    50.0264 |     50.0273 |  0.00087 |   ops/s |
|                                             Median Throughput |                        default |    50.0381 |     50.0443 |  0.00612 |   ops/s |
|                                                Max Throughput |                        default |    50.0914 |     50.0896 | -0.00172 |   ops/s |
|                                       50th percentile latency |                        default |    4.81666 |     4.59082 | -0.22585 |      ms |
|                                       90th percentile latency |                        default |    5.42616 |     5.27536 | -0.15079 |      ms |
|                                       99th percentile latency |                        default |     13.802 |      10.899 | -2.90302 |      ms |
|                                     99.9th percentile latency |                        default |    48.4871 |     19.4301 |  -29.057 |      ms |
|                                      100th percentile latency |                        default |    63.9021 |     22.6654 | -41.2368 |      ms |
|                                  50th percentile service time |                        default |    4.66389 |     4.42536 | -0.23854 |      ms |
|                                  90th percentile service time |                        default |    5.24027 |     5.05257 |  -0.1877 |      ms |
|                                  99th percentile service time |                        default |     10.733 |     9.26762 | -1.46534 |      ms |
|                                99.9th percentile service time |                        default |    20.4528 |     15.6059 | -4.84691 |      ms |
|                                 100th percentile service time |                        default |    63.7959 |     18.9315 | -44.8645 |      ms |
|                                                    error rate |                        default |          0 |           0 |        0 |       % |
|                                                Min Throughput |                           term |    200.046 |     200.054 |  0.00857 |   ops/s |
|                                             Median Throughput |                           term |    200.103 |     200.072 | -0.03171 |   ops/s |
|                                                Max Throughput |                           term |    200.262 |     200.108 | -0.15376 |   ops/s |
|                                       50th percentile latency |                           term |    2.83937 |     3.01057 |   0.1712 |      ms |
|                                       90th percentile latency |                           term |    3.76364 |     3.73891 | -0.02473 |      ms |
|                                       99th percentile latency |                           term |    49.4881 |     12.3567 | -37.1314 |      ms |
|                                     99.9th percentile latency |                           term |    64.9251 |     17.5166 | -47.4085 |      ms |
|                                      100th percentile latency |                           term |    66.0303 |     19.4749 | -46.5554 |      ms |
|                                  50th percentile service time |                           term |    2.40081 |     2.78528 |  0.38448 |      ms |
|                                  90th percentile service time |                           term |    3.52701 |     3.56793 |  0.04092 |      ms |
|                                  99th percentile service time |                           term |     5.9819 |     7.72834 |  1.74644 |      ms |
|                                99.9th percentile service time |                           term |    13.7905 |     15.4018 |  1.61135 |      ms |
|                                 100th percentile service time |                           term |    65.8033 |     16.2241 | -49.5792 |      ms |
|                                                    error rate |                           term |          0 |           0 |        0 |       % |
|                                                Min Throughput |                         phrase |    199.802 |      199.67 | -0.13202 |   ops/s |
|                                             Median Throughput |                         phrase |    200.066 |     200.058 | -0.00853 |   ops/s |
|                                                Max Throughput |                         phrase |    200.118 |     200.218 |   0.1009 |   ops/s |
|                                       50th percentile latency |                         phrase |    2.57122 |      2.3156 | -0.25562 |      ms |
|                                       90th percentile latency |                         phrase |    3.72015 |     4.67726 |  0.95711 |      ms |
|                                       99th percentile latency |                         phrase |     6.4336 |     10.5413 |  4.10766 |      ms |
|                                     99.9th percentile latency |                         phrase |    11.4627 |     13.2151 |  1.75244 |      ms |
|                                      100th percentile latency |                         phrase |    11.6838 |     13.2919 |  1.60811 |      ms |
|                                  50th percentile service time |                         phrase |    2.44046 |     2.17938 | -0.26108 |      ms |
|                                  90th percentile service time |                         phrase |    3.52559 |     4.28193 |  0.75635 |      ms |
|                                  99th percentile service time |                         phrase |    5.11863 |     5.56661 |  0.44798 |      ms |
|                                99.9th percentile service time |                         phrase |     9.8565 |     12.0693 |  2.21279 |      ms |
|                                 100th percentile service time |                         phrase |    10.0205 |     13.1503 |  3.12977 |      ms |
|                                                    error rate |                         phrase |          0 |           0 |        0 |       % |
|                                                Min Throughput |           country_agg_uncached |    2.39848 |     2.33549 | -0.06299 |   ops/s |
|                                             Median Throughput |           country_agg_uncached |    2.40073 |     2.34014 | -0.06059 |   ops/s |
|                                                Max Throughput |           country_agg_uncached |    2.40282 |     2.34268 | -0.06014 |   ops/s |
|                                       50th percentile latency |           country_agg_uncached |    41975.8 |     44754.9 |  2779.11 |      ms |
|                                       90th percentile latency |           country_agg_uncached |    48609.5 |     51610.3 |  3000.77 |      ms |
|                                       99th percentile latency |           country_agg_uncached |    49957.1 |     53330.1 |     3373 |      ms |
|                                      100th percentile latency |           country_agg_uncached |    50131.8 |     53519.9 |  3388.11 |      ms |
|                                  50th percentile service time |           country_agg_uncached |    410.519 |      421.28 |   10.761 |      ms |
|                                  90th percentile service time |           country_agg_uncached |    438.374 |     446.765 |  8.39021 |      ms |
|                                  99th percentile service time |           country_agg_uncached |    488.612 |     501.427 |  12.8154 |      ms |
|                                 100th percentile service time |           country_agg_uncached |    537.589 |     528.904 | -8.68557 |      ms |
|                                                    error rate |           country_agg_uncached |          0 |           0 |        0 |       % |
|                                                Min Throughput |             country_agg_cached |    100.037 |     100.045 |  0.00884 |   ops/s |
|                                             Median Throughput |             country_agg_cached |    100.053 |     100.064 |  0.01067 |   ops/s |
|                                                Max Throughput |             country_agg_cached |    100.155 |     100.118 | -0.03682 |   ops/s |
|                                       50th percentile latency |             country_agg_cached |    4.43335 |     4.41736 | -0.01599 |      ms |
|                                       90th percentile latency |             country_agg_cached |    4.89848 |     4.76643 | -0.13205 |      ms |
|                                       99th percentile latency |             country_agg_cached |    41.2872 |     13.6812 |  -27.606 |      ms |
|                                     99.9th percentile latency |             country_agg_cached |    68.1404 |     31.5278 | -36.6126 |      ms |
|                                      100th percentile latency |             country_agg_cached |    71.1073 |      37.096 | -34.0113 |      ms |
|                                  50th percentile service time |             country_agg_cached |     4.1917 |     4.25326 |  0.06156 |      ms |
|                                  90th percentile service time |             country_agg_cached |    4.57786 |       4.536 | -0.04185 |      ms |
|                                  99th percentile service time |             country_agg_cached |    9.33099 |     8.38117 | -0.94982 |      ms |
|                                99.9th percentile service time |             country_agg_cached |     67.968 |     28.4119 | -39.5561 |      ms |
|                                 100th percentile service time |             country_agg_cached |    70.9247 |     36.9294 | -33.9953 |      ms |
|                                                    error rate |             country_agg_cached |          0 |           0 |        0 |       % |
|                                                Min Throughput |                         scroll |    20.0398 |     20.0424 |  0.00255 | pages/s |
|                                             Median Throughput |                         scroll |    20.0532 |     20.0547 |   0.0015 | pages/s |
|                                                Max Throughput |                         scroll |     20.071 |     20.0697 | -0.00137 | pages/s |
|                                       50th percentile latency |                         scroll |    412.036 |     393.434 | -18.6019 |      ms |
|                                       90th percentile latency |                         scroll |    457.481 |     425.471 | -32.0103 |      ms |
|                                       99th percentile latency |                         scroll |     511.93 |     504.432 | -7.49775 |      ms |
|                                      100th percentile latency |                         scroll |    537.447 |     513.147 | -24.2998 |      ms |
|                                  50th percentile service time |                         scroll |    411.141 |     392.468 | -18.6728 |      ms |
|                                  90th percentile service time |                         scroll |    456.615 |     424.712 | -31.9033 |      ms |
|                                  99th percentile service time |                         scroll |    510.795 |     503.521 | -7.27353 |      ms |
|                                 100th percentile service time |                         scroll |    536.884 |     512.292 | -24.5914 |      ms |
|                                                    error rate |                         scroll |          0 |           0 |        0 |       % |
|                                                Min Throughput |                     expression |    1.24093 |     1.29202 |  0.05109 |   ops/s |
|                                             Median Throughput |                     expression |    1.25319 |     1.29815 |  0.04496 |   ops/s |
|                                                Max Throughput |                     expression |    1.30648 |     1.30304 | -0.00344 |   ops/s |
|                                       50th percentile latency |                     expression |    75261.4 |     68210.9 | -7050.44 |      ms |
|                                       90th percentile latency |                     expression |    88836.2 |     79737.2 | -9098.99 |      ms |
|                                       99th percentile latency |                     expression |    91105.1 |     82410.4 | -8694.71 |      ms |
|                                      100th percentile latency |                     expression |    91314.9 |       82681 | -8633.91 |      ms |
|                                  50th percentile service time |                     expression |    800.379 |     777.247 | -23.1312 |      ms |
|                                  90th percentile service time |                     expression |    1134.67 |     833.324 |  -301.35 |      ms |
|                                  99th percentile service time |                     expression |    1330.58 |     860.375 |  -470.21 |      ms |
|                                 100th percentile service time |                     expression |    1565.72 |     878.702 | -687.023 |      ms |
|                                                    error rate |                     expression |          0 |           0 |        0 |       % |
|                                                Min Throughput |                painless_static |   0.877083 |    0.931729 |  0.05465 |   ops/s |
|                                             Median Throughput |                painless_static |   0.882253 |    0.932982 |  0.05073 |   ops/s |
|                                                Max Throughput |                painless_static |   0.883936 |    0.935446 |  0.05151 |   ops/s |
|                                       50th percentile latency |                painless_static |     117390 |      102191 | -15198.7 |      ms |
|                                       90th percentile latency |                painless_static |     137007 |      118559 | -18448.5 |      ms |
|                                       99th percentile latency |                painless_static |     140843 |      122198 | -18645.2 |      ms |
|                                      100th percentile latency |                painless_static |     141228 |      122601 | -18626.7 |      ms |
|                                  50th percentile service time |                painless_static |    1122.06 |     1077.21 | -44.8448 |      ms |
|                                  90th percentile service time |                painless_static |    1234.68 |     1138.32 |  -96.356 |      ms |
|                                  99th percentile service time |                painless_static |     1428.2 |     1180.46 | -247.741 |      ms |
|                                 100th percentile service time |                painless_static |    1620.91 |     1191.36 | -429.551 |      ms |
|                                                    error rate |                painless_static |          0 |           0 |        0 |       % |
|                                                Min Throughput |               painless_dynamic |    1.02729 |     1.06004 |  0.03275 |   ops/s |
|                                             Median Throughput |               painless_dynamic |    1.03117 |     1.06111 |  0.02994 |   ops/s |
|                                                Max Throughput |               painless_dynamic |    1.03301 |     1.06381 |   0.0308 |   ops/s |
|                                       50th percentile latency |               painless_dynamic |    76625.9 |     69745.5 | -6880.38 |      ms |
|                                       90th percentile latency |               painless_dynamic |    89417.4 |     80007.1 | -9410.35 |      ms |
|                                       99th percentile latency |               painless_dynamic |    92345.2 |     82459.2 | -9886.01 |      ms |
|                                      100th percentile latency |               painless_dynamic |    92696.8 |     82681.9 | -10014.9 |      ms |
|                                  50th percentile service time |               painless_dynamic |    971.882 |     933.768 | -38.1143 |      ms |
|                                  90th percentile service time |               painless_dynamic |    1040.61 |     994.676 | -45.9336 |      ms |
|                                  99th percentile service time |               painless_dynamic |    1143.47 |     1055.88 | -87.5911 |      ms |
|                                 100th percentile service time |               painless_dynamic |    1153.25 |     1069.53 | -83.7162 |      ms |
|                                                    error rate |               painless_dynamic |          0 |           0 |        0 |       % |
|                                                Min Throughput | decay_geo_gauss_function_score |    1.00133 |      1.0014 |    7e-05 |   ops/s |
|                                             Median Throughput | decay_geo_gauss_function_score |    1.00169 |     1.00173 |    4e-05 |   ops/s |
|                                                Max Throughput | decay_geo_gauss_function_score |     1.0024 |     1.00246 |    6e-05 |   ops/s |
|                                       50th percentile latency | decay_geo_gauss_function_score |    577.945 |     568.486 | -9.45888 |      ms |
|                                       90th percentile latency | decay_geo_gauss_function_score |    617.596 |     602.031 |  -15.565 |      ms |
|                                       99th percentile latency | decay_geo_gauss_function_score |    627.227 |     626.788 | -0.43873 |      ms |
|                                      100th percentile latency | decay_geo_gauss_function_score |    663.415 |     665.595 |  2.18001 |      ms |
|                                  50th percentile service time | decay_geo_gauss_function_score |    577.306 |     567.815 | -9.49146 |      ms |
|                                  90th percentile service time | decay_geo_gauss_function_score |    616.992 |      601.49 | -15.5027 |      ms |
|                                  99th percentile service time | decay_geo_gauss_function_score |    626.442 |      626.24 | -0.20256 |      ms |
|                                 100th percentile service time | decay_geo_gauss_function_score |    662.876 |     665.058 |  2.18195 |      ms |
|                                                    error rate | decay_geo_gauss_function_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |   decay_geo_gauss_script_score |    1.00074 |     1.00063 | -0.00011 |   ops/s |
|                                             Median Throughput |   decay_geo_gauss_script_score |    1.00144 |     1.00145 |    1e-05 |   ops/s |
|                                                Max Throughput |   decay_geo_gauss_script_score |    1.00225 |     1.00212 | -0.00012 |   ops/s |
|                                       50th percentile latency |   decay_geo_gauss_script_score |    642.078 |     632.447 | -9.63059 |      ms |
|                                       90th percentile latency |   decay_geo_gauss_script_score |    723.523 |     687.778 | -35.7444 |      ms |
|                                       99th percentile latency |   decay_geo_gauss_script_score |    785.887 |     786.682 |  0.79499 |      ms |
|                                      100th percentile latency |   decay_geo_gauss_script_score |    796.748 |     824.354 |  27.6059 |      ms |
|                                  50th percentile service time |   decay_geo_gauss_script_score |    641.534 |     632.018 | -9.51623 |      ms |
|                                  90th percentile service time |   decay_geo_gauss_script_score |    723.073 |     687.308 | -35.7649 |      ms |
|                                  99th percentile service time |   decay_geo_gauss_script_score |    785.432 |     786.299 |  0.86681 |      ms |
|                                 100th percentile service time |   decay_geo_gauss_script_score |    796.266 |     823.801 |  27.5343 |      ms |
|                                                    error rate |   decay_geo_gauss_script_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |     field_value_function_score |    1.50332 |     1.50306 | -0.00026 |   ops/s |
|                                             Median Throughput |     field_value_function_score |    1.50429 |     1.50389 |  -0.0004 |   ops/s |
|                                                Max Throughput |     field_value_function_score |    1.50532 |     1.50542 |   0.0001 |   ops/s |
|                                       50th percentile latency |     field_value_function_score |    208.874 |     246.172 |  37.2977 |      ms |
|                                       90th percentile latency |     field_value_function_score |    231.017 |     268.829 |  37.8123 |      ms |
|                                       99th percentile latency |     field_value_function_score |    274.562 |     294.795 |   20.233 |      ms |
|                                      100th percentile latency |     field_value_function_score |    338.173 |     338.445 |  0.27154 |      ms |
|                                  50th percentile service time |     field_value_function_score |    208.394 |     245.572 |  37.1785 |      ms |
|                                  90th percentile service time |     field_value_function_score |    230.316 |      268.13 |  37.8135 |      ms |
|                                  99th percentile service time |     field_value_function_score |    273.813 |     294.308 |  20.4955 |      ms |
|                                 100th percentile service time |     field_value_function_score |    337.551 |     337.797 |  0.24514 |      ms |
|                                                    error rate |     field_value_function_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |       field_value_script_score |    1.50173 |     1.50191 |  0.00018 |   ops/s |
|                                             Median Throughput |       field_value_script_score |    1.50275 |      1.5029 |  0.00015 |   ops/s |
|                                                Max Throughput |       field_value_script_score |    1.50378 |      1.5044 |  0.00061 |   ops/s |
|                                       50th percentile latency |       field_value_script_score |    356.498 |     338.477 | -18.0211 |      ms |
|                                       90th percentile latency |       field_value_script_score |    370.987 |     363.905 |   -7.082 |      ms |
|                                       99th percentile latency |       field_value_script_score |    468.319 |     407.626 | -60.6928 |      ms |
|                                      100th percentile latency |       field_value_script_score |    492.465 |      484.38 |  -8.0857 |      ms |
|                                  50th percentile service time |       field_value_script_score |    355.943 |     337.914 | -18.0289 |      ms |
|                                  90th percentile service time |       field_value_script_score |    370.575 |     363.417 | -7.15805 |      ms |
|                                  99th percentile service time |       field_value_script_score |    467.887 |     406.997 | -60.8897 |      ms |
|                                 100th percentile service time |       field_value_script_score |    492.026 |     483.796 | -8.23071 |      ms |
|                                                    error rate |       field_value_script_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |          random_function_score |    1.49916 |     1.49928 |  0.00012 |   ops/s |
|                                             Median Throughput |          random_function_score |    1.50071 |     1.50075 |    4e-05 |   ops/s |
|                                                Max Throughput |          random_function_score |     1.5016 |     1.50176 |  0.00016 |   ops/s |
|                                       50th percentile latency |          random_function_score |    588.617 |     578.176 | -10.4414 |      ms |
|                                       90th percentile latency |          random_function_score |    620.587 |     618.399 | -2.18755 |      ms |
|                                       99th percentile latency |          random_function_score |    720.058 |     724.014 |  3.95553 |      ms |
|                                      100th percentile latency |          random_function_score |    763.752 |     741.745 | -22.0067 |      ms |
|                                  50th percentile service time |          random_function_score |    588.152 |       573.4 | -14.7523 |      ms |
|                                  90th percentile service time |          random_function_score |    620.255 |     617.926 | -2.32913 |      ms |
|                                  99th percentile service time |          random_function_score |    719.765 |     723.721 |  3.95613 |      ms |
|                                 100th percentile service time |          random_function_score |    763.629 |     741.399 | -22.2301 |      ms |
|                                                    error rate |          random_function_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |            random_script_score |    1.49852 |     1.49395 | -0.00457 |   ops/s |
|                                             Median Throughput |            random_script_score |    1.50017 |     1.49981 | -0.00036 |   ops/s |
|                                                Max Throughput |            random_script_score |    1.50094 |     1.50093 |   -1e-05 |   ops/s |
|                                       50th percentile latency |            random_script_score |    642.856 |     676.129 |  33.2732 |      ms |
|                                       90th percentile latency |            random_script_score |    739.716 |     974.004 |  234.288 |      ms |
|                                       99th percentile latency |            random_script_score |    809.802 |     1191.09 |  381.284 |      ms |
|                                      100th percentile latency |            random_script_score |    825.136 |     1225.26 |  400.121 |      ms |
|                                  50th percentile service time |            random_script_score |     632.77 |     642.293 |  9.52368 |      ms |
|                                  90th percentile service time |            random_script_score |    687.759 |     713.851 |  26.0918 |      ms |
|                                  99th percentile service time |            random_script_score |    808.296 |     823.529 |  15.2324 |      ms |
|                                 100th percentile service time |            random_script_score |    824.975 |     824.962 | -0.01359 |      ms |
|                                                    error rate |            random_script_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |                    large_terms |     1.3163 |       1.075 | -0.24129 |   ops/s |
|                                             Median Throughput |                    large_terms |    1.32275 |     1.07814 | -0.24461 |   ops/s |
|                                                Max Throughput |                    large_terms |     1.3251 |      1.0797 |  -0.2454 |   ops/s |
|                                       50th percentile latency |                    large_terms |    22954.3 |     66002.9 |  43048.6 |      ms |
|                                       90th percentile latency |                    large_terms |    26350.6 |     76042.7 |  49692.1 |      ms |
|                                       99th percentile latency |                    large_terms |    27390.6 |     78387.2 |  50996.7 |      ms |
|                                      100th percentile latency |                    large_terms |    27493.2 |     78683.4 |  51190.3 |      ms |
|                                  50th percentile service time |                    large_terms |    746.656 |     914.012 |  167.356 |      ms |
|                                  90th percentile service time |                    large_terms |    793.025 |     965.919 |  172.894 |      ms |
|                                  99th percentile service time |                    large_terms |    855.852 |      1030.8 |   174.95 |      ms |
|                                 100th percentile service time |                    large_terms |    867.982 |     1068.06 |  200.074 |      ms |
|                                                    error rate |                    large_terms |          0 |           0 |        0 |       % |
|                                                Min Throughput |           large_filtered_terms |    1.32372 |     1.09946 | -0.22426 |   ops/s |
|                                             Median Throughput |           large_filtered_terms |    1.32492 |     1.10557 | -0.21935 |   ops/s |
|                                                Max Throughput |           large_filtered_terms |    1.32674 |     1.10734 |  -0.2194 |   ops/s |
|                                       50th percentile latency |           large_filtered_terms |    22600.8 |     60248.9 |  37648.1 |      ms |
|                                       90th percentile latency |           large_filtered_terms |    26341.5 |     70952.9 |  44611.3 |      ms |
|                                       99th percentile latency |           large_filtered_terms |    27066.5 |     73265.5 |    46199 |      ms |
|                                      100th percentile latency |           large_filtered_terms |    27135.7 |     73528.3 |  46392.6 |      ms |
|                                  50th percentile service time |           large_filtered_terms |     756.66 |     907.606 |  150.947 |      ms |
|                                  90th percentile service time |           large_filtered_terms |    808.264 |     987.237 |  178.973 |      ms |
|                                  99th percentile service time |           large_filtered_terms |    880.289 |     1061.81 |  181.521 |      ms |
|                                 100th percentile service time |           large_filtered_terms |    897.621 |     1084.64 |  187.019 |      ms |
|                                                    error rate |           large_filtered_terms |          0 |           0 |        0 |       % |
|                                                Min Throughput |         large_prohibited_terms |     1.4018 |     1.14954 | -0.25226 |   ops/s |
|                                             Median Throughput |         large_prohibited_terms |    1.40361 |      1.1514 | -0.25221 |   ops/s |
|                                                Max Throughput |         large_prohibited_terms |    1.40737 |     1.15314 | -0.25423 |   ops/s |
|                                       50th percentile latency |         large_prohibited_terms |    12266.8 |     51031.6 |  38764.9 |      ms |
|                                       90th percentile latency |         large_prohibited_terms |    14129.8 |     59239.1 |  45109.3 |      ms |
|                                       99th percentile latency |         large_prohibited_terms |    14516.9 |     61183.4 |  46666.5 |      ms |
|                                      100th percentile latency |         large_prohibited_terms |    14580.7 |       61431 |  46850.2 |      ms |
|                                  50th percentile service time |         large_prohibited_terms |    709.467 |     868.756 |   159.29 |      ms |
|                                  90th percentile service time |         large_prohibited_terms |    760.564 |     916.764 |    156.2 |      ms |
|                                  99th percentile service time |         large_prohibited_terms |    822.712 |     969.015 |  146.303 |      ms |
|                                 100th percentile service time |         large_prohibited_terms |    876.083 |      982.54 |  106.457 |      ms |
|                                                    error rate |         large_prohibited_terms |          0 |           0 |        0 |       % |
|                                                Min Throughput |           desc_sort_population |    1.50362 |     1.50368 |    6e-05 |   ops/s |
|                                             Median Throughput |           desc_sort_population |    1.50481 |     1.50455 | -0.00026 |   ops/s |
|                                                Max Throughput |           desc_sort_population |    1.50619 |     1.50632 |  0.00014 |   ops/s |
|                                       50th percentile latency |           desc_sort_population |    121.928 |     170.462 |  48.5345 |      ms |
|                                       90th percentile latency |           desc_sort_population |    174.497 |     184.763 |  10.2661 |      ms |
|                                       99th percentile latency |           desc_sort_population |    187.851 |     187.211 | -0.63973 |      ms |
|                                      100th percentile latency |           desc_sort_population |    255.309 |     190.902 | -64.4067 |      ms |
|                                  50th percentile service time |           desc_sort_population |    121.292 |     169.776 |  48.4847 |      ms |
|                                  90th percentile service time |           desc_sort_population |    173.921 |     184.151 |  10.2305 |      ms |
|                                  99th percentile service time |           desc_sort_population |    187.253 |     186.581 | -0.67199 |      ms |
|                                 100th percentile service time |           desc_sort_population |    254.621 |     190.319 | -64.3023 |      ms |
|                                                    error rate |           desc_sort_population |          0 |           0 |        0 |       % |
|                                                Min Throughput |            asc_sort_population |    1.50383 |     1.50364 | -0.00019 |   ops/s |
|                                             Median Throughput |            asc_sort_population |    1.50487 |     1.50477 |   -9e-05 |   ops/s |
|                                                Max Throughput |            asc_sort_population |    1.50638 |     1.50643 |    5e-05 |   ops/s |
|                                       50th percentile latency |            asc_sort_population |    117.086 |     169.812 |  52.7262 |      ms |
|                                       90th percentile latency |            asc_sort_population |    164.316 |      185.77 |  21.4536 |      ms |
|                                       99th percentile latency |            asc_sort_population |    176.884 |     190.145 |  13.2615 |      ms |
|                                      100th percentile latency |            asc_sort_population |    259.164 |     191.502 | -67.6612 |      ms |
|                                  50th percentile service time |            asc_sort_population |    116.443 |     169.221 |  52.7779 |      ms |
|                                  90th percentile service time |            asc_sort_population |    163.662 |     185.014 |  21.3519 |      ms |
|                                  99th percentile service time |            asc_sort_population |    176.325 |     189.546 |   13.221 |      ms |
|                                 100th percentile service time |            asc_sort_population |    258.921 |     190.743 | -68.1789 |      ms |
|                                                    error rate |            asc_sort_population |          0 |           0 |        0 |       % |
|                                                Min Throughput |            desc_sort_geonameid |    1.50389 |     1.50367 | -0.00022 |   ops/s |
|                                             Median Throughput |            desc_sort_geonameid |     1.5049 |     1.50486 |   -4e-05 |   ops/s |
|                                                Max Throughput |            desc_sort_geonameid |    1.50632 |     1.50652 |   0.0002 |   ops/s |
|                                       50th percentile latency |            desc_sort_geonameid |    108.285 |     158.143 |  49.8573 |      ms |
|                                       90th percentile latency |            desc_sort_geonameid |    156.686 |     178.278 |  21.5914 |      ms |
|                                       99th percentile latency |            desc_sort_geonameid |    169.383 |     184.506 |  15.1228 |      ms |
|                                      100th percentile latency |            desc_sort_geonameid |    178.676 |     186.621 |  7.94528 |      ms |
|                                  50th percentile service time |            desc_sort_geonameid |    107.452 |     157.368 |   49.916 |      ms |
|                                  90th percentile service time |            desc_sort_geonameid |    156.004 |     177.635 |   21.631 |      ms |
|                                  99th percentile service time |            desc_sort_geonameid |    168.721 |     183.905 |  15.1842 |      ms |
|                                 100th percentile service time |            desc_sort_geonameid |    177.864 |     186.419 |  8.55503 |      ms |
|                                                    error rate |            desc_sort_geonameid |          0 |           0 |        0 |       % |
|                                                Min Throughput |             asc_sort_geonameid |    1.50386 |      1.5038 |   -6e-05 |   ops/s |
|                                             Median Throughput |             asc_sort_geonameid |    1.50497 |     1.50461 | -0.00037 |   ops/s |
|                                                Max Throughput |             asc_sort_geonameid |     1.5064 |     1.50652 |  0.00011 |   ops/s |
|                                       50th percentile latency |             asc_sort_geonameid |    110.562 |     160.514 |  49.9524 |      ms |
|                                       90th percentile latency |             asc_sort_geonameid |    159.017 |     179.481 |  20.4638 |      ms |
|                                       99th percentile latency |             asc_sort_geonameid |    173.218 |     183.254 |  10.0356 |      ms |
|                                      100th percentile latency |             asc_sort_geonameid |    176.296 |     183.561 |  7.26494 |      ms |
|                                  50th percentile service time |             asc_sort_geonameid |    109.904 |     159.712 |  49.8083 |      ms |
|                                  90th percentile service time |             asc_sort_geonameid |    158.694 |     178.658 |   19.964 |      ms |
|                                  99th percentile service time |             asc_sort_geonameid |    172.606 |     182.646 |  10.0399 |      ms |
|                                 100th percentile service time |             asc_sort_geonameid |    175.954 |     183.161 |  7.20744 |      ms |
|                                                    error rate |             asc_sort_geonameid |          0 |           0 |        0 |       % |

and

|                                                        Metric |                           Task |   Baseline |   Contender |     Diff |    Unit |
|--------------------------------------------------------------:|-------------------------------:|-----------:|------------:|---------:|--------:|
|                    Cumulative indexing time of primary shards |                                |    50.4493 |      50.151 | -0.29828 |     min |
|             Min cumulative indexing time across primary shard |                                |    10.0005 |     9.91118 | -0.08927 |     min |
|          Median cumulative indexing time across primary shard |                                |    10.0905 |     10.0258 | -0.06468 |     min |
|             Max cumulative indexing time across primary shard |                                |    10.1799 |     10.1354 | -0.04458 |     min |
|           Cumulative indexing throttle time of primary shards |                                |          0 |           0 |        0 |     min |
|    Min cumulative indexing throttle time across primary shard |                                |          0 |           0 |        0 |     min |
| Median cumulative indexing throttle time across primary shard |                                |          0 |           0 |        0 |     min |
|    Max cumulative indexing throttle time across primary shard |                                |          0 |           0 |        0 |     min |
|                       Cumulative merge time of primary shards |                                |    21.9088 |      23.582 |  1.67318 |     min |
|                      Cumulative merge count of primary shards |                                |         65 |          68 |        3 |         |
|                Min cumulative merge time across primary shard |                                |     3.5424 |     3.49028 | -0.05212 |     min |
|             Median cumulative merge time across primary shard |                                |    4.49383 |     4.95863 |   0.4648 |     min |
|                Max cumulative merge time across primary shard |                                |    4.72087 |     5.29753 |  0.57667 |     min |
|              Cumulative merge throttle time of primary shards |                                |    3.49092 |     4.24625 |  0.75533 |     min |
|       Min cumulative merge throttle time across primary shard |                                |   0.457217 |    0.320933 | -0.13628 |     min |
|    Median cumulative merge throttle time across primary shard |                                |   0.652417 |     0.91755 |  0.26513 |     min |
|       Max cumulative merge throttle time across primary shard |                                |   0.901267 |     1.20128 |  0.30002 |     min |
|                     Cumulative refresh time of primary shards |                                |     7.0937 |     7.05507 | -0.03863 |     min |
|                    Cumulative refresh count of primary shards |                                |        279 |         281 |        2 |         |
|              Min cumulative refresh time across primary shard |                                |    1.36415 |     1.38902 |  0.02487 |     min |
|           Median cumulative refresh time across primary shard |                                |    1.44012 |      1.4124 | -0.02772 |     min |
|              Max cumulative refresh time across primary shard |                                |    1.46268 |     1.42743 | -0.03525 |     min |
|                       Cumulative flush time of primary shards |                                |     0.2755 |    0.242467 | -0.03303 |     min |
|                      Cumulative flush count of primary shards |                                |         10 |          10 |        0 |         |
|                Min cumulative flush time across primary shard |                                |  0.0438833 |   0.0444167 |  0.00053 |     min |
|             Median cumulative flush time across primary shard |                                |  0.0529667 |      0.0502 | -0.00277 |     min |
|                Max cumulative flush time across primary shard |                                |  0.0710167 |   0.0516333 | -0.01938 |     min |
|                                            Total Young Gen GC |                                |     28.381 |      31.629 |    3.248 |       s |
|                                              Total Old Gen GC |                                |          0 |           0 |        0 |       s |
|                                                    Store size |                                |    3.06498 |      3.0625 | -0.00247 |      GB |
|                                                 Translog size |                                |    2.79226 |     2.79327 |  0.00101 |      GB |
|                                                    Index size |                                |    5.85725 |     5.85578 | -0.00147 |      GB |
|                                                 Total written |                                |    16.3191 |      16.337 |  0.01789 |      GB |
|                                        Heap used for segments |                                |    4.57981 |     4.52945 | -0.05035 |      MB |
|                                      Heap used for doc values |                                |  0.0425911 |   0.0613441 |  0.01875 |      MB |
|                                           Heap used for terms |                                |    3.41845 |     3.35747 | -0.06098 |      MB |
|                                           Heap used for norms |                                |   0.074707 |   0.0679932 | -0.00671 |      MB |
|                                          Heap used for points |                                |   0.252747 |    0.253653 |  0.00091 |      MB |
|                                   Heap used for stored fields |                                |   0.791313 |    0.788994 | -0.00232 |      MB |
|                                                 Segment count |                                |         97 |          89 |       -8 |         |
|                                                Min Throughput |                   index-append |    26625.5 |     25867.2 | -758.275 |  docs/s |
|                                             Median Throughput |                   index-append |      27290 |     27401.9 |   111.93 |  docs/s |
|                                                Max Throughput |                   index-append |      27612 |       27989 |  377.009 |  docs/s |
|                                       50th percentile latency |                   index-append |    1231.29 |     1184.97 | -46.3216 |      ms |
|                                       90th percentile latency |                   index-append |    2213.74 |     2167.57 | -46.1691 |      ms |
|                                       99th percentile latency |                   index-append |    3144.19 |     3238.65 |  94.4656 |      ms |
|                                     99.9th percentile latency |                   index-append |    3830.78 |     3540.72 | -290.065 |      ms |
|                                      100th percentile latency |                   index-append |    4233.32 |     3783.64 | -449.677 |      ms |
|                                  50th percentile service time |                   index-append |    1231.29 |     1184.97 | -46.3216 |      ms |
|                                  90th percentile service time |                   index-append |    2213.74 |     2167.57 | -46.1691 |      ms |
|                                  99th percentile service time |                   index-append |    3144.19 |     3238.65 |  94.4656 |      ms |
|                                99.9th percentile service time |                   index-append |    3830.78 |     3540.72 | -290.065 |      ms |
|                                 100th percentile service time |                   index-append |    4233.32 |     3783.64 | -449.677 |      ms |
|                                                    error rate |                   index-append |          0 |           0 |        0 |       % |
|                                                Min Throughput |                    index-stats |    84.5559 |     90.0028 |  5.44689 |   ops/s |
|                                             Median Throughput |                    index-stats |     89.816 |     90.0452 |  0.22926 |   ops/s |
|                                                Max Throughput |                    index-stats |    90.0365 |     90.0742 |  0.03764 |   ops/s |
|                                       50th percentile latency |                    index-stats |    48.7618 |     5.54683 |  -43.215 |      ms |
|                                       90th percentile latency |                    index-stats |     435.39 |     7.07354 | -428.317 |      ms |
|                                       99th percentile latency |                    index-stats |    622.172 |     15.7093 | -606.463 |      ms |
|                                     99.9th percentile latency |                    index-stats |    636.124 |     28.0683 | -608.055 |      ms |
|                                      100th percentile latency |                    index-stats |    640.824 |     32.1345 | -608.689 |      ms |
|                                  50th percentile service time |                    index-stats |      10.25 |     5.43003 | -4.81998 |      ms |
|                                  90th percentile service time |                    index-stats |    12.0977 |     6.93253 | -5.16519 |      ms |
|                                  99th percentile service time |                    index-stats |    16.2701 |     11.2574 | -5.01266 |      ms |
|                                99.9th percentile service time |                    index-stats |    60.7318 |     17.3173 | -43.4145 |      ms |
|                                 100th percentile service time |                    index-stats |    65.0757 |     25.8236 |  -39.252 |      ms |
|                                                    error rate |                    index-stats |          0 |           0 |        0 |       % |
|                                                Min Throughput |                     node-stats |    88.0707 |     90.0439 |   1.9732 |   ops/s |
|                                             Median Throughput |                     node-stats |     89.977 |     90.0794 |  0.10239 |   ops/s |
|                                                Max Throughput |                     node-stats |    90.0368 |     90.2792 |  0.24245 |   ops/s |
|                                       50th percentile latency |                     node-stats |      12.34 |     5.11534 | -7.22464 |      ms |
|                                       90th percentile latency |                     node-stats |    93.6032 |     6.84749 | -86.7558 |      ms |
|                                       99th percentile latency |                     node-stats |    123.351 |     17.8996 | -105.451 |      ms |
|                                     99.9th percentile latency |                     node-stats |     126.52 |     29.3618 | -97.1579 |      ms |
|                                      100th percentile latency |                     node-stats |    127.534 |     30.1183 | -97.4157 |      ms |
|                                  50th percentile service time |                     node-stats |    8.97017 |     5.01688 | -3.95329 |      ms |
|                                  90th percentile service time |                     node-stats |    12.3888 |     6.65851 | -5.73034 |      ms |
|                                  99th percentile service time |                     node-stats |    19.0498 |     9.81334 | -9.23648 |      ms |
|                                99.9th percentile service time |                     node-stats |    53.0479 |      20.987 | -32.0609 |      ms |
|                                 100th percentile service time |                     node-stats |    90.9509 |     22.4959 | -68.4551 |      ms |
|                                                    error rate |                     node-stats |          0 |           0 |        0 |       % |
|                                                Min Throughput |                        default |    50.0263 |     49.9944 | -0.03191 |   ops/s |
|                                             Median Throughput |                        default |    50.0408 |     50.0434 |  0.00265 |   ops/s |
|                                                Max Throughput |                        default |    50.0888 |     50.0912 |  0.00235 |   ops/s |
|                                       50th percentile latency |                        default |     2.7486 |     4.97986 |  2.23126 |      ms |
|                                       90th percentile latency |                        default |    5.37125 |     5.56898 |  0.19773 |      ms |
|                                       99th percentile latency |                        default |     9.3967 |     15.9235 |  6.52676 |      ms |
|                                     99.9th percentile latency |                        default |    17.6746 |     27.1842 |  9.50954 |      ms |
|                                      100th percentile latency |                        default |    18.6401 |     27.2754 |  8.63531 |      ms |
|                                  50th percentile service time |                        default |    2.58555 |     4.75953 |  2.17398 |      ms |
|                                  90th percentile service time |                        default |    5.22332 |     5.30763 |  0.08431 |      ms |
|                                  99th percentile service time |                        default |    9.13552 |     15.0452 |   5.9097 |      ms |
|                                99.9th percentile service time |                        default |    17.5686 |     22.9773 |  5.40865 |      ms |
|                                 100th percentile service time |                        default |    18.5007 |     26.8544 |  8.35375 |      ms |
|                                                    error rate |                        default |          0 |           0 |        0 |       % |
|                                                Min Throughput |                           term |    199.461 |     200.043 |  0.58205 |   ops/s |
|                                             Median Throughput |                           term |    200.083 |     200.116 |  0.03249 |   ops/s |
|                                                Max Throughput |                           term |     200.25 |     200.183 | -0.06756 |   ops/s |
|                                       50th percentile latency |                           term |    2.08322 |     2.58205 |  0.49883 |      ms |
|                                       90th percentile latency |                           term |    3.83981 |     3.71407 | -0.12574 |      ms |
|                                       99th percentile latency |                           term |    7.18497 |     69.4424 |  62.2574 |      ms |
|                                     99.9th percentile latency |                           term |    17.0328 |     85.4455 |  68.4127 |      ms |
|                                      100th percentile latency |                           term |    18.5374 |     86.3591 |  67.8217 |      ms |
|                                  50th percentile service time |                           term |    1.86892 |      2.0511 |  0.18218 |      ms |
|                                  90th percentile service time |                           term |    3.67528 |     3.50135 | -0.17393 |      ms |
|                                  99th percentile service time |                           term |     4.5568 |     4.37945 | -0.17735 |      ms |
|                                99.9th percentile service time |                           term |    12.5657 |     8.60522 | -3.96052 |      ms |
|                                 100th percentile service time |                           term |    18.3605 |     86.2002 |  67.8397 |      ms |
|                                                    error rate |                           term |          0 |           0 |        0 |       % |
|                                                Min Throughput |                         phrase |    200.051 |     196.823 | -3.22815 |   ops/s |
|                                             Median Throughput |                         phrase |     200.09 |     200.097 |  0.00704 |   ops/s |
|                                                Max Throughput |                         phrase |    200.102 |     200.221 |  0.11922 |   ops/s |
|                                       50th percentile latency |                         phrase |    2.31168 |     1.96077 | -0.35091 |      ms |
|                                       90th percentile latency |                         phrase |    4.37213 |      4.6121 |  0.23997 |      ms |
|                                       99th percentile latency |                         phrase |    11.2793 |     66.5007 |  55.2214 |      ms |
|                                     99.9th percentile latency |                         phrase |    17.5384 |     70.1328 |  52.5944 |      ms |
|                                      100th percentile latency |                         phrase |    17.6435 |     70.2272 |  52.5837 |      ms |
|                                  50th percentile service time |                         phrase |    2.15025 |     1.79972 | -0.35053 |      ms |
|                                  90th percentile service time |                         phrase |    3.71595 |     4.23069 |  0.51474 |      ms |
|                                  99th percentile service time |                         phrase |     5.8218 |     5.40072 | -0.42108 |      ms |
|                                99.9th percentile service time |                         phrase |    11.1183 |     19.5149 |  8.39657 |      ms |
|                                 100th percentile service time |                         phrase |    16.7769 |     68.5219 |  51.7449 |      ms |
|                                                    error rate |                         phrase |          0 |           0 |        0 |       % |
|                                                Min Throughput |           country_agg_uncached |    2.52641 |     2.41644 | -0.10997 |   ops/s |
|                                             Median Throughput |           country_agg_uncached |     2.5412 |     2.42797 | -0.11323 |   ops/s |
|                                                Max Throughput |           country_agg_uncached |    2.55501 |      2.4399 | -0.11511 |   ops/s |
|                                       50th percentile latency |           country_agg_uncached |    36213.4 |     40811.3 |   4597.9 |      ms |
|                                       90th percentile latency |           country_agg_uncached |    41321.9 |     46648.3 |  5326.41 |      ms |
|                                       99th percentile latency |           country_agg_uncached |    42567.8 |     48140.2 |  5572.42 |      ms |
|                                      100th percentile latency |           country_agg_uncached |    42694.7 |     48317.5 |   5622.8 |      ms |
|                                  50th percentile service time |           country_agg_uncached |    378.755 |     397.783 |   19.028 |      ms |
|                                  90th percentile service time |           country_agg_uncached |    397.009 |     428.122 |  31.1132 |      ms |
|                                  99th percentile service time |           country_agg_uncached |    459.872 |      480.95 |   21.077 |      ms |
|                                 100th percentile service time |           country_agg_uncached |    466.295 |     494.329 |  28.0344 |      ms |
|                                                    error rate |           country_agg_uncached |          0 |           0 |        0 |       % |
|                                                Min Throughput |             country_agg_cached |    100.041 |      100.04 | -0.00123 |   ops/s |
|                                             Median Throughput |             country_agg_cached |    100.067 |     100.068 |  0.00037 |   ops/s |
|                                                Max Throughput |             country_agg_cached |    100.124 |     100.161 |  0.03704 |   ops/s |
|                                       50th percentile latency |             country_agg_cached |    4.47077 |     4.45387 | -0.01689 |      ms |
|                                       90th percentile latency |             country_agg_cached |    4.89488 |     4.75793 | -0.13694 |      ms |
|                                       99th percentile latency |             country_agg_cached |    8.39644 |     10.6241 |  2.22768 |      ms |
|                                     99.9th percentile latency |             country_agg_cached |    14.1659 |     17.3015 |  3.13562 |      ms |
|                                      100th percentile latency |             country_agg_cached |    14.3136 |     19.5656 |  5.25194 |      ms |
|                                  50th percentile service time |             country_agg_cached |    4.30081 |     4.27676 | -0.02405 |      ms |
|                                  90th percentile service time |             country_agg_cached |    4.66843 |     4.51841 | -0.15002 |      ms |
|                                  99th percentile service time |             country_agg_cached |    7.81247 |     7.60537 |  -0.2071 |      ms |
|                                99.9th percentile service time |             country_agg_cached |    12.4717 |     17.1191 |  4.64734 |      ms |
|                                 100th percentile service time |             country_agg_cached |    13.9833 |     19.1943 |  5.21104 |      ms |
|                                                    error rate |             country_agg_cached |          0 |           0 |        0 |       % |
|                                                Min Throughput |                         scroll |    20.0416 |     20.0393 | -0.00231 | pages/s |
|                                             Median Throughput |                         scroll |    20.0549 |     20.0546 |  -0.0003 | pages/s |
|                                                Max Throughput |                         scroll |      20.07 |     20.0681 | -0.00188 | pages/s |
|                                       50th percentile latency |                         scroll |    391.789 |     397.321 |   5.5314 |      ms |
|                                       90th percentile latency |                         scroll |    414.408 |     433.253 |  18.8443 |      ms |
|                                       99th percentile latency |                         scroll |    502.108 |     536.363 |  34.2551 |      ms |
|                                      100th percentile latency |                         scroll |    509.863 |     550.884 |  41.0211 |      ms |
|                                  50th percentile service time |                         scroll |    391.086 |      396.38 |  5.29327 |      ms |
|                                  90th percentile service time |                         scroll |    413.509 |     432.336 |  18.8267 |      ms |
|                                  99th percentile service time |                         scroll |    500.973 |     536.015 |  35.0417 |      ms |
|                                 100th percentile service time |                         scroll |    508.885 |     550.563 |  41.6775 |      ms |
|                                                    error rate |                         scroll |          0 |           0 |        0 |       % |
|                                                Min Throughput |                     expression |    1.42498 |     1.26231 | -0.16267 |   ops/s |
|                                             Median Throughput |                     expression |      1.432 |     1.26727 | -0.16473 |   ops/s |
|                                                Max Throughput |                     expression |    1.43997 |     1.26842 | -0.17155 |   ops/s |
|                                       50th percentile latency |                     expression |    50168.6 |     72909.4 |  22740.9 |      ms |
|                                       90th percentile latency |                     expression |    58963.9 |     85118.4 |  26154.5 |      ms |
|                                       99th percentile latency |                     expression |    60834.6 |     87880.7 |  27046.1 |      ms |
|                                      100th percentile latency |                     expression |    61028.7 |     88140.4 |  27111.8 |      ms |
|                                  50th percentile service time |                     expression |    699.029 |     792.601 |  93.5727 |      ms |
|                                  90th percentile service time |                     expression |    771.864 |     842.425 |  70.5606 |      ms |
|                                  99th percentile service time |                     expression |    826.767 |      894.77 |  68.0024 |      ms |
|                                 100th percentile service time |                     expression |    838.301 |     910.281 |  71.9802 |      ms |
|                                                    error rate |                     expression |          0 |           0 |        0 |       % |
|                                                Min Throughput |                painless_static |   0.958658 |     1.05664 |  0.09799 |   ops/s |
|                                             Median Throughput |                painless_static |   0.959844 |     1.05838 |  0.09854 |   ops/s |
|                                                Max Throughput |                painless_static |   0.961409 |     1.05935 |  0.09794 |   ops/s |
|                                       50th percentile latency |                painless_static |    94769.3 |       70446 | -24323.3 |      ms |
|                                       90th percentile latency |                painless_static |     109405 |     81229.6 | -28175.1 |      ms |
|                                       99th percentile latency |                painless_static |     112666 |     83722.8 | -28943.2 |      ms |
|                                      100th percentile latency |                painless_static |     113047 |     83967.3 | -29079.2 |      ms |
|                                  50th percentile service time |                painless_static |    1028.84 |     931.452 | -97.3918 |      ms |
|                                  90th percentile service time |                painless_static |    1114.86 |     993.803 | -121.059 |      ms |
|                                  99th percentile service time |                painless_static |    1158.41 |     1063.73 | -94.6771 |      ms |
|                                 100th percentile service time |                painless_static |    1169.24 |     1079.89 | -89.3519 |      ms |
|                                                    error rate |                painless_static |          0 |           0 |        0 |       % |
|                                                Min Throughput |               painless_dynamic |     1.0577 |     1.01141 | -0.04629 |   ops/s |
|                                             Median Throughput |               painless_dynamic |    1.05879 |     1.01192 | -0.04687 |   ops/s |
|                                                Max Throughput |               painless_dynamic |    1.05991 |     1.01276 | -0.04715 |   ops/s |
|                                       50th percentile latency |               painless_dynamic |    70150.1 |     81064.3 |  10914.2 |      ms |
|                                       90th percentile latency |               painless_dynamic |    81418.1 |       93994 |    12576 |      ms |
|                                       99th percentile latency |               painless_dynamic |    83912.7 |     96940.6 |  13027.9 |      ms |
|                                      100th percentile latency |               painless_dynamic |    84164.9 |     97228.6 |  13063.6 |      ms |
|                                  50th percentile service time |               painless_dynamic |    927.317 |     985.606 |  58.2894 |      ms |
|                                  90th percentile service time |               painless_dynamic |    1002.98 |     1057.79 |   54.815 |      ms |
|                                  99th percentile service time |               painless_dynamic |    1112.23 |     1132.36 |  20.1259 |      ms |
|                                 100th percentile service time |               painless_dynamic |    1123.05 |     1147.81 |  24.7593 |      ms |
|                                                    error rate |               painless_dynamic |          0 |           0 |        0 |       % |
|                                                Min Throughput | decay_geo_gauss_function_score |    1.00097 |     1.00132 |  0.00034 |   ops/s |
|                                             Median Throughput | decay_geo_gauss_function_score |    1.00165 |      1.0018 |  0.00015 |   ops/s |
|                                                Max Throughput | decay_geo_gauss_function_score |     1.0026 |     1.00269 |    8e-05 |   ops/s |
|                                       50th percentile latency | decay_geo_gauss_function_score |    567.046 |     551.846 | -15.1992 |      ms |
|                                       90th percentile latency | decay_geo_gauss_function_score |    663.957 |     581.037 | -82.9202 |      ms |
|                                       99th percentile latency | decay_geo_gauss_function_score |    741.866 |     660.363 | -81.5027 |      ms |
|                                      100th percentile latency | decay_geo_gauss_function_score |    746.783 |     678.621 | -68.1627 |      ms |
|                                  50th percentile service time | decay_geo_gauss_function_score |    566.576 |     551.196 |   -15.38 |      ms |
|                                  90th percentile service time | decay_geo_gauss_function_score |    663.469 |     580.447 |  -83.022 |      ms |
|                                  99th percentile service time | decay_geo_gauss_function_score |    741.492 |     659.825 | -81.6669 |      ms |
|                                 100th percentile service time | decay_geo_gauss_function_score |    746.366 |     678.101 | -68.2649 |      ms |
|                                                    error rate | decay_geo_gauss_function_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |   decay_geo_gauss_script_score |    1.00024 |     1.00077 |  0.00053 |   ops/s |
|                                             Median Throughput |   decay_geo_gauss_script_score |    1.00123 |     1.00137 |  0.00014 |   ops/s |
|                                                Max Throughput |   decay_geo_gauss_script_score |    1.00194 |     1.00226 |  0.00032 |   ops/s |
|                                       50th percentile latency |   decay_geo_gauss_script_score |    680.176 |     650.359 | -29.8169 |      ms |
|                                       90th percentile latency |   decay_geo_gauss_script_score |    780.527 |     707.153 | -73.3741 |      ms |
|                                       99th percentile latency |   decay_geo_gauss_script_score |    883.367 |     755.547 |  -127.82 |      ms |
|                                      100th percentile latency |   decay_geo_gauss_script_score |    935.202 |     791.134 | -144.068 |      ms |
|                                  50th percentile service time |   decay_geo_gauss_script_score |     679.79 |     649.886 | -29.9038 |      ms |
|                                  90th percentile service time |   decay_geo_gauss_script_score |    780.131 |     706.558 | -73.5731 |      ms |
|                                  99th percentile service time |   decay_geo_gauss_script_score |    883.074 |     755.119 | -127.955 |      ms |
|                                 100th percentile service time |   decay_geo_gauss_script_score |    934.923 |      790.68 | -144.242 |      ms |
|                                                    error rate |   decay_geo_gauss_script_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |     field_value_function_score |    1.50273 |     1.50305 |  0.00032 |   ops/s |
|                                             Median Throughput |     field_value_function_score |    1.50382 |     1.50376 |   -6e-05 |   ops/s |
|                                                Max Throughput |     field_value_function_score |    1.50533 |     1.50523 | -0.00011 |   ops/s |
|                                       50th percentile latency |     field_value_function_score |    256.059 |     257.027 |  0.96819 |      ms |
|                                       90th percentile latency |     field_value_function_score |    277.526 |         273 | -4.52666 |      ms |
|                                       99th percentile latency |     field_value_function_score |    306.771 |     308.211 |  1.43972 |      ms |
|                                      100th percentile latency |     field_value_function_score |    313.519 |     319.641 |  6.12201 |      ms |
|                                  50th percentile service time |     field_value_function_score |    255.369 |      256.47 |  1.10081 |      ms |
|                                  90th percentile service time |     field_value_function_score |     276.86 |     272.477 | -4.38355 |      ms |
|                                  99th percentile service time |     field_value_function_score |    306.235 |     307.745 |  1.51037 |      ms |
|                                 100th percentile service time |     field_value_function_score |    312.998 |      318.93 |  5.93171 |      ms |
|                                                    error rate |     field_value_function_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |       field_value_script_score |    1.50128 |      1.5021 |  0.00083 |   ops/s |
|                                             Median Throughput |       field_value_script_score |    1.50252 |     1.50261 |   0.0001 |   ops/s |
|                                                Max Throughput |       field_value_script_score |    1.50358 |     1.50355 |   -3e-05 |   ops/s |
|                                       50th percentile latency |       field_value_script_score |    377.569 |     380.592 |  3.02361 |      ms |
|                                       90th percentile latency |       field_value_script_score |    403.964 |     394.486 | -9.47893 |      ms |
|                                       99th percentile latency |       field_value_script_score |    506.317 |     425.694 | -80.6232 |      ms |
|                                      100th percentile latency |       field_value_script_score |    509.826 |     445.666 | -64.1593 |      ms |
|                                  50th percentile service time |       field_value_script_score |    377.238 |     380.165 |  2.92623 |      ms |
|                                  90th percentile service time |       field_value_script_score |    403.589 |     394.095 | -9.49411 |      ms |
|                                  99th percentile service time |       field_value_script_score |    506.146 |     425.163 | -80.9836 |      ms |
|                                 100th percentile service time |       field_value_script_score |    509.476 |     445.266 | -64.2098 |      ms |
|                                                    error rate |       field_value_script_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |          random_function_score |    1.49974 |     1.49956 | -0.00018 |   ops/s |
|                                             Median Throughput |          random_function_score |    1.50092 |     1.50087 |   -5e-05 |   ops/s |
|                                                Max Throughput |          random_function_score |    1.50165 |     1.50204 |   0.0004 |   ops/s |
|                                       50th percentile latency |          random_function_score |     567.11 |     576.532 |  9.42197 |      ms |
|                                       90th percentile latency |          random_function_score |    601.348 |     617.666 |  16.3181 |      ms |
|                                       99th percentile latency |          random_function_score |    677.222 |     703.875 |  26.6533 |      ms |
|                                      100th percentile latency |          random_function_score |    690.378 |     710.686 |   20.308 |      ms |
|                                  50th percentile service time |          random_function_score |     566.89 |     576.303 |  9.41376 |      ms |
|                                  90th percentile service time |          random_function_score |    601.178 |     615.683 |  14.5057 |      ms |
|                                  99th percentile service time |          random_function_score |    676.875 |     703.756 |   26.881 |      ms |
|                                 100th percentile service time |          random_function_score |     690.17 |     710.504 |  20.3339 |      ms |
|                                                    error rate |          random_function_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |            random_script_score |    1.49665 |     1.46832 | -0.02832 |   ops/s |
|                                             Median Throughput |            random_script_score |    1.50002 |     1.47751 |  -0.0225 |   ops/s |
|                                                Max Throughput |            random_script_score |    1.50078 |     1.47954 | -0.02124 |   ops/s |
|                                       50th percentile latency |            random_script_score |    652.302 |     3168.86 |  2516.56 |      ms |
|                                       90th percentile latency |            random_script_score |     797.09 |     4758.23 |  3961.14 |      ms |
|                                       99th percentile latency |            random_script_score |    977.764 |     4935.68 |  3957.91 |      ms |
|                                      100th percentile latency |            random_script_score |     980.31 |     4966.73 |  3986.42 |      ms |
|                                  50th percentile service time |            random_script_score |    633.738 |     680.884 |  47.1462 |      ms |
|                                  90th percentile service time |            random_script_score |        698 |     760.011 |  62.0109 |      ms |
|                                  99th percentile service time |            random_script_score |    768.089 |     807.149 |  39.0603 |      ms |
|                                 100th percentile service time |            random_script_score |    787.262 |     814.345 |  27.0831 |      ms |
|                                                    error rate |            random_script_score |          0 |           0 |        0 |       % |
|                                                Min Throughput |                    large_terms |    1.15704 |     1.24952 |  0.09248 |   ops/s |
|                                             Median Throughput |                    large_terms |    1.16474 |     1.25094 |   0.0862 |   ops/s |
|                                                Max Throughput |                    large_terms |    1.16719 |     1.25352 |  0.08633 |   ops/s |
|                                       50th percentile latency |                    large_terms |    48697.3 |       33975 | -14722.3 |      ms |
|                                       90th percentile latency |                    large_terms |    55870.3 |     38776.2 | -17094.1 |      ms |
|                                       99th percentile latency |                    large_terms |    57591.7 |     40016.2 | -17575.5 |      ms |
|                                      100th percentile latency |                    large_terms |    57718.3 |       40090 | -17628.3 |      ms |
|                                  50th percentile service time |                    large_terms |    834.323 |      788.44 | -45.8826 |      ms |
|                                  90th percentile service time |                    large_terms |    902.459 |     831.771 | -70.6887 |      ms |
|                                  99th percentile service time |                    large_terms |    971.217 |     903.091 | -68.1251 |      ms |
|                                 100th percentile service time |                    large_terms |    996.972 |     916.523 | -80.4495 |      ms |
|                                                    error rate |                    large_terms |          0 |           0 |        0 |       % |
|                                                Min Throughput |           large_filtered_terms |    1.14392 |     1.24856 |  0.10463 |   ops/s |
|                                             Median Throughput |           large_filtered_terms |    1.15376 |     1.25013 |  0.09636 |   ops/s |
|                                                Max Throughput |           large_filtered_terms |    1.15749 |     1.25145 |  0.09396 |   ops/s |
|                                       50th percentile latency |           large_filtered_terms |    52335.4 |     33953.8 | -18381.7 |      ms |
|                                       90th percentile latency |           large_filtered_terms |    58366.4 |     39327.4 |   -19039 |      ms |
|                                       99th percentile latency |           large_filtered_terms |    59724.1 |     40512.6 | -19211.6 |      ms |
|                                      100th percentile latency |           large_filtered_terms |    59848.9 |     40609.4 | -19239.5 |      ms |
|                                  50th percentile service time |           large_filtered_terms |    831.582 |     794.409 | -37.1728 |      ms |
|                                  90th percentile service time |           large_filtered_terms |    953.482 |     834.179 | -119.304 |      ms |
|                                  99th percentile service time |           large_filtered_terms |    1169.98 |     925.803 | -244.175 |      ms |
|                                 100th percentile service time |           large_filtered_terms |    1311.47 |     929.117 |  -382.35 |      ms |
|                                                    error rate |           large_filtered_terms |          0 |           0 |        0 |       % |
|                                                Min Throughput |         large_prohibited_terms |     1.2657 |     1.28223 |  0.01653 |   ops/s |
|                                             Median Throughput |         large_prohibited_terms |    1.26799 |     1.28334 |  0.01535 |   ops/s |
|                                                Max Throughput |         large_prohibited_terms |    1.27034 |      1.2853 |  0.01496 |   ops/s |
|                                       50th percentile latency |         large_prohibited_terms |    31242.3 |     28920.2 | -2322.15 |      ms |
|                                       90th percentile latency |         large_prohibited_terms |    35768.5 |     33305.2 | -2463.23 |      ms |
|                                       99th percentile latency |         large_prohibited_terms |    36714.6 |     34281.6 | -2432.96 |      ms |
|                                      100th percentile latency |         large_prohibited_terms |    36824.6 |     34351.3 | -2473.31 |      ms |
|                                  50th percentile service time |         large_prohibited_terms |    779.247 |     768.998 |  -10.249 |      ms |
|                                  90th percentile service time |         large_prohibited_terms |    830.944 |     835.566 |  4.62204 |      ms |
|                                  99th percentile service time |         large_prohibited_terms |    870.747 |     939.652 |  68.9049 |      ms |
|                                 100th percentile service time |         large_prohibited_terms |     872.07 |     978.602 |  106.532 |      ms |
|                                                    error rate |         large_prohibited_terms |          0 |           0 |        0 |       % |
|                                                Min Throughput |           desc_sort_population |    1.50426 |     1.50277 | -0.00148 |   ops/s |
|                                             Median Throughput |           desc_sort_population |     1.5051 |     1.50444 | -0.00067 |   ops/s |
|                                                Max Throughput |           desc_sort_population |     1.5064 |     1.50616 | -0.00024 |   ops/s |
|                                       50th percentile latency |           desc_sort_population |    96.1945 |     185.206 |  89.0117 |      ms |
|                                       90th percentile latency |           desc_sort_population |    152.595 |     201.826 |  49.2311 |      ms |
|                                       99th percentile latency |           desc_sort_population |    167.679 |     217.945 |  50.2665 |      ms |
|                                      100th percentile latency |           desc_sort_population |    168.981 |     314.665 |  145.684 |      ms |
|                                  50th percentile service time |           desc_sort_population |    95.6474 |     184.624 |  88.9767 |      ms |
|                                  90th percentile service time |           desc_sort_population |     151.78 |     201.089 |  49.3093 |      ms |
|                                  99th percentile service time |           desc_sort_population |    167.037 |     217.354 |  50.3177 |      ms |
|                                 100th percentile service time |           desc_sort_population |    168.397 |     314.073 |  145.676 |      ms |
|                                                    error rate |           desc_sort_population |          0 |           0 |        0 |       % |
|                                                Min Throughput |            asc_sort_population |    1.50376 |     1.50355 | -0.00021 |   ops/s |
|                                             Median Throughput |            asc_sort_population |    1.50513 |     1.50488 | -0.00026 |   ops/s |
|                                                Max Throughput |            asc_sort_population |    1.50657 |     1.50617 |  -0.0004 |   ops/s |
|                                       50th percentile latency |            asc_sort_population |    99.1451 |     128.042 |  28.8969 |      ms |
|                                       90th percentile latency |            asc_sort_population |    145.021 |     201.802 |  56.7814 |      ms |
|                                       99th percentile latency |            asc_sort_population |    160.943 |     214.492 |  53.5493 |      ms |
|                                      100th percentile latency |            asc_sort_population |    166.575 |     262.157 |  95.5816 |      ms |
|                                  50th percentile service time |            asc_sort_population |    98.5376 |     127.402 |  28.8642 |      ms |
|                                  90th percentile service time |            asc_sort_population |    144.425 |     201.331 |  56.9055 |      ms |
|                                  99th percentile service time |            asc_sort_population |    160.288 |     213.856 |  53.5677 |      ms |
|                                 100th percentile service time |            asc_sort_population |    165.994 |     261.321 |  95.3268 |      ms |
|                                                    error rate |            asc_sort_population |          0 |           0 |        0 |       % |
|                                                Min Throughput |            desc_sort_geonameid |     1.5042 |     1.50322 | -0.00098 |   ops/s |
|                                             Median Throughput |            desc_sort_geonameid |    1.50507 |     1.50449 | -0.00059 |   ops/s |
|                                                Max Throughput |            desc_sort_geonameid |    1.50644 |     1.50611 | -0.00033 |   ops/s |
|                                       50th percentile latency |            desc_sort_geonameid |    93.0311 |      174.92 |  81.8884 |      ms |
|                                       90th percentile latency |            desc_sort_geonameid |    106.298 |     200.426 |  94.1278 |      ms |
|                                       99th percentile latency |            desc_sort_geonameid |    152.248 |     211.773 |  59.5253 |      ms |
|                                      100th percentile latency |            desc_sort_geonameid |    158.939 |     262.433 |  103.494 |      ms |
|                                  50th percentile service time |            desc_sort_geonameid |    92.3703 |     174.174 |  81.8033 |      ms |
|                                  90th percentile service time |            desc_sort_geonameid |    105.871 |     199.756 |  93.8851 |      ms |
|                                  99th percentile service time |            desc_sort_geonameid |    151.592 |      211.17 |  59.5775 |      ms |
|                                 100th percentile service time |            desc_sort_geonameid |    158.259 |     261.866 |  103.608 |      ms |
|                                                    error rate |            desc_sort_geonameid |          0 |           0 |        0 |       % |
|                                                Min Throughput |             asc_sort_geonameid |    1.50392 |      1.5033 | -0.00062 |   ops/s |
|                                             Median Throughput |             asc_sort_geonameid |    1.50518 |     1.50449 | -0.00069 |   ops/s |
|                                                Max Throughput |             asc_sort_geonameid |     1.5066 |      1.5064 |  -0.0002 |   ops/s |
|                                       50th percentile latency |             asc_sort_geonameid |    93.1485 |     186.124 |   92.975 |      ms |
|                                       90th percentile latency |             asc_sort_geonameid |    111.555 |     199.478 |  87.9231 |      ms |
|                                       99th percentile latency |             asc_sort_geonameid |    143.057 |     237.936 |  94.8791 |      ms |
|                                      100th percentile latency |             asc_sort_geonameid |    190.952 |     302.239 |  111.286 |      ms |
|                                  50th percentile service time |             asc_sort_geonameid |    92.3511 |     185.416 |  93.0652 |      ms |
|                                  90th percentile service time |             asc_sort_geonameid |    111.094 |     198.636 |  87.5416 |      ms |
|                                  99th percentile service time |             asc_sort_geonameid |    142.375 |     237.291 |  94.9155 |      ms |
|                                 100th percentile service time |             asc_sort_geonameid |    190.289 |     301.655 |  111.366 |      ms |
|                                                    error rate |             asc_sort_geonameid |          0 |           0 |        0 |       % |

@henningandersen
Copy link
Contributor Author

@henningandersen henningandersen commented Sep 7, 2019

@dliappis here are the results of comparing G1 using original (baseline) to new settings (contender) when using the PMC rally track:

esrally --track=pmc --distribution-version=7.3.0 --on-error=abort --car="defaults,g1gc" --track-params="ingest_percentage:40"

I ran the test with both settings twice and did pair-wise comparison.

Total runtimes were:

baseline 1: 891s
baseline 2: 908s
contender 1: 906s
contender 2: 890s

|                                                        Metric |                          Task |   Baseline |   Contender |     Diff |    Unit |
|--------------------------------------------------------------:|------------------------------:|-----------:|------------:|---------:|--------:|
|                    Cumulative indexing time of primary shards |                               |    44.4997 |     46.0639 |  1.56418 |     min |
|             Min cumulative indexing time across primary shard |                               |     8.5952 |     9.16972 |  0.57452 |     min |
|          Median cumulative indexing time across primary shard |                               |    8.80395 |     9.20665 |   0.4027 |     min |
|             Max cumulative indexing time across primary shard |                               |     9.2006 |      9.2927 |   0.0921 |     min |
|           Cumulative indexing throttle time of primary shards |                               |          0 |           0 |        0 |     min |
|    Min cumulative indexing throttle time across primary shard |                               |          0 |           0 |        0 |     min |
| Median cumulative indexing throttle time across primary shard |                               |          0 |           0 |        0 |     min |
|    Max cumulative indexing throttle time across primary shard |                               |          0 |           0 |        0 |     min |
|                       Cumulative merge time of primary shards |                               |    28.3498 |     28.7348 |  0.38497 |     min |
|                      Cumulative merge count of primary shards |                               |         75 |          77 |        2 |         |
|                Min cumulative merge time across primary shard |                               |    4.19377 |     4.27847 |   0.0847 |     min |
|             Median cumulative merge time across primary shard |                               |    5.90398 |     6.13612 |  0.23213 |     min |
|                Max cumulative merge time across primary shard |                               |    6.30027 |     6.78793 |  0.48767 |     min |
|              Cumulative merge throttle time of primary shards |                               |    17.0628 |     16.2842 | -0.77865 |     min |
|       Min cumulative merge throttle time across primary shard |                               |    2.25978 |     2.12338 |  -0.1364 |     min |
|    Median cumulative merge throttle time across primary shard |                               |    3.54898 |     3.48705 | -0.06193 |     min |
|       Max cumulative merge throttle time across primary shard |                               |     3.9604 |      4.2761 |   0.3157 |     min |
|                     Cumulative refresh time of primary shards |                               |    6.06422 |     6.95675 |  0.89253 |     min |
|                    Cumulative refresh count of primary shards |                               |        329 |         327 |       -2 |         |
|              Min cumulative refresh time across primary shard |                               |    1.19792 |     1.28045 |  0.08253 |     min |
|           Median cumulative refresh time across primary shard |                               |     1.2071 |     1.40897 |  0.20187 |     min |
|              Max cumulative refresh time across primary shard |                               |     1.2419 |      1.4486 |   0.2067 |     min |
|                       Cumulative flush time of primary shards |                               |   0.638333 |    0.670667 |  0.03233 |     min |
|                      Cumulative flush count of primary shards |                               |         20 |          20 |        0 |         |
|                Min cumulative flush time across primary shard |                               |  0.0936333 |      0.0875 | -0.00613 |     min |
|             Median cumulative flush time across primary shard |                               |     0.1168 |    0.129117 |  0.01232 |     min |
|                Max cumulative flush time across primary shard |                               |   0.167967 |    0.196783 |  0.02882 |     min |
|                                            Total Young Gen GC |                               |      7.448 |      11.112 |    3.664 |       s |
|                                              Total Old Gen GC |                               |          0 |           0 |        0 |       s |
|                                                    Store size |                               |    8.52246 |     8.52703 |  0.00457 |      GB |
|                                                 Translog size |                               |    2.68865 |     2.61248 | -0.07617 |      GB |
|                                                    Index size |                               |    11.2111 |     11.1395 |  -0.0716 |      GB |
|                                                 Total written |                               |     42.287 |     42.1374 | -0.14964 |      GB |
|                                        Heap used for segments |                               |    1.22298 |     1.20845 | -0.01453 |      MB |
|                                      Heap used for doc values |                               |   0.118912 |    0.138092 |  0.01918 |      MB |
|                                           Heap used for terms |                               |   0.274154 |     0.24911 | -0.02504 |      MB |
|                                           Heap used for norms |                               |  0.0366211 |    0.032959 | -0.00366 |      MB |
|                                          Heap used for points |                               | 0.00389957 |  0.00382328 |   -8e-05 |      MB |
|                                   Heap used for stored fields |                               |   0.789398 |     0.78447 | -0.00493 |      MB |
|                                                 Segment count |                               |        120 |         108 |      -12 |         |
|                                                Min Throughput |                  index-append |    652.233 |      627.49 | -24.7437 |  docs/s |
|                                             Median Throughput |                  index-append |    661.413 |     637.833 | -23.5795 |  docs/s |
|                                                Max Throughput |                  index-append |    668.466 |     643.973 | -24.4927 |  docs/s |
|                                       50th percentile latency |                  index-append |    5168.97 |      5578.6 |  409.634 |      ms |
|                                       90th percentile latency |                  index-append |    6975.84 |     7625.53 |  649.692 |      ms |
|                                       99th percentile latency |                  index-append |    8508.63 |     8805.77 |   297.14 |      ms |
|                                      100th percentile latency |                  index-append |    8608.89 |     9002.76 |  393.862 |      ms |
|                                  50th percentile service time |                  index-append |    5168.97 |      5578.6 |  409.634 |      ms |
|                                  90th percentile service time |                  index-append |    6975.84 |     7625.53 |  649.692 |      ms |
|                                  99th percentile service time |                  index-append |    8508.63 |     8805.77 |   297.14 |      ms |
|                                 100th percentile service time |                  index-append |    8608.89 |     9002.76 |  393.862 |      ms |
|                                                    error rate |                  index-append |          0 |           0 |        0 |       % |
|                                                Min Throughput |                       default |    20.0267 |     20.0238 | -0.00282 |   ops/s |
|                                             Median Throughput |                       default |      20.03 |     20.0292 | -0.00082 |   ops/s |
|                                                Max Throughput |                       default |    20.0347 |     20.0344 | -0.00023 |   ops/s |
|                                       50th percentile latency |                       default |    4.95353 |     7.19047 |  2.23694 |      ms |
|                                       90th percentile latency |                       default |    7.38477 |     9.58765 |  2.20289 |      ms |
|                                       99th percentile latency |                       default |    12.6413 |     17.7634 |  5.12209 |      ms |
|                                      100th percentile latency |                       default |    13.4633 |     35.0519 |  21.5886 |      ms |
|                                  50th percentile service time |                       default |    4.71344 |     7.01498 |  2.30154 |      ms |
|                                  90th percentile service time |                       default |     7.1809 |      9.4099 |    2.229 |      ms |
|                                  99th percentile service time |                       default |    12.2731 |     16.2399 |  3.96686 |      ms |
|                                 100th percentile service time |                       default |    13.3153 |     31.9782 |  18.6629 |      ms |
|                                                    error rate |                       default |          0 |           0 |        0 |       % |
|                                                Min Throughput |                          term |    20.0255 |      20.025 | -0.00044 |   ops/s |
|                                             Median Throughput |                          term |    20.0298 |     20.0298 |   -4e-05 |   ops/s |
|                                                Max Throughput |                          term |    20.0344 |     20.0359 |   0.0015 |   ops/s |
|                                       50th percentile latency |                          term |    6.20711 |     6.03664 | -0.17047 |      ms |
|                                       90th percentile latency |                          term |    9.73376 |     7.51154 | -2.22222 |      ms |
|                                       99th percentile latency |                          term |    19.0256 |     10.3053 | -8.72039 |      ms |
|                                      100th percentile latency |                          term |    36.2699 |     18.5719 |  -17.698 |      ms |
|                                  50th percentile service time |                          term |    5.95289 |     5.86639 |  -0.0865 |      ms |
|                                  90th percentile service time |                          term |    9.22959 |     7.41609 | -1.81351 |      ms |
|                                  99th percentile service time |                          term |    15.7453 |     10.1836 | -5.56168 |      ms |
|                                 100th percentile service time |                          term |    25.4176 |     18.3856 |   -7.032 |      ms |
|                                                    error rate |                          term |          0 |           0 |        0 |       % |
|                                                Min Throughput |                        phrase |    20.0248 |     20.0258 |  0.00091 |   ops/s |
|                                             Median Throughput |                        phrase |    20.0288 |     20.0299 |  0.00111 |   ops/s |
|                                                Max Throughput |                        phrase |    20.0359 |     20.0349 | -0.00105 |   ops/s |
|                                       50th percentile latency |                        phrase |    6.08019 |     5.82742 | -0.25277 |      ms |
|                                       90th percentile latency |                        phrase |    8.25385 |       7.591 | -0.66285 |      ms |
|                                       99th percentile latency |                        phrase |    37.1407 |     17.0293 | -20.1114 |      ms |
|                                      100th percentile latency |                        phrase |    71.2224 |      61.677 | -9.54535 |      ms |
|                                  50th percentile service time |                        phrase |    5.90027 |     5.59334 | -0.30693 |      ms |
|                                  90th percentile service time |                        phrase |    7.93482 |     7.36117 | -0.57365 |      ms |
|                                  99th percentile service time |                        phrase |    32.0415 |     15.6444 | -16.3972 |      ms |
|                                 100th percentile service time |                        phrase |    61.8577 |     32.9907 |  -28.867 |      ms |
|                                                    error rate |                        phrase |          0 |           0 |        0 |       % |
|                                                Min Throughput | articles_monthly_agg_uncached |    20.0171 |      20.017 |   -9e-05 |   ops/s |
|                                             Median Throughput | articles_monthly_agg_uncached |      20.02 |     20.0199 |  -0.0001 |   ops/s |
|                                                Max Throughput | articles_monthly_agg_uncached |    20.0349 |     20.0319 |   -0.003 |   ops/s |
|                                       50th percentile latency | articles_monthly_agg_uncached |    21.0327 |     20.6247 | -0.40806 |      ms |
|                                       90th percentile latency | articles_monthly_agg_uncached |    22.4297 |     21.7206 | -0.70903 |      ms |
|                                       99th percentile latency | articles_monthly_agg_uncached |     78.617 |      22.582 |  -56.035 |      ms |
|                                      100th percentile latency | articles_monthly_agg_uncached |    80.7677 |     23.1663 | -57.6014 |      ms |
|                                  50th percentile service time | articles_monthly_agg_uncached |    20.6082 |     20.2418 | -0.36642 |      ms |
|                                  90th percentile service time | articles_monthly_agg_uncached |    21.9549 |     21.3065 | -0.64845 |      ms |
|                                  99th percentile service time | articles_monthly_agg_uncached |    78.2147 |     22.2059 | -56.0089 |      ms |
|                                 100th percentile service time | articles_monthly_agg_uncached |      80.38 |      22.755 | -57.6251 |      ms |
|                                                    error rate | articles_monthly_agg_uncached |          0 |           0 |        0 |       % |
|                                                Min Throughput |   articles_monthly_agg_cached |    20.0252 |     20.0243 | -0.00092 |   ops/s |
|                                             Median Throughput |   articles_monthly_agg_cached |    20.0297 |     20.0293 | -0.00038 |   ops/s |
|                                                Max Throughput |   articles_monthly_agg_cached |    20.0335 |     20.0335 |    1e-05 |   ops/s |
|                                       50th percentile latency |   articles_monthly_agg_cached |    8.01513 |     7.86942 | -0.14571 |      ms |
|                                       90th percentile latency |   articles_monthly_agg_cached |    8.94436 |     8.55231 | -0.39205 |      ms |
|                                       99th percentile latency |   articles_monthly_agg_cached |    34.9649 |     25.6464 | -9.31844 |      ms |
|                                      100th percentile latency |   articles_monthly_agg_cached |    67.1289 |     69.6987 |  2.56988 |      ms |
|                                  50th percentile service time |   articles_monthly_agg_cached |    7.66741 |     7.52789 | -0.13953 |      ms |
|                                  90th percentile service time |   articles_monthly_agg_cached |    8.53701 |     8.04598 | -0.49103 |      ms |
|                                  99th percentile service time |   articles_monthly_agg_cached |      34.67 |     9.56822 | -25.1018 |      ms |
|                                 100th percentile service time |   articles_monthly_agg_cached |    66.7093 |     69.2682 |  2.55888 |      ms |
|                                                    error rate |   articles_monthly_agg_cached |          0 |           0 |        0 |       % |
|                                                Min Throughput |                        scroll |     12.555 |      12.557 |  0.00208 | pages/s |
|                                             Median Throughput |                        scroll |     12.582 |     12.5858 |  0.00386 | pages/s |
|                                                Max Throughput |                        scroll |    12.6638 |      12.668 |  0.00418 | pages/s |
|                                       50th percentile latency |                        scroll |    682.148 |     636.803 | -45.3449 |      ms |
|                                       90th percentile latency |                        scroll |    703.971 |     661.326 | -42.6451 |      ms |
|                                       99th percentile latency |                        scroll |    788.148 |     740.121 | -48.0272 |      ms |
|                                      100th percentile latency |                        scroll |    808.573 |     775.153 | -33.4194 |      ms |
|                                  50th percentile service time |                        scroll |    680.609 |      635.09 | -45.5191 |      ms |
|                                  90th percentile service time |                        scroll |    702.486 |     659.725 |  -42.761 |      ms |
|                                  99th percentile service time |                        scroll |    786.504 |     738.674 | -47.8294 |      ms |
|                                 100th percentile service time |                        scroll |    806.997 |     773.455 | -33.5421 |      ms |
|                                                    error rate |                        scroll |          0 |           0 |        0 |       % |
|                                                        Metric |                          Task |   Baseline |   Contender |     Diff |    Unit |
|--------------------------------------------------------------:|------------------------------:|-----------:|------------:|---------:|--------:|
|                    Cumulative indexing time of primary shards |                               |    46.0019 |     44.9672 | -1.03472 |     min |
|             Min cumulative indexing time across primary shard |                               |    8.97452 |      8.8061 | -0.16842 |     min |
|          Median cumulative indexing time across primary shard |                               |    9.20602 |     9.00915 | -0.19687 |     min |
|             Max cumulative indexing time across primary shard |                               |    9.36133 |      9.1744 | -0.18693 |     min |
|           Cumulative indexing throttle time of primary shards |                               |          0 |           0 |        0 |     min |
|    Min cumulative indexing throttle time across primary shard |                               |          0 |           0 |        0 |     min |
| Median cumulative indexing throttle time across primary shard |                               |          0 |           0 |        0 |     min |
|    Max cumulative indexing throttle time across primary shard |                               |          0 |           0 |        0 |     min |
|                       Cumulative merge time of primary shards |                               |    26.0403 |     29.3706 |  3.33028 |     min |
|                      Cumulative merge count of primary shards |                               |         77 |          77 |        0 |         |
|                Min cumulative merge time across primary shard |                               |    4.01377 |     4.18077 |    0.167 |     min |
|             Median cumulative merge time across primary shard |                               |    4.39615 |     6.38068 |  1.98453 |     min |
|                Max cumulative merge time across primary shard |                               |    7.18653 |     6.73292 | -0.45362 |     min |
|              Cumulative merge throttle time of primary shards |                               |    14.4751 |     17.7771 |  3.30193 |     min |
|       Min cumulative merge throttle time across primary shard |                               |    1.95505 |     2.12692 |  0.17187 |     min |
|    Median cumulative merge throttle time across primary shard |                               |    2.23432 |     3.96917 |  1.73485 |     min |
|       Max cumulative merge throttle time across primary shard |                               |    4.42098 |     4.37123 | -0.04975 |     min |
|                     Cumulative refresh time of primary shards |                               |    6.71507 |     6.48675 | -0.22832 |     min |
|                    Cumulative refresh count of primary shards |                               |        331 |         332 |        1 |         |
|              Min cumulative refresh time across primary shard |                               |    1.23192 |     1.26475 |  0.03283 |     min |
|           Median cumulative refresh time across primary shard |                               |    1.31298 |      1.2908 | -0.02218 |     min |
|              Max cumulative refresh time across primary shard |                               |    1.45567 |     1.32943 | -0.12623 |     min |
|                       Cumulative flush time of primary shards |                               |     0.8317 |    0.610783 | -0.22092 |     min |
|                      Cumulative flush count of primary shards |                               |         20 |          20 |        0 |         |
|                Min cumulative flush time across primary shard |                               |    0.12165 |   0.0902833 | -0.03137 |     min |
|             Median cumulative flush time across primary shard |                               |     0.1862 |    0.127683 | -0.05852 |     min |
|                Max cumulative flush time across primary shard |                               |   0.200067 |    0.145267 |  -0.0548 |     min |
|                                            Total Young Gen GC |                               |      7.261 |      10.039 |    2.778 |       s |
|                                              Total Old Gen GC |                               |          0 |           0 |        0 |       s |
|                                                    Store size |                               |    8.56756 |     8.52641 | -0.04115 |      GB |
|                                                 Translog size |                               |    2.63817 |      2.6979 |  0.05973 |      GB |
|                                                    Index size |                               |    11.2057 |     11.2243 |  0.01858 |      GB |
|                                                 Total written |                               |    39.5773 |       41.73 |  2.15274 |      GB |
|                                        Heap used for segments |                               |    1.17374 |     1.19349 |  0.01975 |      MB |
|                                      Heap used for doc values |                               |   0.100891 |    0.113052 |  0.01216 |      MB |
|                                           Heap used for terms |                               |   0.246739 |    0.254845 |  0.00811 |      MB |
|                                           Heap used for norms |                               |    0.03479 |   0.0335693 | -0.00122 |      MB |
|                                          Heap used for points |                               | 0.00400257 |  0.00386047 | -0.00014 |      MB |
|                                   Heap used for stored fields |                               |   0.787315 |    0.788162 |  0.00085 |      MB |
|                                                 Segment count |                               |        114 |         110 |       -4 |         |
|                                                Min Throughput |                  index-append |    622.698 |     640.213 |  17.5148 |  docs/s |
|                                             Median Throughput |                  index-append |    630.131 |     650.242 |  20.1118 |  docs/s |
|                                                Max Throughput |                  index-append |     640.07 |      658.15 |  18.0807 |  docs/s |
|                                       50th percentile latency |                  index-append |    5322.76 |     5481.36 |  158.596 |      ms |
|                                       90th percentile latency |                  index-append |     7149.8 |     7101.29 | -48.5153 |      ms |
|                                       99th percentile latency |                  index-append |    8803.67 |     8329.55 |  -474.12 |      ms |
|                                      100th percentile latency |                  index-append |    10420.4 |      9226.7 | -1193.67 |      ms |
|                                  50th percentile service time |                  index-append |    5322.76 |     5481.36 |  158.596 |      ms |
|                                  90th percentile service time |                  index-append |     7149.8 |     7101.29 | -48.5153 |      ms |
|                                  99th percentile service time |                  index-append |    8803.67 |     8329.55 |  -474.12 |      ms |
|                                 100th percentile service time |                  index-append |    10420.4 |      9226.7 | -1193.67 |      ms |
|                                                    error rate |                  index-append |          0 |           0 |        0 |       % |
|                                                Min Throughput |                       default |    20.0235 |     20.0259 |  0.00242 |   ops/s |
|                                             Median Throughput |                       default |     20.028 |     20.0271 | -0.00091 |   ops/s |
|                                                Max Throughput |                       default |    20.0351 |     20.0341 | -0.00097 |   ops/s |
|                                       50th percentile latency |                       default |    8.08453 |     6.64832 | -1.43622 |      ms |
|                                       90th percentile latency |                       default |    13.7058 |     11.9562 | -1.74961 |      ms |
|                                       99th percentile latency |                       default |    21.0113 |     131.083 |  110.071 |      ms |
|                                      100th percentile latency |                       default |    23.7928 |     216.835 |  193.042 |      ms |
|                                  50th percentile service time |                       default |    7.89487 |     6.34722 | -1.54765 |      ms |
|                                  90th percentile service time |                       default |    13.5737 |     9.91409 | -3.65961 |      ms |
|                                  99th percentile service time |                       default |    20.6204 |     18.3765 | -2.24395 |      ms |
|                                 100th percentile service time |                       default |     23.642 |     124.852 |   101.21 |      ms |
|                                                    error rate |                       default |          0 |           0 |        0 |       % |
|                                                Min Throughput |                          term |    20.0231 |     20.0231 |    4e-05 |   ops/s |
|                                             Median Throughput |                          term |     20.029 |     20.0291 |    4e-05 |   ops/s |
|                                                Max Throughput |                          term |    20.0337 |     20.0348 |  0.00107 |   ops/s |
|                                       50th percentile latency |                          term |    7.55551 |     6.49031 |  -1.0652 |      ms |
|                                       90th percentile latency |                          term |    10.0462 |     9.71102 | -0.33522 |      ms |
|                                       99th percentile latency |                          term |    40.8672 |     16.4466 | -24.4206 |      ms |
|                                      100th percentile latency |                          term |     83.945 |     33.0841 | -50.8609 |      ms |
|                                  50th percentile service time |                          term |    7.35301 |     6.15263 | -1.20038 |      ms |
|                                  90th percentile service time |                          term |    9.84598 |     9.35866 | -0.48733 |      ms |
|                                  99th percentile service time |                          term |    21.7977 |     16.0618 | -5.73592 |      ms |
|                                 100th percentile service time |                          term |    83.8093 |     27.4565 | -56.3528 |      ms |
|                                                    error rate |                          term |          0 |           0 |        0 |       % |
|                                                Min Throughput |                        phrase |    20.0179 |     20.0228 |  0.00488 |   ops/s |
|                                             Median Throughput |                        phrase |    20.0208 |     20.0303 |  0.00944 |   ops/s |
|                                                Max Throughput |                        phrase |    20.0339 |     20.0352 |  0.00129 |   ops/s |
|                                       50th percentile latency |                        phrase |    18.6428 |     5.80234 | -12.8404 |      ms |
|                                       90th percentile latency |                        phrase |     19.922 |     7.61576 | -12.3062 |      ms |
|                                       99th percentile latency |                        phrase |    23.7369 |      15.497 | -8.23992 |      ms |
|                                      100th percentile latency |                        phrase |    25.1188 |     46.2034 |  21.0846 |      ms |
|                                  50th percentile service time |                        phrase |     18.295 |     5.51482 | -12.7802 |      ms |
|                                  90th percentile service time |                        phrase |    19.5334 |      7.4981 | -12.0353 |      ms |
|                                  99th percentile service time |                        phrase |     23.411 |     14.5757 | -8.83527 |      ms |
|                                 100th percentile service time |                        phrase |    24.9208 |     46.0847 |  21.1639 |      ms |
|                                                    error rate |                        phrase |          0 |           0 |        0 |       % |
|                                                Min Throughput | articles_monthly_agg_uncached |    20.0169 |     19.9877 | -0.02918 |   ops/s |
|                                             Median Throughput | articles_monthly_agg_uncached |    20.0216 |      20.028 |   0.0064 |   ops/s |
|                                                Max Throughput | articles_monthly_agg_uncached |    20.0345 |     20.0353 |  0.00075 |   ops/s |
|                                       50th percentile latency | articles_monthly_agg_uncached |     19.954 |      7.1799 | -12.7741 |      ms |
|                                       90th percentile latency | articles_monthly_agg_uncached |    21.6223 |     10.5411 | -11.0812 |      ms |
|                                       99th percentile latency | articles_monthly_agg_uncached |    47.6041 |      23.413 | -24.1911 |      ms |
|                                      100th percentile latency | articles_monthly_agg_uncached |    78.5041 |     66.6003 | -11.9038 |      ms |
|                                  50th percentile service time | articles_monthly_agg_uncached |    19.5561 |     6.87233 | -12.6838 |      ms |
|                                  90th percentile service time | articles_monthly_agg_uncached |    21.1479 |     10.0751 | -11.0727 |      ms |
|                                  99th percentile service time | articles_monthly_agg_uncached |    22.8028 |     17.0327 | -5.77009 |      ms |
|                                 100th percentile service time | articles_monthly_agg_uncached |    78.3266 |     66.4039 | -11.9227 |      ms |
|                                                    error rate | articles_monthly_agg_uncached |          0 |           0 |        0 |       % |
|                                                Min Throughput |   articles_monthly_agg_cached |    19.9861 |     19.9844 | -0.00166 |   ops/s |
|                                             Median Throughput |   articles_monthly_agg_cached |    20.0266 |     20.0272 |  0.00054 |   ops/s |
|                                                Max Throughput |   articles_monthly_agg_cached |    20.0315 |     20.0354 |  0.00391 |   ops/s |
|                                       50th percentile latency |   articles_monthly_agg_cached |    8.06687 |     7.83586 | -0.23101 |      ms |
|                                       90th percentile latency |   articles_monthly_agg_cached |    9.07538 |     8.33778 | -0.73759 |      ms |
|                                       99th percentile latency |   articles_monthly_agg_cached |    28.2127 |     27.0409 | -1.17185 |      ms |
|                                      100th percentile latency |   articles_monthly_agg_cached |    70.8942 |     70.2328 | -0.66139 |      ms |
|                                  50th percentile service time |   articles_monthly_agg_cached |    7.63757 |      7.4618 | -0.17576 |      ms |
|                                  90th percentile service time |   articles_monthly_agg_cached |    8.65299 |     7.94547 | -0.70752 |      ms |
|                                  99th percentile service time |   articles_monthly_agg_cached |    10.9972 |     9.68823 | -1.30901 |      ms |
|                                 100th percentile service time |   articles_monthly_agg_cached |    70.5053 |     69.8169 |  -0.6884 |      ms |
|                                                    error rate |   articles_monthly_agg_cached |          0 |           0 |        0 |       % |
|                                                Min Throughput |                        scroll |    12.5552 |     12.5528 | -0.00244 | pages/s |
|                                             Median Throughput |                        scroll |    12.5823 |     12.5826 |  0.00032 | pages/s |
|                                                Max Throughput |                        scroll |    12.6651 |     12.6605 |  -0.0046 | pages/s |
|                                       50th percentile latency |                        scroll |    674.972 |     686.816 |  11.8443 |      ms |
|                                       90th percentile latency |                        scroll |    688.706 |     724.174 |  35.4678 |      ms |
|                                       99th percentile latency |                        scroll |    772.996 |     810.814 |  37.8181 |      ms |
|                                      100th percentile latency |                        scroll |     779.58 |       822.8 |  43.2199 |      ms |
|                                  50th percentile service time |                        scroll |    673.316 |     685.149 |  11.8332 |      ms |
|                                  90th percentile service time |                        scroll |    687.026 |     722.544 |  35.5179 |      ms |
|                                  99th percentile service time |                        scroll |    771.335 |     809.179 |  37.8436 |      ms |
|                                 100th percentile service time |                        scroll |    778.117 |     821.147 |  43.0294 |      ms |
|                                                    error rate |                        scroll |          0 |           0 |        0 |       % |

@dliappis
Copy link
Contributor

@dliappis dliappis commented Sep 9, 2019

@henningandersen Thank you for running these. Judging by the names of the operations listed in the results in your last comment, you were using the pmc track as discussed earlier, right?

From the results it looks to me there's very small difference with the IHOP change in the median indexing throughput (~3%) which sounds ok and possibly attributed to run to run variation and increase in time spent in young gen GC.

@henningandersen
Copy link
Contributor Author

@henningandersen henningandersen commented Sep 9, 2019

@dliappis yes, sorry for not including it, this was run against PMC with a 40 percent ingest_percentage, edited my comment above to include this info.

Regarding median indexing throughput, it looks like the four numbers are:

baseline 1: 661.413
baseline 2: 630.131
contender 1: 637.833
contender 2: 650.242

the numbers seem to be equal enough to be considered equal. The cumulative indexing times also add op to near identical numbers.

danielmitterdorfer added a commit to danielmitterdorfer/rally-teams that referenced this issue Sep 12, 2019
With this commit we align the G1-related settings to the new ES defaults
that will be introduced with elastic/elasticsearch#46169
danielmitterdorfer added a commit to elastic/rally-teams that referenced this issue Sep 12, 2019
With this commit we align the G1-related settings to the new ES defaults
that will be introduced with elastic/elasticsearch#46169.

Relates #30
Relates elastic/elasticsearch#46169
danielmitterdorfer added a commit to elastic/rally-teams that referenced this issue Sep 12, 2019
With this commit we align the G1-related settings to the new ES defaults
that will be introduced with elastic/elasticsearch#46169.

Relates #30
Relates elastic/elasticsearch#46169
Copy link
Member

@danielmitterdorfer danielmitterdorfer left a comment

Summary

We see a slight decrease in throughput metrics for a 4GB heap. Overall I'd argue that this change improves the situation especially w.r.t. effectiveness of the real-memory circuit breaker. Thanks for the change. LGTM

Detailed results

I've benchmarked several combinations on our nightly benchmarking environment. Baseline is always G1GC with the previous settings, the contender is G1GC with the settings from this PR. All benchmarks were run on OpenJDK 12 (build 12+32).

nyc_taxis

heap size: 4GB

We see a decrease in median indexing throughput from 79195 docs/s to 77990 docs/s (1.5% decrease).

I also noticed that the range query returns in both configurations with a varying but non-zero error rate (less than 100%) and distance_amount_agg always leads to 100% error rate in both configurations (with CMS this works just fine). Digging further, this is due to the parent circuit breaker tripping.

On the positive side autohisto_agg and date_histogram_agg finish now without any errors and performance is also on par with CMS (the baseline configuration had a 100% error rate).

In the GC logs we see an increase in the number of GC pauses by 55% (from 3183 pauses up to 4946). Total time in GC went up from 37:43 minutes to 43:28 minutes. The maximum pause time went down from 133ms to 90ms.

heap size: 16GB

Median indexing throughput is roughly identical (83576 docs/s for baseline vs. 83841 docs/s with the new configuration). Overall there were no errors returned in any of the configurations. The 99th percentile latency for range drops from 754 ms to 576 ms. Also the 99th percentile latency for autohisto_agg drops from 550ms to 497 ms. The other query latencies are roughly identical.

In the GC logs we see an increase in the number of GC pauses by 27% (from 721 pauses up to 995). Total time in GC went down from 37:59 minutes to 35:56 minutes. The maximum pause time went up from 100ms to 174ms.

pmc

heap size: 4GB

We see a decrease in median indexing throughput from 1272 docs/s to 1246 docs/s (2% decrease). Query latencies are roughly identical but the 99th percentile latency for scroll went up from 1070ms to 1243ms (roughly a 14% decrease).

In the GC logs we see an increase in the number of pauses by 49% (from 560 pauses up to 834). Total time in GC was roughly constant (16:22 minutes before and 16:32 minutes after). The maximum pause time went down from 43ms to 36ms.

@henningandersen
Copy link
Contributor Author

@henningandersen henningandersen commented Sep 23, 2019

Thanks for reviewing @danielmitterdorfer, @jasontedor and @dliappis.

@henningandersen henningandersen merged commit 3074bd1 into elastic:master Sep 23, 2019
8 checks passed
henningandersen added a commit that referenced this issue Sep 23, 2019
G1 GC were setup to use an `InitiatingHeapOccupancyPercent` of 75. This
could leave used memory at a very high level for an extended duration,
triggering the real memory circuit breaker even at low activity levels.
The value is a threshold for old generation usage relative to total heap
size and thus it should leave room for the new generation. Default in
G1 is to allow up to 60 percent for new generation and this could mean that the
threshold was effectively at 135% heap usage. GC would still kick in of course and
eventually enough mixed collections would take place such that adaptive adjustment
of IHOP kicks in.

The JVM has adaptive setting of the IHOP, but this does not kick in
until it has sampled a few collections. A newly started, relatively
quiet server with primarily new generation activity could thus
experience heap above 95% frequently for a duration.

The changes here are two-fold:
1. Use 30% default for IHOP (the JVM default of 45 could still mean
105% heap usage threshold and did not fully ensure not to hit the
circuit breaker with low activity)
2. Set G1ReservePercent=25. This is used by the adaptive IHOP mechanism,
meaning old/mixed GC should kick in no later than at 75% heap. This
ensures IHOP stays compatible with the real memory circuit breaker also
after being adjusted by adaptive IHOP.
henningandersen added a commit that referenced this issue Sep 23, 2019
G1 GC were setup to use an `InitiatingHeapOccupancyPercent` of 75. This
could leave used memory at a very high level for an extended duration,
triggering the real memory circuit breaker even at low activity levels.
The value is a threshold for old generation usage relative to total heap
size and thus it should leave room for the new generation. Default in
G1 is to allow up to 60 percent for new generation and this could mean that the
threshold was effectively at 135% heap usage. GC would still kick in of course and
eventually enough mixed collections would take place such that adaptive adjustment
of IHOP kicks in.

The JVM has adaptive setting of the IHOP, but this does not kick in
until it has sampled a few collections. A newly started, relatively
quiet server with primarily new generation activity could thus
experience heap above 95% frequently for a duration.

The changes here are two-fold:
1. Use 30% default for IHOP (the JVM default of 45 could still mean
105% heap usage threshold and did not fully ensure not to hit the
circuit breaker with low activity)
2. Set G1ReservePercent=25. This is used by the adaptive IHOP mechanism,
meaning old/mixed GC should kick in no later than at 75% heap. This
ensures IHOP stays compatible with the real memory circuit breaker also
after being adjusted by adaptive IHOP.
jkakavas pushed a commit to jkakavas/elasticsearch that referenced this issue Sep 25, 2019
G1 GC were setup to use an `InitiatingHeapOccupancyPercent` of 75. This
could leave used memory at a very high level for an extended duration,
triggering the real memory circuit breaker even at low activity levels.
The value is a threshold for old generation usage relative to total heap
size and thus it should leave room for the new generation. Default in
G1 is to allow up to 60 percent for new generation and this could mean that the
threshold was effectively at 135% heap usage. GC would still kick in of course and
eventually enough mixed collections would take place such that adaptive adjustment
of IHOP kicks in.

The JVM has adaptive setting of the IHOP, but this does not kick in
until it has sampled a few collections. A newly started, relatively
quiet server with primarily new generation activity could thus
experience heap above 95% frequently for a duration.

The changes here are two-fold:
1. Use 30% default for IHOP (the JVM default of 45 could still mean
105% heap usage threshold and did not fully ensure not to hit the
circuit breaker with low activity)
2. Set G1ReservePercent=25. This is used by the adaptive IHOP mechanism,
meaning old/mixed GC should kick in no later than at 75% heap. This
ensures IHOP stays compatible with the real memory circuit breaker also
after being adjusted by adaptive IHOP.
@colings86 colings86 added v7.4.0 and removed v7.4.1 labels Sep 25, 2019
@fhalde
Copy link

@fhalde fhalde commented May 10, 2020

@henningandersen

Does it make sense to have this setting too?

-XX:-G1UseAdaptiveIHOP

basically disable the AdaptiveIHOP feature. Otherwise what's the point of setting IHOP in the first place? I'm debugging a CircuitBreaker issue with my cluster

We do have the settings mentioned by you'll in our cluster & things were fine for quite some time. However the CB tripping is back. We suspect a couple of things and it is totally possible that we are over-allocating & need to tweak something else (we've observed Humongous allocations happening in our logs)

@henningandersen
Copy link
Contributor Author

@henningandersen henningandersen commented May 11, 2020

@fhalde I think not as a general recommendation, but for very specific scenarios it might make sense after careful benchmarking and tuning.

The original IHOP is just that the outset for the value until the adaptive algorithm of G1 kicks in.

I should be interested in seeing more details on this like GC logs and information on the workload. I would like to redirect you to the discuss forums for this investigation, we prefer to use github only for confirmed bugs and enhancements. Feel free to mention me on the discuss posting (HenningAndersen).

@fhalde
Copy link

@fhalde fhalde commented May 11, 2020

Sure @henningandersen
I have a thread already opened for this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Circuit Breakers >enhancement v7.4.0 v7.5.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants