Failure in HDRPercentilesIT.testScriptSingleValued

[DaveCTurner]

CI Link

https://gradle-enterprise.elastic.co/s/4loz5k2jllcvg

Repro line

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.aggregations.metrics.HDRPercentilesIT.testScriptSingleValued" -Dtests.seed=928B8BBC20C73401 -Dtests.locale=de-CH -Dtests.timezone=America/Argentina/Catamarca -Druntime.java=19

Does it reproduce?

Yes

Applicable branches

#92817 (a small change over main)

Failure history

No response

Failure excerpt

I encountered this a few times in #92817, which is effectively main plus the following patch which should not cause any failures:

diff --git a/test/framework/src/main/java/org/elasticsearch/test/InternalTestCluster.java b/test/framework/src/main/java/org/elasticsearch/test/InternalTestCluster.java
index 18f21cb7368..c51df18675c 100644
--- a/test/framework/src/main/java/org/elasticsearch/test/InternalTestCluster.java
+++ b/test/framework/src/main/java/org/elasticsearch/test/InternalTestCluster.java
@@ -152,7 +152,6 @@ import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.greaterThanOrEqualTo;
 import static org.hamcrest.Matchers.not;
-import static org.hamcrest.Matchers.nullValue;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertThat;
@@ -603,6 +602,8 @@ public final class InternalTestCluster extends TestCluster {
             );
         }

+        builder.put(INITIAL_STATE_TIMEOUT_SETTING.getKey(), "0s");
+
         return builder.build();
     }

@@ -731,14 +732,6 @@ public final class InternalTestCluster extends TestCluster {
         updatedSettings.put("node.name", name);
         updatedSettings.put(NodeEnvironment.NODE_ID_SEED_SETTING.getKey(), seed);

-        if (autoManageMasterNodes) {
-            assertThat(
-                "if master nodes are automatically managed then nodes must complete a join cycle when starting",
-                updatedSettings.get(INITIAL_STATE_TIMEOUT_SETTING.getKey()),
-                nullValue()
-            );
-        }
-
         return updatedSettings.build();
     }

The failure is as follows:

org.elasticsearch.search.aggregations.metrics.HDRPercentilesIT > testScriptSingleValued FAILED	
    java.lang.AssertionError: 	
    Expected: a value less than or equal to <10.00100068359375>	
         but: <10.0068359375> was greater than <10.00100068359375>	
        at __randomizedtesting.SeedInfo.seed([928B8BBC20C73401:2E3A0A78C1851C92]:0)	
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)	
        at org.junit.Assert.assertThat(Assert.java:956)	
        at org.junit.Assert.assertThat(Assert.java:923)	
        at org.elasticsearch.search.aggregations.metrics.HDRPercentilesIT.assertConsistent(HDRPercentilesIT.java:87)	
        at org.elasticsearch.search.aggregations.metrics.HDRPercentilesIT.testScriptSingleValued(HDRPercentilesIT.java:376)	

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[DaveCTurner]

In fact I seem to be getting something similar on other tests in the same suite too - at least:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.aggregations.metrics.HDRPercentilesIT.testSingleValuedFieldPartiallyUnmapped" -Dtests.seed=CC100B3B0FAA4466 -Dtests.locale=lt-LT -Dtests.timezone=America/Cayman -Druntime.java=19

[DaveCTurner]

Oh this failed in main too: https://gradle-enterprise.elastic.co/s/atstclct72y24

[DaveCTurner]

Muted in a2adedf.

[salvatore-campagna]

I investigated this isssue and I think the problem is the following.

When creating a sketch for a HdrHistogram we need to provide the numberOfSignificantValueDigits that is used by the HdrHistogram algorithm to calculate percentiles with a specific precision. Some tests return empty results...I guess this happens because the test uses few documents and some shards have no document indexed. The test generates values for numberOfSignificantValueDigits in the range [0, 5]. So, if a test generates a value that is 4 or 5 it expects a certain precision. Anyway, at reduction time we might merge results from sketches using different precisions. Empty results use sketches with 3 significant digits, non-empty results sketches with the number of digits expected by the test.

As an additional note...testing in the main branch I could reproduce the issue for test testSingleValuedFieldPartiallyUnmapped using the seed from the Gradle reproductioin line and not in other cases.
I guess anyway that the issue here is triggered depending on how documents are distributed across shards.

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.aggregations.metrics.HDRPercentilesIT.testSingleValuedFieldPartiallyUnmapped" -Dtests.seed=CC100B3B0FAA4466 -Dtests.locale=lt-LT -Dtests.timezone=America/Cayman -Druntime.java=19