[CI] MetricsApmIT testApmIntegration failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #4612 / rhel-7_platform-support-unix
• elasticsearch-periodic #4626 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic #4616 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic #4596 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic #4591 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic #4586 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic #4581 / openjdk23_checkpart1_java-matrix

Reproduction Line:

./gradlew ':test:external-modules:test-apm-integration:javaRestTest' --tests "org.elasticsearch.test.apmintegration.MetricsApmIT.testApmIntegration" -Dtests.seed=2DC38638E8DAD72D -Dtests.locale=th -Dtests.timezone=America/Argentina/San_Juan -Druntime.java=22

Applicable branches:
8.15

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Timeout when waiting for assertions to complete. Remaining assertions to match: {es.test.async_long_counter.total=es.test.async_long_counter.total <1>, es.test.long_gauge.current=es.test.long_gauge.current <1>, es.test.long_counter.total=es.test.long_counter.total a numeric value within <0.001> of <1.0>, es.test.async_double_counter.total=es.test.async_double_counter.total a numeric value within <0.001> of <1.0>, es.test.long_histogram.histogram=es.test.long_histogram.histogram <2>, es.test.double_histogram.histogram=es.test.double_histogram.histogram <2>, es.test.double_gauge.current=es.test.double_gauge.current a numeric value within <0.001> of <1.0>, es.test.double_counter.total=es.test.double_counter.total a numeric value within <0.001> of <1.0>}

Issue Reasons:

• [8.15] 7 failures in test testApmIntegration (1.9% fail rate in 372 executions)
• [8.15] 6 failures in step openjdk23_checkpart1_java-matrix (75.0% fail rate in 8 executions)
• [8.15] 6 failures in pipeline elasticsearch-periodic (46.2% fail rate in 13 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 5 failures in test testApmIntegration (0.7% fail rate in 762 executions)
• [main] 3 failures in step openjdk23_checkpart1_java-matrix (27.3% fail rate in 11 executions)
• [main] 3 failures in pipeline elasticsearch-periodic (18.8% fail rate in 16 executions)

Build Scans:

• elasticsearch-intake #11833 / part1
• elasticsearch-pull-request #37714 / part-1
• elasticsearch-periodic #4502 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic #4487 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic #4482 / openjdk23_checkpart1_java-matrix

[ldematte]

The openjdk23_checkpart1_java-matrix / elasticsearch-periodic are from before the fix (#115194)
The other 2 are new and around the same thing (histogram metrics missing). I'll investigate them more, it might/should be just just a timing issue as they happen quite rarely (apparently) (but with the test muted too early we cannot know!)

[ldematte]

It seems that the new library might on occasion emit histogram values separately - and our assertion does not take that into account.
Definitely a test issue though, I will think how we can change the assertions for histogram (not immediate).