[CI] ApmIT testCapturesTracesForHttpTraffic failing

[DaveCTurner]

Looks like APM wasn't indexing docs within 1 minute - maybe just not long enough or maybe something else is stuck?

Build scan:
https://gradle-enterprise.elastic.co/s/gssyp74wldwds/tests/:qa:apm:integTest/org.elasticsearch.tracing.apm.ApmIT/testCapturesTracesForHttpTraffic

Reproduction line:
./gradlew ':qa:apm:integTest' --tests "org.elasticsearch.tracing.apm.ApmIT.testCapturesTracesForHttpTraffic" -Dtests.seed=28D57BF7CF358A67 -Dtests.locale=es-ES -Dtests.timezone=Pacific/Norfolk -Druntime.java=17

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.tracing.apm.ApmIT&tests.test=testCapturesTracesForHttpTraffic

Failure excerpt:

java.lang.AssertionError: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:49186], URI [/_data_stream/traces-apm-default], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}

  at __randomizedtesting.SeedInfo.seed([28D57BF7CF358A67:AF106D147BC11DD]:0)
  at org.elasticsearch.tracing.apm.ApmIT.performRequestTolerantly(ApmIT.java:168)
  at org.elasticsearch.tracing.apm.ApmIT.lambda$checkTracesDataStream$0(ApmIT.java:57)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1104)
  at org.elasticsearch.tracing.apm.ApmIT.checkTracesDataStream(ApmIT.java:56)
  at org.elasticsearch.tracing.apm.ApmIT.testCapturesTracesForHttpTraffic(ApmIT.java:50)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:49186], URI [/_data_stream/traces-apm-default], status line [HTTP/1.1 404 Not Found]
  {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}

    at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
    at org.elasticsearch.tracing.apm.ApmIT.performRequestTolerantly(ApmIT.java:166)
    at org.elasticsearch.tracing.apm.ApmIT.lambda$checkTracesDataStream$0(ApmIT.java:57)
    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1104)
    at org.elasticsearch.tracing.apm.ApmIT.checkTracesDataStream(ApmIT.java:56)
    at org.elasticsearch.tracing.apm.ApmIT.testCapturesTracesForHttpTraffic(ApmIT.java:50)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:568)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:833)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[joegallo]

I just ran into this one, too. https://gradle-enterprise.elastic.co/s/4762jiblg3goi/tests/:qa:apm:integTest/org.elasticsearch.tracing.apm.ApmIT/testCapturesTracesForHttpTraffic?top-execution=1

[stu-elastic]

Hit this one twice:

• https://gradle-enterprise.elastic.co/s/ic43rgmaja7vc
• https://gradle-enterprise.elastic.co/s/eout3dmrl3uqi

[masseyke]

It's happening a lot today:

• https://gradle-enterprise.elastic.co/s/qnnvmgpkewwk4
• https://gradle-enterprise.elastic.co/s/vcw6n5cga3m42
• https://gradle-enterprise.elastic.co/s/fkdp55cair2lk

[grcevski]

Maybe we should mute, I hit it too: https://gradle-enterprise.elastic.co/s/dvaotnrevetmm/tests/:qa:apm:integTest/org.elasticsearch.tracing.apm.ApmIT/testCapturesTracesForHttpTraffic

[grcevski]

I'm going to assign this bug to me and mute the test in the meantime.

[pugnascotia]

@grcevski I ended up disabling the :qa:apm project entirely in #90291 due to issues with pulling the 8.6.0-SNAPSHOT Kibana image (as in, a build problem means it isn't published yet). Going forwards, we need to work out:

• Whether we need to use the same Kibana and Elastic Agent image versions as ES in these tests, or whether we can use the previous minor. This would protect us from post-release issues to some extend, but potentially complicate the feature freeze / release finalization processes
• Whether we actually need Kibana at all. We need the Elastic Agent as it provides the APM server in these tests.

The origin of the docker-compose in :qa:apm is a generator from (I think) the APM team. I think it might be useful to have Kibana available in case someone wants to go poking into what the test captured, but it isn't strictly required.