[CI] ShardSearchPhaseAPMMetricsTests testTimeRangeFilterAllResults failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #31118 / part1
• elasticsearch-pull-request #103778 / part-1
• elasticsearch-intake #31115 / part1
• elasticsearch-pull-request #103778 / part-1
• elasticsearch-pull-request #103776 / part-1
• elasticsearch-intake #31114 / part1

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.search.TelemetryMetrics.ShardSearchPhaseAPMMetricsTests.testTimeRangeFilterAllResults" -Dtests.seed=10B5180855995F12 -Dtests.locale=en-IO -Dtests.timezone=Asia/Katmandu -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: expected:<5> but was:<4>

Issue Reasons:

• [main] 2 consecutive failures in test testTimeRangeFilterAllResults
• [main] 3 consecutive failures in step part1
• [main] 3 consecutive failures in pipeline elasticsearch-intake
• [main] 6 failures in test testTimeRangeFilterAllResults (2.7% fail rate in 221 executions)
• [main] 3 failures in step part1 (10.3% fail rate in 29 executions)
• [main] 3 failures in step part-1 (1.9% fail rate in 158 executions)
• [main] 3 failures in pipeline elasticsearch-intake (10.3% fail rate in 29 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.3% fail rate in 156 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in step part1
• [main] 2 consecutive failures in pipeline elasticsearch-intake
• [main] 4 failures in test testTimeRangeFilterAllResults (1.8% fail rate in 221 executions)
• [main] 2 failures in step part1 (7.1% fail rate in 28 executions)
• [main] 2 failures in step part-1 (1.3% fail rate in 158 executions)
• [main] 2 failures in pipeline elasticsearch-intake (7.1% fail rate in 28 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.3% fail rate in 155 executions)

Build Scans:

• elasticsearch-intake #31115 / part1
• elasticsearch-pull-request #103778 / part-1
• elasticsearch-pull-request #103776 / part-1
• elasticsearch-intake #31114 / part1

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[benchaplin]

Looks like this is due to #137196.

[chrisparrinello]

Digging into this, it looks like the assumption made in the can-match phase around all of the shard search requests having the same request attributes is incorrect so the attributes recorded for the metrics are a little non-deterministic depending on which request gets processed first. Even the reproduction line sometimes passes.

I'm looking into understanding why the shard search requests have different attributes. The difference that causes the tests to fail is the lack of the time range search time in some of the requests.

[chrisparrinello]

This appears to be a result of querying an index with empty shards. If a shard is empty, it is completely skipped in the can-match phase. As a result, the search terms are not parsed, we don't get the time range attribute to add to the metrics. If that empty shard happens to be the first shard we process, all of the subsequent metrics for that request will all lack the time range attribute since we only attempt to get the search request attributes one to reduce overhead on gathering these attributes.

I think the empty shards is a fairly edge case and I don't think we want to increase the overhead in the can-match phase to gather attributes for each individual shard. So the fix here is to fix the test case to handle the skipped empty shards.