[CI] SearchServiceSingleNodeTests testBeforeShardLockDuringShardCreate failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #7679 / oraclelinux-8_platform-support-unix
• elasticsearch-pull-request #67921 / part-1
• elasticsearch-pull-request #67912 / part-1
• elasticsearch-pull-request #67912 / part-1
• elasticsearch-pull-request #67909 / part-1
• elasticsearch-pull-request #67906 / part-1
• elasticsearch-pull-request #67902 / part-1
• elasticsearch-intake #21810 / part1
• elasticsearch-intake #21808 / part1

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.search.SearchServiceSingleNodeTests.testCreateReduceContext" -Dtests.seed=BB78CEAC17853E03 -Dtests.locale=fo-Latn-FO -Dtests.timezone=America/Merida -Druntime.java=24

Applicable branches:
main, 8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: There are still [1] in-flight contexts. The first one's creation site is listed as the cause of this exception.

Issue Reasons:

• [main] 2 consecutive failures in test testBeforeShardLockDuringShardCreate
• [main] 9 failures in test testBeforeShardLockDuringShardCreate (3.8% fail rate in 240 executions)
• [main] 6 failures in step part-1 (4.8% fail rate in 126 executions)
• [main] 2 failures in step part1 (8.0% fail rate in 25 executions)
• [main] 5 failures in pipeline elasticsearch-pull-request (4.1% fail rate in 123 executions)
• [main] 2 failures in pipeline elasticsearch-intake (8.0% fail rate in 25 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in test testBeforeShardLockDuringShardCreate
• [main] 9 failures in test testBeforeShardLockDuringShardCreate (3.8% fail rate in 240 executions)
• [main] 6 failures in step part-1 (4.8% fail rate in 126 executions)
• [main] 2 failures in step part1 (8.0% fail rate in 25 executions)
• [main] 5 failures in pipeline elasticsearch-pull-request (4.1% fail rate in 123 executions)
• [main] 2 failures in pipeline elasticsearch-intake (8.0% fail rate in 25 executions)

Build Scans:

• elasticsearch-periodic-platform-support #7679 / oraclelinux-8_platform-support-unix
• elasticsearch-pull-request #67921 / part-1
• elasticsearch-pull-request #67912 / part-1
• elasticsearch-pull-request #67912 / part-1
• elasticsearch-pull-request #67909 / part-1
• elasticsearch-pull-request #67906 / part-1
• elasticsearch-pull-request #67902 / part-1
• elasticsearch-intake #21810 / part1
• elasticsearch-intake #21808 / part1

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[benchaplin]

The first of these test failures I could find occurred Apr 14, 2025 @ 16:25:27.069.

I took a look at the git history and #123855 is looking mighty suspicious, although the change seems sensible to me.

If I understand it correctly, the assumption was: 'if a shard is being created for a new index, there won't be any active ReaderContexts for that shard, so don't bother walking the activeReaders CHM looking for any to free.'

However, if this is the cause of the test failures, that means that in these tests, there were ReaderContexts for new index shards that were freed before #123855, and are hanging around after.

One of the following must be true:

• Avoid walking the complete list of search contexts on shard creation #123855 is based on false assumptions
• The test setting (MockSearchService) is incorrectly constructed
• Avoid walking the complete list of search contexts on shard creation #123855 is not the cause of the test failures

I'll need to investigate more.