SearchableSnapshotsCanMatchOnCoordinatorIntegTests.testQueryPhaseIsExecutedInAnAvailableNodeWhenAllShardsCanBeSkipped failure

[nik9000]

Build scan:
https://gradle-enterprise.elastic.co/s/eyuntluqkrc6u/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsCanMatchOnCoordinatorIntegTests/testQueryPhaseIsExecutedInAnAvailableNodeWhenAllShardsCanBeSkipped#1

Repro line:

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' \
  --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsCanMatchOnCoordinatorIntegTests.testQueryPhaseIsExecutedInAnAvailableNodeWhenAllShardsCanBeSkipped" \
  -Dtests.seed=119FBF889C9916E -Dtests.security.manager=true -Dtests.locale=en \
  -Dtests.timezone=Canada/Newfoundland -Druntime.java=11 -Dtests.fips.enabled=true

Reproduces locally?:
nope

Applicable branches:
seems to be just 7.12, though a PR against master caught it too?

Failure history:
https://build-stats.elastic.co/goto/31ac343cdaf215db4714dd933cdc570e

Failure excerpt:

org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsCanMatchOnCoordinatorIntegTests > testQueryPhaseIsExecutedInAnAvailableNodeWhenAllShardsCanBeSkipped FAILED
    java.lang.AssertionError: 
    Expected: <3>
         but: was <4>
        at __randomizedtesting.SeedInfo.seed([119FBF889C9916E:6E35F52890D149F7]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsCanMatchOnCoordinatorIntegTests.testQueryPhaseIsExecutedInAnAvailableNodeWhenAllShardsCanBeSkipped(SearchableSnapshotsCanMatchOnCoordinatorIntegTests.java:297)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsCanMatchOnCoordinatorIntegTests.testQueryPhaseIsExecutedInAnAvailableNodeWhenAllShardsCanBeSkipped" -Dtests.seed=119FBF889C9916E -Dtests.security.manager=true -Dtests.locale=en -Dtests.timezone=Canada/Newfoundland -Druntime.java=11 -Dtests.fips.enabled=true

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[original-brownbear]

The master failure here is from the 8th, one day before I fixed it via #70136 ... looking into what may have caused 7.12 to have failed twice here now.

Interestingly tt's a FIPS run for both failures there, let's see :)

update: this reproduces easily on 7.12 running in a loop with the given seed above