[CI] FullClusterRestartIT testGeoIpSystemFeaturesMigration {cluster=UPGRADED} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #17066 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54520 / 8.19.0_bwc-snapshots
• elasticsearch-intake #17032 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54491 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54467 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54462 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54431 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54369 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54343 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #54332 / 8.19.0_bwc-snapshots

Reproduction Line:

./gradlew ":modules:ingest-geoip:qa:full-cluster-restart:v8.19.0#bwcTest" -Dtests.class="org.elasticsearch.ingest.geoip.FullClusterRestartIT" -Dtests.method="testGeoIpSystemFeaturesMigration {cluster=UPGRADED}" -Dtests.seed=728EAD38AD55261E -Dtests.bwc=true -Dtests.locale=tr -Dtests.timezone=Europe/Andorra -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:38811], URI [_cat/indices/*?s=index&h=index&expand_wildcards=all], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.geoip_databases-reindexed-for-10]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".geoip_databases-reindexed-for-10","index":".geoip_databases-reindexed-for-10"}],"type":"index_not_found_exception","reason":"no such index [.geoip_databases-reindexed-for-10]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".geoip_databases-reindexed-for-10","index":".geoip_databases-reindexed-for-10"},"status":404}

Issue Reasons:

• [main] 54 failures in test testGeoIpSystemFeaturesMigration {cluster=UPGRADED} (9.9% fail rate in 546 executions)
• [main] 17 failures in step 8.19.0_bwc-snapshots (8.9% fail rate in 190 executions)
• [main] 35 failures in step 8.18.0_bwc-snapshots (11.8% fail rate in 296 executions)
• [main] 12 failures in pipeline elasticsearch-intake (7.5% fail rate in 159 executions)
• [main] 41 failures in pipeline elasticsearch-pull-request (12.4% fail rate in 330 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testGeoIpSystemFeaturesMigration {cluster=UPGRADED} (1.4% fail rate in 209 executions)
• [main] 3 failures in step 8.18.0_bwc-snapshots (1.6% fail rate in 189 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.6% fail rate in 129 executions)

Build Scans:

• elasticsearch-intake #16652 / 8.18.0_bwc-snapshots
• elasticsearch-pull-request #53519 / 8.18.0_bwc-snapshots
• elasticsearch-pull-request #53513 / 8.18.0_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[joegallo]

This is #121085 (not #120966). Either way, I'm self assigning.

The issue here is the combination of my PRs and also #120168 both of which happened to merged within the window where they would have each had green CI individually, but never would have worked correctly together.

[joegallo]

#121122 should square this away (now that #121119 is in which reverted #120168).