[CI] FileSettingsRoleMappingUpgradeIT testRoleMappingsAppliedOnUpgrade {upgradedNodes=3} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5333 / 8.5.3_bwc
• elasticsearch-pull-request #45827 / 8.5.3_bwc
• elasticsearch-pull-request #45801 / 8.5.3_bwc
• elasticsearch-periodic #5321 / 8.5.3_bwc
• elasticsearch-periodic #5309 / 8.5.3_bwc
• elasticsearch-periodic #5279 / 8.5.3_bwc

Reproduction Line:

./gradlew ":qa:rolling-upgrade:v8.5.3#bwcTest" -Dtests.class="org.elasticsearch.upgrades.FileSettingsRoleMappingUpgradeIT" -Dtests.method="testRoleMappingsAppliedOnUpgrade {upgradedNodes=3}" -Dtests.seed=839510D5116047CB -Dtests.bwc=true -Dtests.locale=pa-Arab-PK -Dtests.timezone=Africa/Asmera -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [8.x] 6 consecutive failures in step 8.5.3_bwc
• [8.x] 6 failures in test testRoleMappingsAppliedOnUpgrade {upgradedNodes=3} (4.3% fail rate in 140 executions)
• [8.x] 6 failures in step 8.5.3_bwc (100.0% fail rate in 6 executions)
• [8.x] 4 failures in pipeline elasticsearch-periodic (40.0% fail rate in 10 executions)
• [8.x] 2 failures in pipeline elasticsearch-pull-request (25.0% fail rate in 8 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[jfreden]

I see this in the logs:

[2024-12-10T23:52:01,874][ERROR][o.e.r.s.FileSettingsService] [test-cluster-0] Error processing operator settings json file java.lang.IllegalStateException: Error processing state change request for file_settings, errors: Security index is not on the current version - the native realm will not be operational until the upgrade API is run on the security index
	at org.elasticsearch.server@8.5.3/org.elasticsearch.reservedstate.service.ReservedClusterStateService.checkAndReportError(ReservedClusterStateService.java:248)
	at org.elasticsearch.server@8.5.3/org.elasticsearch.reservedstate.service.ReservedClusterStateService$1.onFailure(ReservedClusterStateService.java:223)

This happens because of a race condition in older versions of operator defined role mappings, where the FileSettingsService tries to add role mappings to the security index before it has been initilized.

The test checks oldClusterHasFeature("gte_v8.7.0") so it shouldn't even run for 8.5.3. I wonder if it happens because the cluster is started even though the test shouldn't run (static class variable).

[jfreden]

Same as: #110884

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 6 consecutive failures in step 8.5.3_bwc
• [8.x] 6 failures in test testRoleMappingsAppliedOnUpgrade {upgradedNodes=3} (4.3% fail rate in 140 executions)
• [8.x] 6 failures in step 8.5.3_bwc (100.0% fail rate in 6 executions)
• [8.x] 4 failures in pipeline elasticsearch-periodic (40.0% fail rate in 10 executions)
• [8.x] 2 failures in pipeline elasticsearch-pull-request (25.0% fail rate in 8 executions)

Build Scans:

• elasticsearch-periodic #5333 / 8.5.3_bwc
• elasticsearch-pull-request #45827 / 8.5.3_bwc
• elasticsearch-pull-request #45801 / 8.5.3_bwc
• elasticsearch-periodic #5321 / 8.5.3_bwc
• elasticsearch-periodic #5309 / 8.5.3_bwc
• elasticsearch-periodic #5279 / 8.5.3_bwc

[jfreden]

Fixed in: 68d38a6