org.elasticsearch.upgrades.ClusterFeatureMigrationIT fails intermittently

[ChrisHegarty]

Reproduce with:

./gradlew ":qa:rolling-upgrade:v8.16.0#bwcTest" -Dtests.class="org.elasticsearch.upgrades.ClusterFeatureMigrationIT" -Dtests.method="testClusterFeatureMigration {upgradedNodes=1}"

fails with:

ClusterFeatureMigrationIT > testClusterFeatureMigration {upgradedNodes=1} FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([DB7C7EA7CA82F8CD:FB3214C70CD760E]:0)
        at org.elasticsearch.upgrades.ParameterizedRollingUpgradeTestCase.oldClusterHasFeature(ParameterizedRollingUpgradeTestCase.java:143)
        at org.elasticsearch.upgrades.ParameterizedRollingUpgradeTestCase.oldClusterHasFeature(ParameterizedRollingUpgradeTestCase.java:148)
        at org.elasticsearch.upgrades.ClusterFeatureMigrationIT.checkMigrationVersion(ClusterFeatureMigrationIT.java:34)

The test appears to hit an assert when trying to determine whether or not the old cluster has a feature.

  protected static boolean oldClusterHasFeature(String featureId) {
      assert oldClusterTestFeatureService != null;    <<<<<<<<<<<< HERE
      return oldClusterTestFeatureService.clusterHasFeature(featureId);
  }

I believe that the intermittency of the failure comes from the randomisation of the execution order of the three @Before methods in ParameterizedRollingUpgradeTestCase. Are these order dependent? kinda looks like they are; or at least extractOldClusterFeatures needs to be run before upgradeNode.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

Thank for the analysis, Chris. I agree with your assessment and opened #114677.