[CI] FeatureStateResetApiIT testResetSystemIndices and feature-migration/line_47 failing

[kingherc]

So far I notice this test failure goes hand in hand with the following test failure:

REPRODUCE WITH: ./gradlew ':docs:yamlRestTest' --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/migration/apis/feature-migration/line_47}" -Dtests.seed=ED495A645B8EA9A0 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=fi -Dtests.timezone=Europe/Istanbul -Druntime.java=20	
org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT > test {yaml=reference/migration/apis/feature-migration/line_47} FAILED	
    java.lang.AssertionError: Failure at [reference/migration/apis/feature-migration:15]: Expected a map containing	
             features: a list containing	
                      0: a map containing	
                  indices: an empty list	
             feature_name: "async_search"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      1: a map containing	
                  indices: an empty list	
             feature_name: "enrich"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      2: a map containing	
                  indices: an empty list	
             feature_name: "ent_search"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      3: a map containing	
                  indices: an empty list	
             feature_name: "fleet"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      4: a map containing	
                  indices: an empty list	
             feature_name: "geoip"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      5: a map containing	
                  indices: an empty list	
             feature_name: "kibana"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      6: a map containing	
                  indices: an empty list	
             feature_name: "logstash_management"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      7: a map containing	
                  indices: an empty list	
             feature_name: "machine_learning"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      8: a map containing	
                  indices: an empty list	
             feature_name: "searchable_snapshots"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                      9: a map containing	
                  indices: an empty list	
             feature_name: "security"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                     10: a map containing	
                  indices: an empty list	
             feature_name: expected "synonyms" but was "tasks"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                     11: a map containing	
                  indices: an empty list	
             feature_name: expected "tasks" but was "transform"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                     12: a map containing	
                  indices: an empty list	
             feature_name: expected "transform" but was "watcher"	
         migration_status: "NO_MIGRATION_NEEDED"	
    minimum_index_version: "8.9.0"	
                     13: expected a map but was <missing>	
     migration_status: "NO_MIGRATION_NEEDED"	
        at __randomizedtesting.SeedInfo.seed([ED495A645B8EA9A0:651D65BEF572C458]:0)	
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:572)	
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:524)	
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)	
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)	
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)	
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)	
        at java.base/java.lang.Thread.run(Thread.java:1623)

Build scan:
https://gradle-enterprise.elastic.co/s/qs5kpprbtirdq/tests/:server:internalClusterTest/org.elasticsearch.snapshots.FeatureStateResetApiIT/testResetSystemIndices

Reproduction line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.snapshots.FeatureStateResetApiIT.testResetSystemIndices" -Dtests.seed=90E6E28A2194E5A1 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=bg-BG -Dtests.timezone=Africa/Windhoek -Druntime.java=20

Applicable branches:
main, 8.9

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.snapshots.FeatureStateResetApiIT&tests.test=testResetSystemIndices

Failure excerpt:

java.lang.AssertionError: 
Expected: iterable with items [<ResetFeatureStateStatus{featureName='SystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='SecondSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='EvilSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='tasks', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='synonyms', status=SUCCESS, exception='null'}>] in any order
     but: no item matches: <ResetFeatureStateStatus{featureName='synonyms', status=SUCCESS, exception='null'}> in [<ResetFeatureStateStatus{featureName='EvilSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='SecondSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='SystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='tasks', status=SUCCESS, exception='null'}>]

  at __randomizedtesting.SeedInfo.seed([90E6E28A2194E5A1:3E3A135F1F5D364A]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.snapshots.FeatureStateResetApiIT.testResetSystemIndices(FeatureStateResetApiIT.java:74)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[kingherc]

Appears also several other times on main and 8.9 so will try to mute.

[rjernst]

@williamrandolph Can you take a look here?

[williamrandolph]

The problem in both cases looks to be that the synonyms feature is missing. There's a feature flag for the synonyms API; if the feature is disabled the system index descriptor doesn't get loaded at runtime. What we're seeing is consistent with the synonyms API being disable for these tests. I've been poking around to try to figure out how these flags get set and unset, but I'm not having any luck today.

[rjernst]

It seems odd the tasks and synonyms system indices are checked at all in this test. The test is checking feature reset with artificial system indices. It looks like synonyms was moved into server with the synonyms api work in #95895.

IMO those feature descriptors should be overridden completely by the test. I think this can be done by moving the definition of the server features to Node so that they can be merged with those from plugins before calling the SystemIndices ctor, and having it overrideable by MockNode. Then whether a particular real feature is available would be irrelevant to the test.