[CI] OperatorPrivilegesIT testEveryActionIsEitherOperatorOnlyOrNonOperator failing

[ywangd]

This fails on release tests. So most likely feature flag related.

Build scan:
https://gradle-enterprise.elastic.co/s/jutz7xtqbsct2/tests/:x-pack:plugin:security:qa:operator-privileges-tests:javaRestTest/org.elasticsearch.xpack.security.operator.OperatorPrivilegesIT/testEveryActionIsEitherOperatorOnlyOrNonOperator

Reproduction line:

./gradlew ':x-pack:plugin:security:qa:operator-privileges-tests:javaRestTest' --tests "org.elasticsearch.xpack.security.operator.OperatorPrivilegesIT.testEveryActionIsEitherOperatorOnlyOrNonOperator" -Dtests.seed=57F1F58DA15C8AB6 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=de -Dtests.timezone=America/Puerto_Rico -Druntime.java=20

Applicable branches:
8.9

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.security.operator.OperatorPrivilegesIT&tests.test=testEveryActionIsEitherOperatorOnlyOrNonOperator

Failure excerpt:

java.lang.AssertionError: Actions are neither operator-only nor non-operator: [[cluster:monitor/xpack/usage/data_lifecycle]]. An action should be declared to be either operator-only in [org.elasticsearch.xpack.security.operator.DefaultOperatorOnlyRegistry] or non-operator in [org.elasticsearch.xpack.security.operator.Constants]

  at __randomizedtesting.SeedInfo.seed([57F1F58DA15C8AB6:B8CD6326473F8B11]:0)
  at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.elasticsearch.xpack.security.operator.OperatorPrivilegesIT.testEveryActionIsEitherOperatorOnlyOrNonOperator(OperatorPrivilegesIT.java:115)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

I think this was caused by #97425, and I think the fix is just the same thing done here in main. @gmarouli or @andreidan might want to double-check me on that though. It looks like 8.10 already has the fix, so I think it's just a problem in 8.9.

[ywangd]

@masseyke I raised #98760 for a fix. It touches more than just the data_lifecycle action. There are a few other actions need to be adjusted for feature flags for the test to pass. Do you mind review it? 8.9 will not be maintain anymore once 8.10 is released (in about 3 weeks time). So we may also choose to do nothing. I don't mind either way. Thanks!

[thecoop]

Also fails on 8.9: https://gradle-enterprise.elastic.co/s/zu5hyja5qeygq/tests/task/:x-pack:plugin:security:qa:operator-privileges-tests:javaRestTest/details/org.elasticsearch.xpack.security.operator.OperatorPrivilegesIT/testEveryActionIsEitherOperatorOnlyOrNonOperator?top-execution=1