Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch keystore test failures in JDK22 #104858

Closed
jan-elastic opened this issue Jan 29, 2024 · 17 comments · Fixed by #106559
Closed

Elasticsearch keystore test failures in JDK22 #104858

jan-elastic opened this issue Jan 29, 2024 · 17 comments · Fixed by #106559
Labels
blocker :Core/Infra/Core Core issues without another label :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC Team:Core/Infra Meta label for core/infra team Team:Security Meta label for security team >test-failure Triaged test failures from CI

Comments

@jan-elastic
Copy link
Contributor

Build scan:
https://gradle-enterprise.elastic.co/s/j2hllfv26rfzi/tests/:x-pack:plugin:security:qa:multi-cluster:v7.17.18%23bwcTest/org.elasticsearch.xpack.remotecluster.RemoteClusterSecurityBwcRestIT

Reproduction line:

null

Applicable branches:
main, 8.12

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.xpack.remotecluster.RemoteClusterSecurityBwcRestIT#classMethod

Failure excerpt:

java.lang.RuntimeException: An error occurred orchestrating test cluster.

  at __randomizedtesting.SeedInfo.seed([4E404BA9DD1E8282]:0)
  at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:264)
  at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:269)
  at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.start(DefaultLocalClusterHandle.java:72)
  at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:45)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1570)

  Caused by: java.lang.RuntimeException: Execution of elasticsearch-keystore failed with exit code 1

    at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.runToolScript(AbstractLocalClusterFactory.java:789)
    at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.createKeystore(AbstractLocalClusterFactory.java:465)
    at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.start(AbstractLocalClusterFactory.java:169)
    at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$0(DefaultLocalClusterHandle.java:72)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
    at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:759)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
    at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:676)
    at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:264)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:785)
    at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$1(DefaultLocalClusterHandle.java:72)
    at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$execute$16(DefaultLocalClusterHandle.java:270)
    at java.util.concurrent.ForkJoinTask$AdaptedInterruptibleCallable.compute(ForkJoinTask.java:1689)
    at java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1491)
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2073)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2035)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

@jan-elastic jan-elastic added :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC >test-failure Triaged test failures from CI labels Jan 29, 2024
@elasticsearchmachine elasticsearchmachine added blocker Team:Security Meta label for security team labels Jan 29, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

@slobodanadamovic
Copy link
Contributor

This issue is reproducible on openjdk22 (which is not yet officially supported) and seems like a duplicate of #103963.

@slobodanadamovic slobodanadamovic added medium-risk An open issue or test failure that is a medium risk to future releases and removed blocker labels Jan 29, 2024
@piergm
Copy link
Member

piergm commented Feb 6, 2024

Another instance occurred today: https://gradle-enterprise.elastic.co/s/danbligjgug4q

@joegallo
Copy link
Contributor

joegallo commented Feb 8, 2024

@iverase
Copy link
Contributor

iverase commented Feb 9, 2024

@kingherc
Copy link
Contributor

@kingherc
Copy link
Contributor

Another one at elasticsearch / periodic / main / openjdk22 / bwcTestSnapshots / java-matrix

I'll mute the whole test suite since this is happening a lot lately! Security team, please revert the mute if that's not OK.

kingherc added a commit to kingherc/elasticsearch that referenced this issue Feb 14, 2024
kingherc added a commit to kingherc/elasticsearch that referenced this issue Feb 14, 2024
@benwtrent
Copy link
Member

benwtrent commented Mar 6, 2024

many 8.12 & 7.17 test builds with JDK22 all just failed with issue.

Process 'command './bin/elasticsearch-keystore'' finished with non-zero exit value 1
or some variation of elasticsearch-keystore failed with exit code 1

JDK22 just isn't playing nice with this.

I could list all the build failures, but the list would be rather long.

@benwtrent benwtrent changed the title [CI] RemoteClusterSecurityBwcRestIT classMethod failing Elasticsearch keystore test failures in JDK22 Mar 6, 2024
@benwtrent benwtrent added blocker :Core/Infra/Core Core issues without another label and removed medium-risk An open issue or test failure that is a medium risk to future releases labels Mar 6, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Core/Infra Meta label for core/infra team label Mar 6, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@benwtrent
Copy link
Member

I am moving and relabeling this issue. Numerous test suites are failing JDK22 due to keystore exiting with a non-zero status.

@rjernst
Copy link
Member

rjernst commented Mar 6, 2024

It looks like these are all the result of a change to Console in JDK 22 which we already addressed in #103614, but was not backported to 8.12 or 7.17. I'll work on backporting to those branches.

@rjernst
Copy link
Member

rjernst commented Mar 6, 2024

8.12: #106049
7.17: #106051

rjernst added a commit to rjernst/elasticsearch that referenced this issue Mar 7, 2024
Jdk 22 had test failures which caused this test to be muted. It looks
like this bwc test may have failed because earlier ES versions had not
yet had the necessary change to Terminal backported yet. That should be
fixed now with elastic#106049 and elastic#106051. This commit re-enables the bwc test.

relates elastic#104858
rjernst added a commit that referenced this issue Mar 7, 2024
Jdk 22 had test failures which caused this test to be muted. It looks
like this bwc test may have failed because earlier ES versions had not
yet had the necessary change to Terminal backported yet. That should be
fixed now with #106049 and #106051. This commit re-enables the bwc test.

relates #104858
fang-xing-esql pushed a commit to fang-xing-esql/Elasticsearch that referenced this issue Mar 8, 2024
Jdk 22 had test failures which caused this test to be muted. It looks
like this bwc test may have failed because earlier ES versions had not
yet had the necessary change to Terminal backported yet. That should be
fixed now with elastic#106049 and elastic#106051. This commit re-enables the bwc test.

relates elastic#104858
@williamrandolph
Copy link
Contributor

Since the fix has been backported and we haven't seen issues, we're closing this as fixed.

@mosche
Copy link
Contributor

mosche commented Mar 20, 2024

The issue seems to be that we're also forcing JDK 22 to run the old distributions in bwc tests.
Obviously, these are not build to support JDK 22 yet.

E.g. in this case elasticsearch-keystore is invoked with JAVA resolving to JDK 22 in elasticsearch-env and JAVA_TYPE being ES_JAVA_HOME.

elasticsearchmachine pushed a commit that referenced this issue Mar 20, 2024
…06559)

We have instances where BWC tests configure old ES version nodes with
the integTest distribution. This isn't a valid configuration, and while
we in reality resolve the default distribution artifact, we have other
configuration logic that behaves differently based on whether the
integTest distro was _requested_. Specifically, what to set ES_JAVA_HOME
to. This bug resulted in us attempting to run old nodes using the
current bundled JDK version, which may be incompatible with that older
version of Elasticsearch.

Closes #104858
mark-vieira added a commit to mark-vieira/elasticsearch that referenced this issue Mar 20, 2024
…astic#106559)

We have instances where BWC tests configure old ES version nodes with
the integTest distribution. This isn't a valid configuration, and while
we in reality resolve the default distribution artifact, we have other
configuration logic that behaves differently based on whether the
integTest distro was _requested_. Specifically, what to set ES_JAVA_HOME
to. This bug resulted in us attempting to run old nodes using the
current bundled JDK version, which may be incompatible with that older
version of Elasticsearch.

Closes elastic#104858
elasticsearchmachine pushed a commit that referenced this issue Mar 20, 2024
…06559) (#106565)

We have instances where BWC tests configure old ES version nodes with
the integTest distribution. This isn't a valid configuration, and while
we in reality resolve the default distribution artifact, we have other
configuration logic that behaves differently based on whether the
integTest distro was _requested_. Specifically, what to set ES_JAVA_HOME
to. This bug resulted in us attempting to run old nodes using the
current bundled JDK version, which may be incompatible with that older
version of Elasticsearch.

Closes #104858
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker :Core/Infra/Core Core issues without another label :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC Team:Core/Infra Meta label for core/infra team Team:Security Meta label for security team >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.