DieWithDignityIT.testDieWithDignity failures on CI

[gwbrown]

Build scan: https://gradle-enterprise.elastic.co/s/fl56dmvxet2zw

Repro line: ./gradlew ':test:external-modules:test-die-with-dignity:javaRestTest' --tests "org.elasticsearch.qa.die_with_dignity.DieWithDignityIT.testDieWithDignity" -Dtests.seed=C3A1E728D7318E71 -Dtests.locale=es-CU -Dtests.timezone=Etc/GMT+9 -Druntime.java=8

Reproduces locally?: No

Applicable branches: master and 7.x

Failure history: First failed on Sep. 1 (build scan), has failed 26 times since.

Failure excerpt:

java.lang.AssertionError: (No message provided)	
	at __randomizedtesting.SeedInfo.seed([C3A1E728D7318E71:CA6A313201AA1C6A]:0)	
	at org.junit.Assert.fail(Assert.java:86)	
	at org.junit.Assert.assertTrue(Assert.java:41)	
	at org.junit.Assert.assertTrue(Assert.java:52)	
	at org.elasticsearch.qa.die_with_dignity.DieWithDignityIT.testDieWithDignity(DieWithDignityIT.java:45)	
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)	
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	
	at java.lang.reflect.Method.invoke(Method.java:498)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)	
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)	
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)	
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)	
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)	
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)	
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)	
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)	
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)	
	at java.lang.Thread.run(Thread.java:748)

/cc @jasontedor as it looks like you touched this test in #77039 not too long before the first failure, could be related?

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[gwbrown]

Muting on master and 7.x in #77283 as this is failing quite a bit.

[valeriy42]

There are two failures that may be related:

• https://gradle-enterprise.elastic.co/s/jaip6jdm3s4q4
• https://gradle-enterprise.elastic.co/s/yrdu67o6onlts

[jasontedor]

Here's the problem. We run jps to check that an Elasticsearch instance with the appropriate flags has been started. We later verify that this instance has been killed, by running jps again, after we hit the /_die_with_dignity endpoint. The problem is that jps truncates the output to 1024 characters, and sometimes the flag that we are looking for is past the truncation. This leads to spurious failures. It also means that the method of asserting that there is not an instance with the die with dignity flag running is prone to false negatives. This is a known issue with jps that the JDK maintainers marked as won't fix.

[ywelsch]

Looks to still fail: https://gradle-enterprise.elastic.co/s/oxkjscbj6d5fq

Now with:

java.lang.AssertionError: jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 -XX:+HeapDumpOnOutOfMemoryError -Xss2m --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED -Dfile.encoding=UTF8 -Duser.country=US -Duser.language=en -Duser.variant ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xms256m -Xmx512m -Dfile.encoding=UTF8 -Duser.country=US -Duser.language=en -Duser.variant ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED -Xmx512m -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED -Xmx512m -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 -Xmx64m -Xms64m -Dorg.gradle.appname=gradlew ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 -Djava.util.logging.config.file=C:\Program Files\jenkins\logging.properties -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=20 ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 -Xms4m -Xmx16m ,jvm_args: -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8 -Des.scripting.update.ctx_in_params=false -Des.search.rewrite_sort=true -Des.transport.cname_in_publish_address=true -Dgradle.dist.lib=C:\Users\jenkins\.gradle\wrapper\dists\gradle-7.2-all\260hg96vuh6ex27h9vo47iv4d\gradle-7.2/lib -Dgradle.user.home=C:\Users\jenkins\.gradle -Dgradle.worker.jar=C:\Users\jenkins\.gradle/caches/7.2/workerMain/gradle-worker.jar -Dio.netty.noKeySetOptimization=true -Dio.netty.noUnsafe=true -Dio.netty.recycler.maxCapacityPerThread=0 -Djava.awt.headless=true -Djava.locale.providers=SPI,COMPAT -Djna.nosys=true -Dlog=C:\Users\jenkins\workspace\platform-support\257\test\external-modules\die-with-dignity\build\testclusters\javaRestTest-0\logs\javaRestTest_server.json -Dorg.gradle.internal.worker.tmpdir=C:\Users\jenkins\workspace\platform-support\257\test\external-modules\die-with-dignity\build\tmp\javaRestTest\work -Dorg.gradle.native=false -Druntime.java.home=C:\Users\jenkins\.java\java11 -Dtests.artifact=test-die-with-dignity -Dtests.cluster=127.0.0.1:54686,[::1]:54687 -Dtests.clustername=javaRestTest -Dtests.gradle=true -Dtests.logger.level=WARN -Dtests.rest.cluster=127.0.0.1:54693,[::1]:54694 -Dtests.rest.load_packaged=false -Dtests.security.manager=false -Dtests.seed=4E184FDEBD0448C6 -Dtests.system_call_filter=false -Dtests.task=:test:external-modules:test-die-with-dignity:javaRestTest -Dtests.timeoutSuite=2400000! --illegal-access=deny --add-opens=java.base/java.security.cert=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/javax.net.ssl=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.management/java.lang.management=ALL-UNNAMED -XX:+HeapDumpOnOutOfMemoryError -esa -XX:HeapDumpPath=C:\Users\jenkins\workspace\platform-support\257\test\external-modules\die-with-dignity\build\heapdump --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED -Xms512m -Xmx512m -Dfile.encoding=UTF-8 -Djava.io.tmpdir=C:\Users\jenkins\workspace\platform-support\257\test\external-mo…


at __randomizedtesting.SeedInfo.seed([4E184FDEBD0448C6:47D399C46B9FDADD]:0) |  
-- | --
  |   | at org.junit.Assert.fail(Assert.java:88) |  
  |   | at org.junit.Assert.assertTrue(Assert.java:41) |  
  |   | at org.elasticsearch.qa.die_with_dignity.DieWithDignityIT.testDieWithDignity(DieWithDignityIT.java:35) |  
  |   | at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) |  
  |   | at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) |  
  |   | at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) |  
  |   | at java.lang.reflect.Method.invoke(Method.java:566) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) |  
  |   | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  |   | at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) |  
  |   | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  |   | at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) |  
  |   | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  |   | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  |   | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  |   | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375) |  
  |   | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824) |  
  |   | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) |  
  |   | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) |  
  |   | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  |   | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  |   | at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) |  
  |   | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  |   | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  |   | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  |   | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  |   | at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |  
  |   | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  |   | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  |   | at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) |  
  |   | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  |   | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375) |  
  |   | at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831) |  
  |   | at java.lang.Thread.run(Thread.java:834)