Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] FileSettingsServiceTests testStopWorksIfProcessingDidntReturnYet failing #106968

Closed
mark-vieira opened this issue Apr 1, 2024 · 2 comments · Fixed by #108100
Closed

[CI] FileSettingsServiceTests testStopWorksIfProcessingDidntReturnYet failing #106968

mark-vieira opened this issue Apr 1, 2024 · 2 comments · Fixed by #108100
Assignees
Labels
:Core/Infra/Settings Settings infrastructure and APIs low-risk An open issue or test failure that is a low risk to future releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI

Comments

@mark-vieira
Copy link
Contributor

Only seen one instance of this but I was able to reproduce locally.

Build scan:
https://gradle-enterprise.elastic.co/s/vpruqgunowhmy/tests/:server:test/org.elasticsearch.reservedstate.service.FileSettingsServiceTests/testStopWorksIfProcessingDidntReturnYet

Reproduction line:

./gradlew ':server:test' --tests "org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testStopWorksIfProcessingDidntReturnYet" -Dtests.seed=947676C3E5264287 -Dtests.locale=hu-HU -Dtests.timezone=America/Virgin -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
Yes

Failure history:
Failure dashboard for org.elasticsearch.reservedstate.service.FileSettingsServiceTests#testStopWorksIfProcessingDidntReturnYet

Failure excerpt:

java.lang.AssertionError: (No message provided)

  at __randomizedtesting.SeedInfo.seed([947676C3E5264287:765AF5DC9A86EE1A]:0)
  at org.junit.Assert.fail(Assert.java:87)
  at org.junit.Assert.assertTrue(Assert.java:42)
  at org.junit.Assert.assertTrue(Assert.java:53)
  at org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testStopWorksIfProcessingDidntReturnYet(FileSettingsServiceTests.java:295)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1570)

@mark-vieira mark-vieira added :Core/Infra/Settings Settings infrastructure and APIs >test-failure Triaged test failures from CI labels Apr 1, 2024
@elasticsearchmachine elasticsearchmachine added blocker Team:Core/Infra Meta label for core/infra team labels Apr 1, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

mark-vieira added a commit that referenced this issue Apr 1, 2024
mark-vieira added a commit that referenced this issue Apr 2, 2024
@mark-vieira
Copy link
Contributor Author

Getting loads of these failures in other test methods so I've muted the entire class.

@elasticsearchmachine elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label Apr 24, 2024
rjernst added a commit to rjernst/elasticsearch that referenced this issue Apr 30, 2024
When the file watched by file settings is initially missing, a special
method in reserved state service is called to write a dummy cluster
state entry. In the case of tests, there is no real running master
service, so when the task is submitted, the file watcher thread actually
barfs and the watcher dies, silently. That then causes the test to
timeout as it waits indefinitely but the file watcher is no longer
watching for the test file that was written.

This commit mocks out writing this empty state in the reserved state
service. It also collapses the two tests that check stopping while
blocked in processing works since they were almost exactly the same.

closes elastic#106968
@rjernst rjernst added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Apr 30, 2024
rjernst added a commit that referenced this issue May 8, 2024
When the file watched by file settings is initially missing, a special
method in reserved state service is called to write a dummy cluster
state entry. In the case of tests, there is no real running master
service, so when the task is submitted, the file watcher thread actually
barfs and the watcher dies, silently. That then causes the test to
timeout as it waits indefinitely but the file watcher is no longer
watching for the test file that was written.

This commit mocks out writing this empty state in the reserved state
service. It also collapses the two tests that check stopping while
blocked in processing works since they were almost exactly the same.

closes #106968
markjhoy pushed a commit to markjhoy/elasticsearch that referenced this issue May 9, 2024
When the file watched by file settings is initially missing, a special
method in reserved state service is called to write a dummy cluster
state entry. In the case of tests, there is no real running master
service, so when the task is submitted, the file watcher thread actually
barfs and the watcher dies, silently. That then causes the test to
timeout as it waits indefinitely but the file watcher is no longer
watching for the test file that was written.

This commit mocks out writing this empty state in the reserved state
service. It also collapses the two tests that check stopping while
blocked in processing works since they were almost exactly the same.

closes elastic#106968
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Settings Settings infrastructure and APIs low-risk An open issue or test failure that is a low risk to future releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants