Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] ReplicaShardAllocatorIT testFullClusterRestartPerformNoopRecovery failing #108037

Closed
davidkyle opened this issue Apr 29, 2024 · 2 comments
Closed

[CI] ReplicaShardAllocatorIT testFullClusterRestartPerformNoopRecovery failing #108037

davidkyle opened this issue Apr 29, 2024 · 2 comments
Assignees
@arteam
Labels
:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@davidkyle
Copy link
Member

Build scan:
https://gradle-enterprise.elastic.co/s/sfl5iqhsuazz6/tests/:server:internalClusterTest/org.elasticsearch.gateway.ReplicaShardAllocatorIT/testFullClusterRestartPerformNoopRecovery

Reproduction line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.gateway.ReplicaShardAllocatorIT.testFullClusterRestartPerformNoopRecovery" -Dtests.seed=162905307FB74387 -Dtests.locale=nb-NO -Dtests.timezone=Etc/GMT+9 -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.gateway.ReplicaShardAllocatorIT#testFullClusterRestartPerformNoopRecovery

Failure excerpt:

java.lang.AssertionError: 
Expected: <0>
     but: was <-1>

  at __randomizedtesting.SeedInfo.seed([162905307FB74387:FF8A98C97A397724]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2150)
  at org.elasticsearch.gateway.ReplicaShardAllocatorIT.assertNoOpRecoveries(ReplicaShardAllocatorIT.java:442)
  at org.elasticsearch.gateway.ReplicaShardAllocatorIT.testFullClusterRestartPerformNoopRecovery(ReplicaShardAllocatorIT.java:259)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)
@davidkyle davidkyle added :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >test-failure Triaged test failures from CI labels Apr 29, 2024
@elasticsearchmachine elasticsearchmachine added Team:Distributed Meta label for distributed team needs:risk Requires assignment of a risk label (low, medium, blocker) labels Apr 29, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@arteam arteam self-assigned this May 2, 2024
@henningandersen henningandersen added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels May 6, 2024
arteam added a commit that referenced this issue May 21, 2024
@arteam
Print out recovery translog state
7dfa99e 
The test in #108037 failed only once in 3 months, it never reproduces on my machine.
Let's print the state of translog before verifying, so we have more debug information
about it if the failure ever happens again.

See #108037
@arteam arteam mentioned this issue May 21, 2024
Merged
arteam added a commit that referenced this issue May 22, 2024
@arteam @DaveCTurner
[test] Print out recovery translog state (#108858)
1148861 
The test in #108037 failed only once in 3 months, it never reproduces on my machine.
Let's print the state of translog before verifying, so we have more debug information
about it if the failure ever happens again.

See #108037

* Update server/src/main/java/org/elasticsearch/indices/recovery/RecoveryState.java

Co-authored-by: David Turner <david.turner@elastic.co>

---------

Co-authored-by: David Turner <david.turner@elastic.co>
@arteam
Copy link
Contributor

arteam commented Jun 10, 2024

The test hasn't failed over a month in 13,676 executions, closing.

@arteam arteam closed this as completed Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arteam arteam

Labels
:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@arteam @davidkyle @henningandersen @elasticsearchmachine