Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] FollowersCheckerTests#testFailsNodeThatIsUnhealthy fails on master #59172

Closed
cbuescher opened this issue Jul 7, 2020 · 3 comments · Fixed by #59277
Closed

[CI] FollowersCheckerTests#testFailsNodeThatIsUnhealthy fails on master #59172

cbuescher opened this issue Jul 7, 2020 · 3 comments · Fixed by #59277
Assignees
@DaveCTurner
Labels
:Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI v8.0.0-alpha1

Comments

@cbuescher
Copy link
Member

Build scan:
https://gradle-enterprise.elastic.co/s/uwmxo33dnzojo

Repro line:

./gradlew ':server:test' --tests "org.elasticsearch.cluster.coordination.FollowersCheckerTests.testFailsNodeThatIsUnhealthy" \
  -Dtests.seed=2C28E5E8C9CA7F3B \
  -Dtests.security.manager=true \
  -Dtests.locale=en-ZA \
  -Dtests.timezone=Etc/GMT+7 \
  -Druntime.java=11

Reproduces locally?:

Yes

Failure excerpt:

java.lang.AssertionError: 
Expected: "health check failed"
     but: was "followers check retry count exceeded"
	at __randomizedtesting.SeedInfo.seed([2C28E5E8C9CA7F3B:EF7745CE84E82FC0]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.cluster.coordination.FollowersCheckerTests.lambda$testBehaviourOfFailingNode$19(FollowersCheckerTests.java:363)
	at org.elasticsearch.cluster.coordination.FollowersChecker$FollowerChecker$2.run(FollowersChecker.java:370)
	at org.elasticsearch.cluster.coordination.DeterministicTaskQueue.runTask(DeterministicTaskQueue.java:133)
	at org.elasticsearch.cluster.coordination.DeterministicTaskQueue.runRandomTask(DeterministicTaskQueue.java:127)
	at org.elasticsearch.cluster.coordination.DeterministicTaskQueue.runAllRunnableTasks(DeterministicTaskQueue.java:77)
	at org.elasticsearch.cluster.coordination.FollowersCheckerTests.testBehaviourOfFailingNode(FollowersCheckerTests.java:372)
	at org.elasticsearch.cluster.coordination.FollowersCheckerTests.testFailsNodeThatIsUnhealthy(FollowersCheckerTests.java:310)
@cbuescher cbuescher added >test-failure Triaged test failures from CI :Distributed/CCR Issues around the Cross Cluster State Replication features v8.0.0 labels Jul 7, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/CCR)

@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Jul 7, 2020
@Bukhtawar Bukhtawar mentioned this issue Jul 7, 2020
Closed
@dnhatn dnhatn added :Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. and removed :Distributed/CCR Issues around the Cross Cluster State Replication features labels Jul 8, 2020
@dnhatn
Copy link
Member

dnhatn commented Jul 8, 2020

Another instance https://gradle-enterprise.elastic.co/s/6mksdzenq77hc/tests/:server:test/org.elasticsearch.cluster.coordination.FollowersCheckerTests/testFailsNodeThatIsUnhealthy#1

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Jul 9, 2020
@DaveCTurner
Fix node health-check-related test failures
8fa17a5 
In elastic#52680 we introduced a new health check mechanism. This commit fixes
up some sporadic related test failures, and improves the behaviour of
the `FollowersChecker` slightly in the case that no retries are
configured.

Closes elastic#59252
Closes elastic#59172
@DaveCTurner DaveCTurner mentioned this issue Jul 9, 2020
Merged
@danielmitterdorfer
Copy link
Member

Another instance in https://gradle-enterprise.elastic.co/s/lb6poefv2wphw.

DaveCTurner added a commit that referenced this issue Jul 9, 2020
@DaveCTurner
Fix node health-check-related test failures (#59277)
5e73d71 
In #52680 we introduced a new health check mechanism. This commit fixes
up some sporadic related test failures, and improves the behaviour of
the `FollowersChecker` slightly in the case that no retries are
configured.

Closes #59252
Closes #59172
DaveCTurner added a commit that referenced this issue Jul 9, 2020
@DaveCTurner
Fix node health-check-related test failures (#59277)
d56fc72 
In #52680 we introduced a new health check mechanism. This commit fixes
up some sporadic related test failures, and improves the behaviour of
the `FollowersChecker` slightly in the case that no retries are
configured.

Closes #59252
Closes #59172
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
:Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
6 participants
@jakelandis @danielmitterdorfer @DaveCTurner @cbuescher @dnhatn @elasticmachine