Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Test Failure CcrRetentionLeaseIT.testRetentionLeaseIsRenewedDuringRecovery #39268

Closed
benwtrent opened this issue Feb 21, 2019 · 4 comments
Closed

[CI] Test Failure CcrRetentionLeaseIT.testRetentionLeaseIsRenewedDuringRecovery #39268

benwtrent opened this issue Feb 21, 2019 · 4 comments
Assignees
@jasontedor
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI

Comments

@benwtrent
Copy link
Member

benwtrent commented Feb 21, 2019 •
edited
Loading

Build:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/1654/console

Unable to reproduce locally:

./gradlew :x-pack:plugin:ccr:internalClusterTest -Dtests.seed=BF18D99722E0EC18 -Dtests.class=org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT -Dtests.method="testRetentionLeaseIsRenewedDuringRecovery" -Dtests.security.manager=true -Dtests.locale=zh-TW -Dtests.timezone=America/Moncton -Dcompiler.java=11 -Druntime.java=8

Failure trace:

11:47:02 FAILURE 29.4s J4 | CcrRetentionLeaseIT.testRetentionLeaseIsRenewedDuringRecovery <<< FAILURES!
11:47:02    > Throwable #1: java.lang.AssertionError: 
11:47:02    > Expected: a value greater than <1550771187477L>
11:47:02    >      but: <1550771187477L> was equal to <1550771187477L>
11:47:02    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:47:02    > 	at org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT.lambda$testRetentionLeaseIsRenewedDuringRecovery$3(CcrRetentionLeaseIT.java:267)
11:47:02    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:868)
11:47:02    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:842)
11:47:02    > 	at org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT.testRetentionLeaseIsRenewedDuringRecovery(CcrRetentionLeaseIT.java:254)
11:47:02    > 	at java.lang.Thread.run(Thread.java:748)
11:47:02    > 	Suppressed: java.lang.AssertionError:

Possibly related to #39011
@benwtrent benwtrent added >test-failure Triaged test failures from CI :Distributed/CCR Issues around the Cross Cluster State Replication features labels Feb 21, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

This was referenced Feb 21, 2019
Merged
Merged
Merged
Merged
@jasontedor
Copy link
Member

I think it's a race condition in the test and I know how to address it. I will try to get to in the next few days. I am so sorry to the team for the disruption caused by this test.

@markharwood
Copy link
Contributor

Build-stats link

@jasontedor jasontedor self-assigned this Feb 22, 2019
jasontedor added a commit that referenced this issue Feb 22, 2019
@jasontedor
Add test logging to CCR retention lease test
fee33d5 
This commit adds some additional logging to a failing CCR retention
lease test so that the nature of the failure can be completely
understood before attempting to push a fix.

Relates #39268
jasontedor added a commit that referenced this issue Feb 23, 2019
@jasontedor
Fix failing CCR retention lease test
6e06f82 
Finally! This commit should fix the issues with the CCR retention lease
that has been plaguing build failures. The issue here is that we are
trying to prevent the clear session requests from being executed until
after we have been able to validate that retention leases are being
renewed. However, we were only blocking the clear session requests but
not blocking them when they are proxied through another node. This
commit addresses that.

Relates #39268
jasontedor added a commit that referenced this issue Feb 23, 2019
@jasontedor
Fix failing CCR retention lease test
22e27c3 
Finally! This commit should fix the issues with the CCR retention lease
that has been plaguing build failures. The issue here is that we are
trying to prevent the clear session requests from being executed until
after we have been able to validate that retention leases are being
renewed. However, we were only blocking the clear session requests but
not blocking them when they are proxied through another node. This
commit addresses that.

Relates #39268
jasontedor added a commit that referenced this issue Feb 23, 2019
@jasontedor
Fix failing CCR retention lease test
41a3e03 
Finally! This commit should fix the issues with the CCR retention lease
that has been plaguing build failures. The issue here is that we are
trying to prevent the clear session requests from being executed until
after we have been able to validate that retention leases are being
renewed. However, we were only blocking the clear session requests but
not blocking them when they are proxied through another node. This
commit addresses that.

Relates #39268
@jasontedor
Copy link
Member

This test has not failed since the fix that I pushed last night.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jasontedor jasontedor

Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@markharwood @benwtrent @jasontedor @elasticmachine