Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] testRetentionLeaseIsRenewedDuringRecovery fails #39011

Closed
dnhatn opened this issue Feb 17, 2019 · 5 comments
Closed

[CI] testRetentionLeaseIsRenewedDuringRecovery fails #39011

dnhatn opened this issue Feb 17, 2019 · 5 comments
Assignees
@jasontedor
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI

Comments

@dnhatn
Copy link
Member

dnhatn commented Feb 17, 2019

CcrRetentionLeaseIT#testRetentionLeaseIsRenewedDuringRecovery is failing frequently.

Some CI instances:

FAILURE 21.4s J4 | CcrRetentionLeaseIT.testRetentionLeaseIsRenewedDuringRecovery <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: a value greater than <1550430923773L>
   >      but: <1550430923773L> was equal to <1550430923773L>
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT.lambda$testRetentionLeaseIsRenewedDuringRecovery$3(CcrRetentionLeaseIT.java:269)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:858)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:832)
   > 	at org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT.testRetentionLeaseIsRenewedDuringRecovery(CcrRetentionLeaseIT.java:253)
   > 	at java.lang.Thread.run(Thread.java:748)

Relates #38829
@dnhatn dnhatn added >test-failure Triaged test failures from CI :Distributed/CCR Issues around the Cross Cluster State Replication features labels Feb 17, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

dnhatn added a commit that referenced this issue Feb 17, 2019
@dnhatn
Mute testRetentionLeaseIsRenewedDuringRecovery
e17f004 
Tracked at #39011
dnhatn added a commit that referenced this issue Feb 17, 2019
@dnhatn
Mute testRetentionLeaseIsRenewedDuringRecovery
204480d 
Tracked at #39011
dnhatn added a commit that referenced this issue Feb 17, 2019
@dnhatn
Mute testRetentionLeaseIsRenewedDuringRecovery
c4ba05a 
Tracked at #39011
dnhatn added a commit that referenced this issue Feb 17, 2019
@dnhatn
Mute testRetentionLeaseIsRenewedDuringRecovery
b986526 
Tracked at #39011
@dnhatn
Copy link
Member Author

dnhatn commented Feb 17, 2019

I have muted this test in master, 7.x, 7.0 and 6.7.

@jasontedor
Copy link
Member

Thanks @dnhatn, I will take a look soon.

jasontedor added a commit that referenced this issue Feb 21, 2019
@jasontedor
Fix CCR tests that manipulate transport requests
b764c00 
We have some CCR tests where we use mock transport send rules to control
the behavior that we desire in these tests. Namely, we want to simulate
an exception being thrown on the leader side, or a variety of other
situations. These send rules were put in place between the data nodes on
each side. However, it might not be the case that these requests are
being sent between data nodes. For example, a request that is handled on
a non-data master node would not be sent from a data node. And it might
not be the case that the request is sent to a data node, as it could be
proxied through a non-data coordinating node. This commit addresses this
by putting these send rules in places between all nodes on each side.

Closes #39011
Closes #39201
jasontedor added a commit that referenced this issue Feb 21, 2019
@jasontedor
Fix CCR tests that manipulate transport requests
ac18af2 
We have some CCR tests where we use mock transport send rules to control
the behavior that we desire in these tests. Namely, we want to simulate
an exception being thrown on the leader side, or a variety of other
situations. These send rules were put in place between the data nodes on
each side. However, it might not be the case that these requests are
being sent between data nodes. For example, a request that is handled on
a non-data master node would not be sent from a data node. And it might
not be the case that the request is sent to a data node, as it could be
proxied through a non-data coordinating node. This commit addresses this
by putting these send rules in places between all nodes on each side.

Closes #39011
Closes #39201
jasontedor added a commit that referenced this issue Feb 21, 2019
@jasontedor
Fix CCR tests that manipulate transport requests
719c38a 
We have some CCR tests where we use mock transport send rules to control
the behavior that we desire in these tests. Namely, we want to simulate
an exception being thrown on the leader side, or a variety of other
situations. These send rules were put in place between the data nodes on
each side. However, it might not be the case that these requests are
being sent between data nodes. For example, a request that is handled on
a non-data master node would not be sent from a data node. And it might
not be the case that the request is sent to a data node, as it could be
proxied through a non-data coordinating node. This commit addresses this
by putting these send rules in places between all nodes on each side.

Closes #39011
Closes #39201
@benwtrent
Copy link
Member

@jasontedor it seems that the fix did not work, or made another issues visible.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/1654/console

@jasontedor
Copy link
Member

@benwtrent At a glance, that looks like a different issue. Would you mind opening a new issue, and muting the test again? Sorry, I am mobile right now otherwise I would do that sort of work without burdening you for it.

@benwtrent benwtrent mentioned this issue Feb 21, 2019
Closed
weizijun pushed a commit to weizijun/elasticsearch that referenced this issue Feb 22, 2019
@jasontedor @weizijun
Fix CCR tests that manipulate transport requests
679f92f 
We have some CCR tests where we use mock transport send rules to control
the behavior that we desire in these tests. Namely, we want to simulate
an exception being thrown on the leader side, or a variety of other
situations. These send rules were put in place between the data nodes on
each side. However, it might not be the case that these requests are
being sent between data nodes. For example, a request that is handled on
a non-data master node would not be sent from a data node. And it might
not be the case that the request is sent to a data node, as it could be
proxied through a non-data coordinating node. This commit addresses this
by putting these send rules in places between all nodes on each side.

Closes elastic#39011
Closes elastic#39201
weizijun pushed a commit to weizijun/elasticsearch that referenced this issue Feb 22, 2019
@jasontedor @weizijun
Fix CCR tests that manipulate transport requests
d8b47b1 
We have some CCR tests where we use mock transport send rules to control
the behavior that we desire in these tests. Namely, we want to simulate
an exception being thrown on the leader side, or a variety of other
situations. These send rules were put in place between the data nodes on
each side. However, it might not be the case that these requests are
being sent between data nodes. For example, a request that is handled on
a non-data master node would not be sent from a data node. And it might
not be the case that the request is sent to a data node, as it could be
proxied through a non-data coordinating node. This commit addresses this
by putting these send rules in places between all nodes on each side.

Closes elastic#39011
Closes elastic#39201
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jasontedor jasontedor

Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@benwtrent @jasontedor @dnhatn @elasticmachine