Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated Fails on 7.x #59608

Closed
original-brownbear opened this issue Jul 15, 2020 · 2 comments · Fixed by #59611
Closed

[CI] ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated Fails on 7.x #59608

original-brownbear opened this issue Jul 15, 2020 · 2 comments · Fixed by #59611
Assignees
@original-brownbear
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@original-brownbear
Copy link
Member

This only fails on 7.x and 7.9 it seems and just now started after back porting the concurrent snapshot logic.

In https://gradle-enterprise.elastic.co/s/e7obweucvwtv4/tests/:server:internalClusterTest/org.elasticsearch.snapshots.ConcurrentSnapshotsIT/testEquivalentDeletesAreDeduplicated#1

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.snapshots.ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated" -Dtests.seed=B89D42C420C554C -Dtests.security.manager=true -Dtests.locale=uk -Dtests.timezone=Jamaica -Druntime.java=8

fails with:


org.elasticsearch.snapshots.ConcurrentSnapshotsIT > testEquivalentDeletesAreDeduplicated FAILED
    java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertFalse(Assert.java:64)
        at org.junit.Assert.assertFalse(Assert.java:74)
        at org.elasticsearch.snapshots.ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated(ConcurrentSnapshotsIT.java:941)

    java.lang.AssertionError: Found leaked finalizations [test-repo] on [{node_t0}{d1OdKDoTTVyP2Wu5t7ohIA}{ZOmVO76zShCS0JcTnGKJpQ}{127.0.0.1}{127.0.0.1:39923}{m}]
        at org.elasticsearch.snapshots.SnapshotsService.assertAllListenersResolved(SnapshotsService.java:2414)
        at org.elasticsearch.snapshots.AbstractSnapshotIntegTestCase.lambda$verifyNoLeakedListeners$0(AbstractSnapshotIntegTestCase.java:113)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:951)
        at org.elasticsearch.snapshots.AbstractSnapshotIntegTestCase.verifyNoLeakedListeners(AbstractSnapshotIntegTestCase.java:111)
        at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)

This reproduces locally after a few iterations also. Looks like this is likely a test infra issue somehow since the relevant code seems to be the same between master and 7.x but either way I'm on it.
@original-brownbear original-brownbear added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Jul 15, 2020
@original-brownbear original-brownbear self-assigned this Jul 15, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Jul 15, 2020
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jul 15, 2020
@original-brownbear
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated
7138286 
Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes elastic#59608
@original-brownbear original-brownbear mentioned this issue Jul 15, 2020
Merged
@dakrone
Copy link
Member

dakrone commented Jul 15, 2020

Another failure: https://gradle-enterprise.elastic.co/s/afdpahqegjmny

original-brownbear added a commit that referenced this issue Jul 15, 2020
@original-brownbear
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated (#59611)
c51d765 
Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes #59608
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jul 15, 2020
@original-brownbear
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated (elast…
e4eed67 
…ic#59611)

Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes elastic#59608
@original-brownbear original-brownbear mentioned this issue Jul 15, 2020
Merged
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jul 15, 2020
@original-brownbear
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated (elast…
d82621d 
…ic#59611)

Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes elastic#59608
@original-brownbear original-brownbear mentioned this issue Jul 15, 2020
Merged
original-brownbear added a commit that referenced this issue Jul 16, 2020
@original-brownbear
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated (#59611)…
6db481f 
… (#59653)

Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes #59608
original-brownbear added a commit that referenced this issue Jul 16, 2020
@original-brownbear
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated (#59611)…
dacfe6a 
… (#59654)

Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes #59608
malpani pushed a commit to malpani/elasticsearch that referenced this issue Jul 17, 2020
@original-brownbear @malpani
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated (elast…
a508f90 
…ic#59611)

Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes elastic#59608
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated original-brownbear/elasticsearch
3 participants
@dakrone @original-brownbear @elasticmachine