Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] the org.opensearch.snapshots.DeleteSnapshotIT tests are flaky #9208

Open
reta opened this issue Aug 9, 2023 · 5 comments
Open

[BUG] the org.opensearch.snapshots.DeleteSnapshotIT tests are flaky #9208

reta opened this issue Aug 9, 2023 · 5 comments
Assignees
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Storage:Snapshots

Comments

@reta
Copy link
Collaborator

reta commented Aug 9, 2023

Describe the bug

org.opensearch.snapshots.DeleteSnapshotIT.testRemoteStoreCleanupForDeletedIndex

java.lang.AssertionError: 
Expected: is <2>
     but: was <1>
	at __randomizedtesting.SeedInfo.seed([89E1AFC9EA8EC59C:CCF6AA4C23189F3D]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.createFullSnapshot(AbstractSnapshotIntegTestCase.java:488)
	at org.opensearch.snapshots.DeleteSnapshotIT.createNSnapshots(DeleteSnapshotIT.java:369)
	at org.opensearch.snapshots.DeleteSnapshotIT.testRemoteStoreCleanupForDeletedIndex(DeleteSnapshotI
org.opensearch.snapshots.DeleteSnapshotIT.testDeleteMultipleShallowCopySnapshotsCase1

java.lang.AssertionError: 
Expected: is <2>
     but: was <1>
	at __randomizedtesting.SeedInfo.seed([89E1AFC9EA8EC59C:3253EBC0E19FED17]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.createFullSnapshot(AbstractSnapshotIntegTestCase.java:488)
	at org.opensearch.snapshots.DeleteSnapshotIT.createNSnapshots(DeleteSnapshotIT.java:369)
	at org.opensearch.snapshots.DeleteSnapshotIT.testDeleteMultipleShallowCopySnapshotsCase1(DeleteSnapshotIT.java:128)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.DeleteSnapshotIT.testRemoteStoreCleanupForDeletedIndex" -Dtests.seed=89E1AFC9EA8EC59C
./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.DeleteSnapshotIT.testRemoteStoreCleanupForDeletedIndex" -Dtests.seed=89E1AFC9EA8EC59C

Expected behavior
Tests must always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • CI

Additional context
Came out in #7497

CC @kasundra07

@dblock
Copy link
Member

dblock commented Oct 6, 2023

#10388 (comment)

@rramachand21
Copy link
Member

@harishbhakuni , is this issue fixed? if not, are we planning this for 2.13? Lets add the untriaged label back if we do not have clarity and we can review it during the triage and backlog review meeting.

@shwetathareja
Copy link
Member

Failure at different step

java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
	at __randomizedtesting.SeedInfo.seed([D7D594AAC34E2463:92C2912F0AD87EC2]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1118)
	at org.opensearch.snapshots.DeleteSnapshotIT.testRemoteStoreCleanupForDeletedIndex(DeleteSnapshotIT.java:336)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more
	Suppressed: java.lang.AssertionError: 
Expected: a value equal to <0>
     but: <19> was greater than <0>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.snapshots.DeleteSnapshotIT.lambda$testRemoteStoreCleanupForDeletedIndex$1(DeleteSnapshotIT.java:338)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1106)
		... 40 more

@shwetathareja
Copy link
Member

#12252 (comment)

@akolarkunnu
Copy link
Contributor

An another test case(testDeleteSnapshot) failure from the same test class:

https://build.ci.opensearch.org/job/gradle-check/39694/testReport/junit/org.opensearch.snapshots/DeleteSnapshotIT/testDeleteSnapshot/

java.lang.AssertionError: shard [index-1][5] is not locked
at __randomizedtesting.SeedInfo.seed([E4DF0BE046504A99]:0)
at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1244)
at org.opensearch.index.IndexService.onShardClose(IndexService.java:719)
at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:842)
at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:829)
at org.opensearch.index.store.Store.closeInternal(Store.java:573)
at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
at org.opensearch.index.store.Store.decRef(Store.java:546)
at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1777)
at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1753)
at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4634)
at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1067)
at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1211)
at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Storage:Snapshots
Projects
Status: 🆕 New
Development

No branches or pull requests

9 participants