[BUG] org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex is flaky

[ashking94]

Describe the bug
org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex test is flaky.

To Reproduce
./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex" -Dtests.seed=CA129BD4A6BD4BD1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-HK -Dtests.timezone=CNT -Druntime.java=20

Expected behavior
Test should pass

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Jenkins build failure link - https://build.ci.opensearch.org/job/gradle-check/21872/

[ashking94]

failure - #8758 (comment)

[ashking94]

Other instances mentioned in #8279

[sachinpkale]

@harishbhakuni Can you please take a look at this bug?

[peternied]

Impacted pull request #9357, see jenkins testReport

Repro

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex" -Dtests.seed=DF962FC54758F6F7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=et-EE -Dtests.timezone=Etc/GMT-4 -Druntime.java=17

Error

java.lang.AssertionError: doc with id0 is not present for index testindex1
	at __randomizedtesting.SeedInfo.seed([DF962FC54758F6F7:276070C607309758]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.opensearch.remotestore.RemoteRestoreSnapshotIT.assertDocsPresentInIndex(RemoteRestoreSnapshotIT.java:95)
	at org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex(RemoteRestoreSnapshotIT.java:355)
	```

[anasalkouz]

@harishbhakuni what is the update on this?

[harishbhakuni]

@anasalkouz yes i am looking into this, i was not able to reproduce this locally. ran it for 150+ runs locally and it worked fine. but will fix this.

[kotwanikunal]

Just saw this in #9915

[linuxpi]

@BhumikaSaini-Amazon can you take a look at this test?

[harishbhakuni]

this is an old issue which got fixed in 2.11. ran the test for 100+ runs without any failures.