[BUG] org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1 is flaky

[dblock]

Describe the bug

org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1 is flaky

#10256 (comment)

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1" -Dtests.seed=52E42E635A4E8817 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-QA -Dtests.timezone=Atlantic/Azores -Druntime.java=20

org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests > testRetrieveShallowCopySnapshotCase1 FAILED
    java.lang.AssertionError: 
    Expected: is <SUCCESS>
         but: was <PARTIAL>
        at __randomizedtesting.SeedInfo.seed([52E42E635A4E8817:D84FA7EB206944B4]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.repositories.blobstore.BlobStoreRepositoryHelperTests.createSnapshot(BlobStoreRepositoryHelperTests.java:142)
        at org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1(BlobStoreRepositoryRemoteIndexTests.java:156)

[harishbhakuni]

this test is passing with the given seed now:

./gradlew ':server:test' --tests "org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1" -Dtests.seed=52E42E635A4E8817

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Mac OS X 13.6.3 (aarch64)
  JDK Version           : 18 (Amazon Corretto JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/amazon-corretto-18.jdk/Contents/Home
  Random Testing Seed   : 52E42E635A4E8817
  In FIPS 140 mode      : false
=======================================

> Task :server:test
Jan 22, 2024 10:05:28 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/Volumes/workplace/os-fork/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/Users/hbhakuni/.gradle/wrapper/dists/gradle-8.5-all/3zlzzgtsutfj0pbojr50n2l7z/gradle-8.5/lib/plugins/gradle-testing-base-8.5.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

this is also same issue fixed as part of #10217 .
also, ran it locally 100+ times without any failure.

[shwetathareja]

Test failed again in backport PR here - https://build.ci.opensearch.org/job/gradle-check/33283/

[peternied]

[Triage - attendees 1 2 3 4 5]
@dblock Thanks for filing this issue.