[BUG] CreateRemoteIndexClusterDefaultDocRepIT.testDefaultRemoteStoreNoUserOverrideExceptReplicationTypeSegment is flaky

[mch2]

Describe the bug
From https://build.ci.opensearch.org/job/gradle-check/31138/
PR: #11534

[linuxpi]

Not able to repro failure in 3000 iterations.

[linuxpi]

closing this due to lack of occurrence and repro. Please reopen if you find any failures

[reta]

The issue is not gone:

• https://build.ci.opensearch.org/job/gradle-check/33717/testReport/junit/org.opensearch.remotestore/CreateRemoteIndexClusterDefaultDocRepIT/testDefaultRemoteStoreNoUserOverrideExceptReplicationTypeSegment/

[linuxpi]

Thanks @reta for reopening this. I need to think a bit more on how to repro this, the build logs are not enough to debug

[linuxpi]

Looks like an issue while test tearDown

java.lang.AssertionError: shard [test-idx-1][0] is not locked
	at __randomizedtesting.SeedInfo.seed([37532D70A3B76622]:0)
	at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
	at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1231)
	at org.opensearch.index.IndexService.onShardClose(IndexService.java:678)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:801)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:788)
	at org.opensearch.index.store.Store.closeInternal(Store.java:573)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:546)
	at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1777)
	at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1753)
	at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4498)
	at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1026)
	at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1170)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)