[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}} is flaky

[harshavamsi]

Describe the bug

java.lang.AssertionError: shard [test][2] is not locked
	at __randomizedtesting.SeedInfo.seed([E30F90427F5357C9]:0)
	at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
	at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1244)
	at org.opensearch.index.IndexService.onShardClose(IndexService.java:719)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:842)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:829)
	at org.opensearch.index.store.Store.closeInternal(Store.java:573)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:546)
	at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1777)
	at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1753)
	at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4633)
	at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1067)
	at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1211)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Build

To Reproduce

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.store.IndicesStoreIntegrationIT" -Dtests.method="testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}}" -Dtests.seed=E30F90427F5357C9 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hr-HR -Dtests.timezone=America/Argentina/Jujuy -Druntime.java=21
NOTE: test params are: codec=Asserting(Lucene99): {index_uuid=PostingsFormat(name=Asserting), type=PostingsFormat(name=Asserting)}, docValues:{}, maxPointsInLeafNode=1663, maxMBSortInHeap=6.680419868828231, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=hr-HR, timezone=America/Argentina/Jujuy
NOTE: Linux 5.15.0-1056-aws amd64/Eclipse Adoptium 21.0.3 (64-bit)/cpus=32,threads=1,free=120826952,total=622854144
NOTE: All tests run in this JVM: [ClusterStatsIT, IndicesExistsIT, BulkRejectionIT, RecoveryWithUnsupportedIndicesIT, SimpleAllocationIT, WeightedRoutingIT, DiscoveryDisruptionIT, RecoverAfterNodesIT, InternalEngineMergeIT, CorruptedFileIT, SimpleGetFieldMappingsIT, SegmentReplicationResizeRequestIT, IndicesStoreIntegrationIT]

Expected behavior

Tests pass

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[peternied]

• Duplicate of [BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788