[BUG] org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod is flaky

[reta]

Describe the bug

The test case org.opensearch.action.admin.indices.create.RemoteSplitIndexIT is flaky

java.lang.RuntimeException: file handle leaks: [InputStream(/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.action.admin.indices.create.RemoteSplitIndexIT_B58713BC91CE2F3B-001/tempDir-004/repos/SbZlxzvSVn/111110111101000/RHVynzYRTKmReaBzNHE60Q/1/translog/data/2/translog-30.tlog), InputStream(/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.action.admin.indices.create.RemoteSplitIndexIT_B58713BC91CE2F3B-001/tempDir-004/repos/SbZlxzvSVn/111110111101000/RHVynzYRTKmReaBzNHE60Q/1/translog/data/2/translog-19.tlog)]
	at __randomizedtesting.SeedInfo.seed([B58713BC91CE2F3B]:0)
	at org.apache.lucene.tests.mockfile.LeakFS.onClose(LeakFS.java:63)
	at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:69)
	at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:70)
	at org.apache.lucene.tests.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:223)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception
	at org.apache.lucene.tests.mockfile.LeakFS.onOpen(LeakFS.java:46)
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.newInputStream(HandleTrackingFS.java:125)
	at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:195)
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.newInputStream(HandleTrackingFS.java:94)
	at java.base/java.nio.file.Files.newInputStream(Files.java:159)
	at org.opensearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:170)
	at org.opensearch.remotestore.translogmetadata.mocks.MockFsMetadataSupportedBlobContainer.readBlobWithMetadata(MockFsMetadataSupportedBlobContainer.java:84)
	at org.opensearch.index.translog.transfer.BlobStoreTransferService.downloadBlobWithMetadata(BlobStoreTransferService.java:207)
	at org.opensearch.index.translog.transfer.TranslogTransferManager.downloadToFS(TranslogTransferManager.java:303)
	at org.opensearch.index.translog.transfer.TranslogTransferManager.downloadTranslog(TranslogTransferManager.java:258)
	at org.opensearch.index.translog.RemoteFsTranslog.downloadOnce(RemoteFsTranslog.java:250)
	at org.opensearch.index.translog.RemoteFsTranslog.download(RemoteFsTranslog.java:216)
	at org.opensearch.index.translog.RemoteFsTranslog.download(RemoteFsTranslog.java:198)
	at org.opensearch.index.shard.IndexShard.syncTranslogFilesFromRemoteTranslog(IndexShard.java:5004)
	at org.opensearch.index.shard.IndexShard.syncRemoteTranslogAndUpdateGlobalCheckpoint(IndexShard.java:4972)
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2564)
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2534)
	at org.opensearch.index.shard.IndexShard.openEngineAndSkipTranslogRecovery(IndexShard.java:2516)
	at org.opensearch.indices.recovery.RecoveryTarget.lambda$prepareForTranslogOperations$2(RecoveryTarget.java:224)
	at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344)
	at org.opensearch.indices.recovery.RecoveryTarget.prepareForTranslogOperations(RecoveryTarget.java:213)
	at org.opensearch.indices.recovery.PeerRecoveryTargetService$PrepareForTranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:404)
	at org.opensearch.indices.recovery.PeerRecoveryTargetService$PrepareForTranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:393)
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108)
	at org.opensearch.transport.NativeMessageHandler$RequestHandler.doRun(NativeMessageHandler.java:488)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	... 1 more

Related component

Storage:Remote

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.indices.create.RemoteSplitIndexIT" -Dtests.seed=B58713BC91CE2F3B 

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/39240/testReport/junit/org.opensearch.action.admin.indices.create/RemoteSplitIndexIT/classMethod/

[peternied]

[Triage - attendees 1 2 3 4 5 6
@reta Thanks for creating this issue

[bowenlan-amzn]

https://build.ci.opensearch.org/job/gradle-check/41277/testReport/junit/org.opensearch.action.admin.indices.create/RemoteSplitIndexIT/classMethod/

[reta]

Closing in favour of #14296