[CI] LookupFromIndexOperatorTests testSimpleFinishClose failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #11101 / sles-15_platform-support-unix
• elasticsearch-periodic-platform-support #11080 / oraclelinux-9_platform-support-unix

Reproduction Line:

./gradlew ":x-pack:plugin:esql:test" --tests "org.elasticsearch.xpack.esql.enrich.LookupFromIndexOperatorTests.testSimpleFinishClose" -Dtests.seed=E89B737ADC0AE170 -Dtests.locale=ccp-BD -Dtests.timezone=Pacific/Fakaofo -Druntime.java=25

Applicable branches:
8.18

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 2 open files: {_1.si=1, segments_3=1}

Issue Reasons:

• [8.18] 2 failures in test testSimpleFinishClose (1.9% fail rate in 106 executions)
• [8.18] 2 failures in pipeline elasticsearch-periodic-platform-support (50.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[astefan]

REPRODUCE WITH: ./gradlew ":x-pack:plugin:esql:test" --tests "org.elasticsearch.xpack.esql.enrich.LookupFromIndexOperatorTests.testSimpleFinishClose" -Dtests.seed=E89B737ADC0AE170 -Dtests.locale=ccp-BD -Dtests.timezone=Pacific/Fakaofo -Druntime.java=25

LookupFromIndexOperatorTests > testSimpleFinishClose FAILED
    java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 2 open files: {_1.si=1, segments_3=1}
        at __randomizedtesting.SeedInfo.seed([E89B737ADC0AE170:BAD66B33CB8E1622]:0)
        at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:885)
        at org.elasticsearch.core.IOUtils.close(IOUtils.java:71)
        at org.elasticsearch.xpack.esql.enrich.LookupFromIndexOperatorTests.closeIndex(LookupFromIndexOperatorTests.java:258)

        Caused by:
        java.lang.RuntimeException: unclosed IndexInput: segments_3
            at org.apache.lucene.tests.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:783)
            at org.apache.lucene.tests.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:844)
            at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156)
            at org.apache.lucene.tests.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1130)
            at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:310)
            at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:88)
            at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77)
            at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:828)
            at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
            at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:67)
            at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:60)
            at org.elasticsearch.xpack.esql.enrich.LookupFromIndexOperatorTests.lambda$lookupShardContextFactory$3(LookupFromIndexOperatorTests.java:239)

[astefan]

I cannot repro this. @dnhatn do you have any clues as to what this is about? (it's on 8.18)

[dnhatn]

This was fixed in #128162. For async operators, tests should wait for async actions to complete before closing resources

[dnhatn]

Fixed in #138447