[CI] TimeSeriesRateAggregatorTests fails with file handle leak

[cbuescher]

CI Link

https://gradle-enterprise.elastic.co/s/4r6lsclc3lrou/tests/overview?outcome=FAILED

Repro line

./gradlew ":x-pack:plugin:analytics:test" --tests "org.elasticsearch.xpack.analytics.rate.TimeSeriesRateAggregatorTests" -Dtests.seed=509E2AFA17E6D235 -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=22

Does it reproduce?

Yes

Applicable branches

main

Failure history

No response

Failure excerpt

First seen on Lucene Snapshot brand now merged into "main", reproduced for this particular seed:

1> [2024-10-21T14:15:12,018][INFO ][o.e.x.a.r.TimeSeriesRateAggregatorTests] [testNestedWithinAutoDateHistogram] after test
  2> java.lang.RuntimeException: file handle leaks: [FileChannel(/Users/christoph/workspace/elasticsearch/x-pack/plugin/analytics/build/testrun/test/temp/org.elasticsearch.xpack.analytics.rate.TimeSeriesRateAggregatorTests_509E2AFA17E6D235-001/index-MMapDirectory-001/write.lock)]
        at __randomizedtesting.SeedInfo.seed([509E2AFA17E6D235]:0)
        at org.apache.lucene.tests.mockfile.LeakFS.onClose(LeakFS.java:63)
        at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:69)
        at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:70)
        at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:70)
        at org.apache.lucene.tests.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:223)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1570)

        Caused by:
        java.lang.Exception
            at org.apache.lucene.tests.mockfile.LeakFS.onOpen(LeakFS.java:46)
            at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
            at org.apache.lucene.tests.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:202)
            at org.apache.lucene.tests.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:171)
            at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:206)
            at java.base/java.nio.channels.FileChannel.open(FileChannel.java:309)
            at java.base/java.nio.channels.FileChannel.open(FileChannel.java:369)
            at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:112)
            at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
            at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
            at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
            at org.apache.lucene.tests.store.MockDirectoryWrapper.obtainLock(MockDirectoryWrapper.java:1105)
            at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:955)
            at org.apache.lucene.tests.index.RandomIndexWriter$2.<init>(RandomIndexWriter.java:109)
            at org.apache.lucene.tests.index.RandomIndexWriter.mockIndexWriter(RandomIndexWriter.java:108)
            at org.apache.lucene.tests.index.RandomIndexWriter.mockIndexWriter(RandomIndexWriter.java:78)
            at org.apache.lucene.tests.index.RandomIndexWriter.<init>(RandomIndexWriter.java:162)
            at org.apache.lucene.tests.index.RandomIndexWriter.<init>(RandomIndexWriter.java:141)
            at org.elasticsearch.search.aggregations.AggregatorTestCase.testCase(AggregatorTestCase.java:754)
            at org.elasticsearch.xpack.analytics.rate.TimeSeriesRateAggregatorTests.lambda$testNestedWithinAutoDateHistogram$6(TimeSeriesRateAggregatorTests.java:158)
            at org.apache.lucene.tests.util.LuceneTestCase._expectThrows(LuceneTestCase.java:3044)
            at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2884)
            at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2875)
            at org.elasticsearch.xpack.analytics.rate.TimeSeriesRateAggregatorTests.testNestedWithinAutoDateHistogram(TimeSeriesRateAggregatorTests.java:158)

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[cbuescher]

Without the random seed you need to run a few hundred iterations to run into this but it still reproduces. I think the problem can be isolated to "testNestedWithinAutoDateHistogram", in particular the last part of the test that checks for an IllegalArgumentException being thrown when indexing documents. I will open a fix shortly.

[cbuescher]

Digging in some more I think we are getting a different IllegalArgumentException now that before the Lucene 10 merge.
While with commit 8efd08b~1 the IAE message when printed out reads:
"Wrapping a time-series rate aggregation within a DeferableBucketAggregator is not supported. Consider using an alternative outer aggregation type to avoid this, e.g. date_histogram instead of auto_date_histogram." and comes from the TimeSeriesRateAggregator ctor, with Lucene 10 we now get an IAE that reads "a parent field must be set in order to use document blocks with index sorting; see IndexWriterConfig#setParentField" and comes from CheckIndex

[javanna]

We seem hit additional validation for a scenario that was permitted before Lucene 10 and no longer allowed with Lucene 10. I wonder if that is exactly what this test it testing, in which case we change the assertion, or whether there is still a way to get the previous error by tweaking the test. We can also hand it over to the the area team that knows best about that specific test if needed.