[BUG] org.opensearch.search.sort.FieldSortIT.testSimpleSorts {p0={"search.concurrent_segment_search.enabled":"true"}} if flaky

[reta]

Describe the bug

The test case org.opensearch.search.sort.FieldSortIT.testSimpleSorts {p0={"search.concurrent_segment_search.enabled":"true"}} is flaky:

Failed to execute phase [query], all shards failed; shardFailures {[Di7AFLIFTB20nfl7X5doBg][test][0]: RemoteTransportException[[node_s0][127.0.0.1:42601][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: UnsupportedOperationException; }

Failed to execute phase [query], all shards failed; shardFailures {[Di7AFLIFTB20nfl7X5doBg][test][0]: RemoteTransportException[[node_s0][127.0.0.1:42601][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: UnsupportedOperationException; }
	at __randomizedtesting.SeedInfo.seed([90490F3B9D1AFC66:D232EAF3A97C8078]:0)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:718)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:379)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:757)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:511)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:301)
	at app//org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104)
	at app//org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:75)
	at app//org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:755)
	at app//org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81)
	at app//org.opensearch.transport.TransportService$9.handleException(TransportService.java:1690)
	at app//org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1476)
	at app//org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1590)
	at app//org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1564)
	at app//org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:81)
	at app//org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:73)
	at app//org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70)
	at app//org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:104)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54)
	at app//org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at app//org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base@21.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@21.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@21.0.1/java.lang.Thread.run(Thread.java:1583)
Caused by: OpenSearchException; nested: UnsupportedOperationException;
	at app//org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:708)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:377)
	... 24 more
Caused by: java.lang.UnsupportedOperationException
	at org.opensearch.index.fielddata.AbstractNumericDocValues.advance(AbstractNumericDocValues.java:57)
	at org.apache.lucene.search.comparators.NumericComparator$NumericLeafComparator$2.advance(NumericComparator.java:407)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:286)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:236)
	at org.opensearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:71)
	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38)
	at org.opensearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:327)
	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:283)
	at org.apache.lucene.search.IndexSearcher.lambda$search$2(IndexSearcher.java:721)
	at org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 8 more

Related component

Search

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.sort.FieldSortIT" -Dtests.method="testSimpleSorts {p0={"search.concurrent_segment_search.enabled":"true"}}" -Dtests.seed=90490F3B9D1AFC66

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context
Add any other context about the problem here.

[reta]

@mch2 @msfroh I am not sure this is related to Apache Lucene 9.9.1, just want to bring it to you attention folks

[peternied]

[Triage - attendees 1 2 3 4]
Thanks for filing

[jed326]

Able to reproduce this 100% with the provided test seed:

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.sort.FieldSortIT" -Dtests.method="testSimpleSorts {p0={"search.concurrent_segment_search.enabled":"true"}}" -Dtests.seed=90490F3B9D1AFC66 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv -Dtests.timezone=America/Virgin -Druntime.java=21

Exception is coming from here:

OpenSearch/server/src/main/java/org/opensearch/index/fielddata/AbstractNumericDocValues.java

Lines 55 to 58 in 7b75fb4

	@Override
	public int advance(int target) throws IOException {
	throw new UnsupportedOperationException();
	}

This comes from the searchLeaf path which should look the same for both concurrent and non-concurrent search so next step is to figure out why it's only failing for concurrent search enabled.

[jed326]

This is (one of) the problematic queries:

OpenSearch/server/src/internalClusterTest/java/org/opensearch/search/sort/FieldSortIT.java

Line 815 in b18ee7d

searchResponse = client().prepareSearch().setQuery(matchAllQuery()).setSize(size).addSort("half_float_value", SortOrder.DESC).get();

[reta]

This is the problematic query:

This is interesting, I think we handle HALF_FLOAT / UNSIGNED_LONG differently but I am wondering how come the test fails from time to time?

[jed326]

This is interesting, I think we handle HALF_FLOAT / UNSIGNED_LONG differently but I am wondering how come the test fails from time to time?

It looks like we encounter the problem with sort for, double_value, and unsigned_long_value as well.

The problem seems to be related to this change: apache/lucene#12405
Specifically: apache/lucene@d910990#diff-79c6a57519ecd1ef504629e62e13d17859a4ffedc58f4602e583ce758a15adc8R291-R295

As the comparator is getting set to NumericComparator::NumericLeafComparator. In the non-concurrent search case we don't go into the if statement so the competitiveIterator is not updated.

The if statement seems to be related to the number of documents on the segment though so it seems like we could still see this in the non-concurrent search case.

[jed326]

It seems to me the naive solution here is to change this:

OpenSearch/server/src/main/java/org/opensearch/index/fielddata/fieldcomparator/HalfFloatValuesComparatorSource.java

Line 50 in 4a42150

return new HalfFloatComparator(numHits, fieldname, fMissingValue, reversed, filterPruning(pruning)) {

to use Pruning.None which I think means we won't update the competitiveIterator to prune results. That seems like there would be performance regression implications though. That's basically reverting the changes from #8168

@reta @mch2 as you are more familiar with the Lucene 9.9 upgrade would you mind giving a second set of eyes here?

[reta]

@reta @mch2 as you are more familiar with the Lucene 9.9 upgrade would you mind giving a second set of eyes here?

Sure, I will take a look shortly, thank you @jed326