Switch ContextIndexSearcher to use Lucene's TaskExecutor #101537
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have contributed back to Lucene the changes we had made around running concurrent tasks. These include waiting for all tasks to finish when an exception is thrown, as well as not starting tasks when one of the previously run tasks throws an exception. The execution of concurrent tasks is now generalized within Lucene and exposed through a TaskExecutor that can be retrieved from the IndexSearcher and used to run tasks. We still have customizations that require us to override some of the search method, but with this change we rely on standard Lucene's behaviour for running concurrent tasks.
javanna
added
:Search/Search
|
Pinging @elastic/es-search (Team:Search) |
javanna
commented
Oct 30, 2023
| search(Arrays.asList(leaves), weight, collector); | ||
| if (timeExceeded) { | ||
| for (Future<?> future : listTasks) { | ||
| FutureUtils.cancel(future); |
There was a problem hiding this comment.
we used to have cancellation on timeout as well. This is not currently implemented in Lucene (we cancel only on exceptions), and looking further at implementing that, I realized that it's perhaps not needed, because if one task times out, it is expected that others will do too, as the timeout impl is shared across the different tasks. That is even better, as it will make running tasks stop too, which is something that tasks cancellation does not support yet.
romseygeek
approved these changes
Oct 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have contributed back to Lucene the changes we had made around running
concurrent tasks. These include waiting for all tasks to finish when an
exception is thrown, as well as not starting tasks when one of the
previously run tasks throws an exception.
The execution of concurrent tasks is now generalized within Lucene and
exposed through a TaskExecutor that can be retrieved from the
IndexSearcher and used to run tasks. We still have customizations that
require us to override some of the search method, but with this change
we rely on standard Lucene's behaviour for running concurrent tasks.