-
Notifications
You must be signed in to change notification settings - Fork 24.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Handle parallel calls to createWeight when profiling is on (#108041)
We disable inter-segment concurrency in the query phase whenever profile is on, because there are known concurrency issues that need fixing. The way we disable concurrency is by creating a single slice that search will execute against. We still offload the execution to the search workers thread pool. Inter-segment concurrency in Lucene is though not always based on slices. Knn query (as well as terms enum loading and other places) parallelizes across all segments independently of slices that group multiple segments together. That behavior is not easy to disable unless you don't set the executor to the searcher, in which case though you entirely disable using the separate executor for potentially heavy CPU/IO based loads which is not desirable. That means that when executing a knn query, it will execute in parallel (in DFS as well as in the query phase) no matter if inter-segment concurrency has been disabled because profiling is on. When using pre-filtering, there are queries like multi term queries that will call createWeight from each segment, in parallel, when pulling the scorer. That causes non-deterministic behavior as the profiler does not support concurrent access to some of its data structures. This commit protects the profiler from concurrent access to its data structures by synchronizing access to its tree. Performance is not a concern here, as profiler is already known to slow down query execution. Closes #104235 Closes #104131
- Loading branch information
Showing
4 changed files
with
16 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
pr: 108041 | ||
summary: Handle parallel calls to `createWeight` when profiling is on | ||
area: Search | ||
type: bug | ||
issues: | ||
- 104131 | ||
- 104235 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters