Improve scroll search by using IndexSearcher#searchAfter(...) #4968
Conversation
| @@ -134,6 +136,78 @@ private static long optionalSum(long left, long right) { | |||
| return Math.min(left, right) == -1 ? -1 : left + right; | |||
| } | |||
|
|
|||
| public ScoreDoc[] sortDocsForScroll(AtomicArray<? extends QuerySearchResultProvider> resultsArr) throws IOException { | |||
There was a problem hiding this comment.
The body of this method is very similar to sortDocs, would it possible somehow to share more code with sortDocs?
There was a problem hiding this comment.
It's not that similar. The sortDocs method need to take from into account all the time, while the sortDocsForScroll method never needs to.
|
Updated PR that includes @jpountz and @spinscale feedback points. |
| if (lastEmittedDocPerShard[scoreDoc.shardIndex] == null) { | ||
| lastEmittedDocPerShard[scoreDoc.shardIndex] = scoreDoc; | ||
| } | ||
| } |
There was a problem hiding this comment.
Maybe it would be simpler by iterating forward as this would remove the need for a null check:
for (int i = 0; i < sortedShardList.length; ++i) {
ScoreDoc scoreDoc = sortedShardList[i];
lastEmittedDocPerShard[scoreDoc.shardIndex] = scoreDoc;
}
|
@jpountz Your comments make sense, I updated the PR with another commit that addresses your comments. |
| this.docIds = list.buffer; | ||
| this.size = list.size(); | ||
| this.lastEmittedDoc = lastEmittedDoc; | ||
| } |
There was a problem hiding this comment.
Could the other constructor call this one?
|
This pull request looks good in general:
However, there are a few places where the |
|
LGTM |
…of regular search methods which rely on `from` for pagination. This prevents the creation of priority queues of `from + size`, instead the size of the priority queue will always be equal to `size`. Closes elastic#4940
clintongormley
added
>enhancement
v2.0.0-beta1
v1.2.0
:Search/Search
PR for #4940