Add sort and collapse info to SearchHits transport serialization #36555
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
javanna
added
>enhancement
:Search/Search
|
Pinging @elastic/es-search |
11 tasks
|
retest this please |
…Writeable We already have writeArray(Writer, array), writeArray(Writeable[]), writeOptionalArray(Writeable[]), but we were missing this one.
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 13, 2018
This was referenced Dec 13, 2018
s1monw
approved these changes
Dec 14, 2018
server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
Show resolved
Hide resolved
| @@ -577,6 +564,21 @@ public static SortField readSortField(StreamInput in) throws IOException { | |||
| return sortField; | |||
| } | |||
|
|
|||
| public static SortField[] readSortFields(StreamInput in) throws IOException { | |||
| SortField[] fields = new SortField[in.readVInt()]; | |||
There was a problem hiding this comment.
can't you use writeArray with a writer? I think that would be good instead of iterating manually. same on teh read end
There was a problem hiding this comment.
yes, not sure why I missed this ;P
javanna
force-pushed
the
enhancement/search_hits_add_info
branch
from
03e4965
to
0bc1858
Compare
…thod can be removed as it's a one liner
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 17, 2018
…stic#36555) In order for CCS alternate execution mode (see elastic#32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of the `SearchHits`, specifically: - lucene `SortField[]` which contains info about the fields that sorting was performed on and their type, which depends on mappings (that the CCS node does not know about) - collapse field (`String`) that field collapsing was executed on, if requested - collapse values (`Object[]`) that field collapsing was based on, if requested This info is needed to be able to reconstruct the `TopFieldDocs` or `CollapseFieldTopDocs` in the CCS coordinating node to feed the `mergeTopDocs` method and reduce multiple search responses received (one per cluster) into one. This commit adds such information to the `SearchHits` class. It's nullable info that is not serialized through the REST layer. `SearchPhaseController` sets such info at the end of the hits reduction phase.
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 17, 2018
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 17, 2018
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 17, 2018
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 18, 2018
…stic#36555) In order for CCS alternate execution mode (see elastic#32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of the `SearchHits`, specifically: - lucene `SortField[]` which contains info about the fields that sorting was performed on and their type, which depends on mappings (that the CCS node does not know about) - collapse field (`String`) that field collapsing was executed on, if requested - collapse values (`Object[]`) that field collapsing was based on, if requested This info is needed to be able to reconstruct the `TopFieldDocs` or `CollapseFieldTopDocs` in the CCS coordinating node to feed the `mergeTopDocs` method and reduce multiple search responses received (one per cluster) into one. This commit adds such information to the `SearchHits` class. It's nullable info that is not serialized through the REST layer. `SearchPhaseController` sets such info at the end of the hits reduction phase.
javanna
added a commit
that referenced
this pull request
Dec 18, 2018
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 18, 2018
Now that elastic#36555 is backported to 6.x, the version in the transport serialization conditionals for sort and collapse info can be updated to 6.6.0
javanna
added a commit
that referenced
this pull request
Dec 18, 2018
) In order for CCS alternate execution mode (see #32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of the `SearchHits`, specifically: - lucene `SortField[]` which contains info about the fields that sorting was performed on and their type, which depends on mappings (that the CCS node does not know about) - collapse field (`String`) that field collapsing was executed on, if requested - collapse values (`Object[]`) that field collapsing was based on, if requested This info is needed to be able to reconstruct the `TopFieldDocs` or `CollapseFieldTopDocs` in the CCS coordinating node to feed the `mergeTopDocs` method and reduce multiple search responses received (one per cluster) into one. This commit adds such information to the `SearchHits` class. It's nullable info that is not serialized through the REST layer. `SearchPhaseController` sets such info at the end of the hits reduction phase.
javanna
added a commit
that referenced
this pull request
Dec 18, 2018
Now that #36555 is backported to 6.x, the version in the transport serialization conditionals for sort and collapse info can be updated to 6.6.0
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Dec 18, 2018
* elastic/master: (31 commits) enable bwc tests and switch transport serialization version to 6.6.0 for CAS features [DOCs] Adds ml-cpp PRs to alpha release notes (elastic#36790) Synchronize WriteReplicaResult callbacks (elastic#36770) Add CcrRestoreSourceService to track sessions (elastic#36578) [Painless] Add tests for boxed return types (elastic#36747) Internal: Remove originalSettings from Node (elastic#36569) [ILM][DOCS] Update ILM API authorization docs (elastic#36749) Core: Deprecate use of scientific notation in epoch time parsing (elastic#36691) [ML] Merge the Jindex master feature branch (elastic#36702) Tests: Mute SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization Update versions in SearchSortValues transport serialization Update version in SearchHits transport serialization [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (elastic#36751) [Docs] Fix error in Common Grams Token Filter (elastic#36774) Fix rollup search statistics (elastic#36674) SQL: Fix wrong appliance of StackOverflow limit for IN (elastic#36724) [TEST] Added more logging Invalidate Token API enhancements - HLRC (elastic#36362) Deprecate types in index API (elastic#36575) Disable bwc tests until elastic#36555 backport is complete (elastic#36737) ...
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Dec 18, 2018
* elastic/master: (61 commits) enable bwc tests and switch transport serialization version to 6.6.0 for CAS features [DOCs] Adds ml-cpp PRs to alpha release notes (elastic#36790) Synchronize WriteReplicaResult callbacks (elastic#36770) Add CcrRestoreSourceService to track sessions (elastic#36578) [Painless] Add tests for boxed return types (elastic#36747) Internal: Remove originalSettings from Node (elastic#36569) [ILM][DOCS] Update ILM API authorization docs (elastic#36749) Core: Deprecate use of scientific notation in epoch time parsing (elastic#36691) [ML] Merge the Jindex master feature branch (elastic#36702) Tests: Mute SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization Update versions in SearchSortValues transport serialization Update version in SearchHits transport serialization [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (elastic#36751) [Docs] Fix error in Common Grams Token Filter (elastic#36774) Fix rollup search statistics (elastic#36674) SQL: Fix wrong appliance of StackOverflow limit for IN (elastic#36724) [TEST] Added more logging Invalidate Token API enhancements - HLRC (elastic#36362) Deprecate types in index API (elastic#36575) Disable bwc tests until elastic#36555 backport is complete (elastic#36737) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order for CCS alternate execution mode (see #32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of the
SearchHits, specifically:SortField[]which contains info about the fields that sorting was performed on and their type, which depends on mappings (that the CCS node does not know about)String) that field collapsing was executed on, if requestedObject[]) that field collapsing was based on, if requestedThis info is needed to be able to reconstruct the TopFieldDocs or CollapseFieldTopDocs in the CCS coordinating node to feed the mergeTopDocs method and reduce multiple search responses received (one per cluster) into one.
This PR adds such information to the
SearchHitsclass. It's nullable info that is not serialized through the REST layer.SearchPhaseControllersets such info at the end of the hits reduction phase.