Cross Cluster Search: propagate original indices per cluster #24328
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
javanna
added
:Search/Search
s1monw
requested changes
Apr 26, 2017
| throws IOException { | ||
| IndexService indexService = indicesService.indexServiceSafe(request.shardId().getIndex()); | ||
| IndexShard indexShard = indexService.getShard(request.shardId().getId()); | ||
| SearchShardTarget shardTarget = new SearchShardTarget(clusterService.localNode().getId(), indexShard.shardId()); | ||
| SearchShardTarget shardTarget = new SearchShardTarget(clusterService.localNode().getId(), indexShard.shardId(), originalIndices); |
There was a problem hiding this comment.
I don't think we need this here at all lets pass always null here to the SearchShardTarget
There was a problem hiding this comment.
thanks for bringing this up, I was not too sure about this.
| @@ -30,14 +30,14 @@ | |||
| * ShardsIterators are always returned in ascending order independently of their order at construction | |||
| * time. The incoming iterators are sorted to ensure consistent iteration behavior across Nodes / JVMs. | |||
| */ | |||
| public final class GroupShardsIterator implements Iterable<ShardIterator> { | |||
| public final class GroupShardsIterator<S extends ShardIterator> implements Iterable<S> { | |||
| @@ -112,7 +112,7 @@ private void innerRun() throws IOException { | |||
| final IntArrayList[] docIdsToLoad = searchPhaseController.fillDocIdsToLoad(numShards, reducedQueryPhase.scoreDocs); | |||
| if (reducedQueryPhase.scoreDocs.length == 0) { // no docs to fetch -- sidestep everything and return | |||
| phaseResults.stream() | |||
| .map(e -> e.queryResult()) | |||
| .map(SearchPhaseResult::queryResult) | |||
| shards.add(shardIterator); | ||
| } | ||
| for (ShardIterator shardIterator : localShardsIterator) { | ||
| shards.add(shardIterator); | ||
| List<ShardRouting> shardRoutings = new ArrayList<>(); | ||
| for (ShardRouting shardRouting : shardIterator.asUnordered()) { |
There was a problem hiding this comment.
I hate this copying here :) can we maybe do this:
- make
ShardsIterator extends Iterable<ShardRouting> - remove
ShardsIterator#asUnordered() - add
List<ShardRouting> ShardsIterator#getShardRoutings()
that way we can just reuse the list?
There was a problem hiding this comment.
loving this plan. I don't like this copying either.
|
@s1monw I addressed your comments and added some tests. Can you have another look please? |
…the original indices
…the original indices
… add getShardRoutings to it.
… add getShardRoutings to it.
javanna
force-pushed
the
fix/cross_cluster_search_original_indices
branch
from
3f690a3
to
48e6430
Compare
s1monw
approved these changes
Apr 26, 2017
| @@ -74,7 +75,12 @@ public int sizeActive() { | |||
| } | |||
|
|
|||
| @Override | |||
| public Iterable<ShardRouting> asUnordered() { | |||
| public List<ShardRouting> getShardRoutings() { | |||
| return shards; | |||
There was a problem hiding this comment.
Maybe wrap it in an unmodifiable list in the ctor? I think we should not allow mutation here?
javanna
added a commit
that referenced
this pull request
Apr 26, 2017
In case of a Cross Cluster Search, the coordinating node should split the original indices per cluster, and send over to each cluster only its own set of original indices, rather than the set taken from the original search request which contains all the indices. In fact, each remote cluster should not be aware of the indices belonging to other remote clusters.
javanna
added a commit
that referenced
this pull request
Apr 26, 2017
In case of a Cross Cluster Search, the coordinating node should split the original indices per cluster, and send over to each cluster only its own set of original indices, rather than the set taken from the original search request which contains all the indices. In fact, each remote cluster should not be aware of the indices belonging to other remote clusters.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In case of a Cross Cluster Search, the coordinating node should split the original indices per cluster, and send over to each cluster only its own set of original indices, rather than set taken from the original search request which contains all the indices.
In fact, each remote cluster should not be aware of the indices belonging to other remote clusters.