Add additional shards routing info in ShardSearchRequest #29533

jimczi · 2018-04-16T13:55:42Z

This commit propagates the preference and routing of the original SearchRequest in the ShardSearchRequest.
This information is then use to fix a bug in sliced scrolls when executed with a preference (or a routing).
Instead of computing the slice query from the total number of shards in the index, this commit computes this number from the number of shards per index that participates in the request.

Fixes #27550

This commit adds two new methods to ShardSearchRequest: * #numberOfShardsIndex() that returns the number of shards of this index that participates in the request. * #remapShardId() that returns the remapped shard id of this shard for this request. The remapped shard id is the id of the requested shard among all shards of this index that are part of the request. Note that the remapped shard id is equal to the original shard id if all shards of this index are part of the request. These informations are useful when the _search is executed with a preference (or a routing) that restricts the number of shards requested for an index. This change allows to fix a bug in sliced scrolls executed with a preference (or a routing). Instead of computing the slice query from the total number of shards in the index, this change allows to compute this number from the number of shards per index that participates in the request. Fixes elastic#27550

elasticmachine · 2018-04-16T13:55:44Z

Pinging @elastic/es-search-aggs

s1monw

left 2 comments / ideas

s1monw · 2018-04-19T07:16:35Z

server/src/main/java/org/elasticsearch/search/internal/ShardSearchRequest.java

+     * of this index that are part of the request. Note that the remapped shard id
+     * is equal to the original shard id if all shards of this index are part of the request.
+     */
+    int remapShardId();


I wonder if we should call it shardRequestOrdinal shard ID is trappy at least for me since I alwasy think of ShardID.java when I read it.

+1. I pushed b90716c

s1monw · 2018-04-19T07:25:17Z

server/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java

@@ -318,8 +319,11 @@ public final ShardSearchTransportRequest buildShardSearchRequest(SearchShardIter
        AliasFilter filter = aliasFilter.get(shardIt.shardId().getIndex().getUUID());
        assert filter != null;
        float indexBoost = concreteIndexBoosts.getOrDefault(shardIt.shardId().getIndex().getUUID(), DEFAULT_INDEX_BOOST);
-        return new ShardSearchTransportRequest(shardIt.getOriginalIndices(), request, shardIt.shardId(), getNumShards(),
-            filter, indexBoost, timeProvider.getAbsoluteStartMillis(), clusterAlias);
+        int[] indexShards = getIndexShards(shardIt.shardId().getIndex());


this is quite a complex operation since we call if for every shard and then do consume the entire iterator again. I wonder if we can pre-sort the ShardRoutings in SearchShardIterator and then calculate this on the fly and simply call SearchShardIterator#getIndexShardOrdinal() to get it?

I changed the logic to compute the needed informations only once in InitialSearchPhase constructor. I need the complete GroupShardsIterator to do so which is why it's not in SearchShardIterator but it's the same idea:
b90716c

Rename remapShardId to shardRequestOrdinal. Compute request ordinal per shard and number of request shard per index only once.

Propagate preference and routing in the ShardSearchRequest in order to compute the shard ordinal and number of requested shard lazily on each node. The computation is done in the slice builder only when preference or routing are set on the original search request.

jimczi · 2018-04-20T10:09:03Z

We discussed with Simon and decided to propagate the routing and preference in the ShardSearchRequest (rather than computing the request shard ordinal and number of shards in the coordinating node). The slice builder can use this information to compute the needed information lazily. @s1monw can you take another look ?

s1monw · 2018-04-20T12:57:58Z

server/src/main/java/org/elasticsearch/search/internal/ShardSearchLocalRequest.java

@@ -250,7 +268,12 @@ protected void innerWriteTo(StreamOutput out, boolean asKey) throws IOException
        if (out.getVersion().onOrAfter(Version.V_6_3_0)) {
            out.writeOptionalBoolean(allowPartialSearchResults);
        }
-
+        if (!asKey) {


can we use == false?

s1monw · 2018-04-20T12:58:35Z

server/src/main/java/org/elasticsearch/search/slice/SliceBuilder.java

+             * slices use the same numbers.
+             */
+            GroupShardsIterator<ShardIterator> group = buildShardIterator(clusterService, request);
+            assert group.size() <= numShards;


give this assert a message it's a bummer when it fails and we don't have one

s1monw · 2018-04-20T12:59:30Z

server/src/main/java/org/elasticsearch/search/slice/SliceBuilder.java

+            numShards = group.size();
+            int ord = 0;
+            shardId = -1;
+            for (ShardIterator it : group) {


can you leave an inline comment what we are doing here?

s1monw · 2018-04-20T13:00:13Z

server/src/main/java/org/elasticsearch/search/slice/SliceBuilder.java

+        final Settings settings = clusterService.getSettings();
+        final String[] indices;
+        if (request instanceof IndicesRequest) {
+            indices = ((IndicesRequest) request).indices();


should this always be the index name?

oh now I see. Damned I didn't think of Aliases. If you have a routing alias we have a race condition here. I think we can't do it this way. The alias might change on the way to the shard which will cause wrong results. I think that renders my idea here as invalid?

Maybe we can change the ShardSearchRequest and instead of sending the global routing for the request we send the list of extracted routings for the requested index. ShardSearchRequest#routings and when there is no alias routing it returns the global routing in a singleton ?

yeah I think the routing needs to be resolved on the coordinator

s1monw

this looks great! thanks for the extra iterations!!!

This commit propagates the preference and routing of the original SearchRequest in the ShardSearchRequest. This information is then use to fix a bug in sliced scrolls when executed with a preference (or a routing). Instead of computing the slice query from the total number of shards in the index, this commit computes this number from the number of shards per index that participates in the request. Fixes #27550

Relates #29533

* 6.x: Watcher: Ensure mail message ids are unique per watch action (#30112) SQL: Correct error message (#30138) SQL: Add BinaryMathProcessor to named writeables list (#30127) Tests: Use buildDir as base for generated-resources (#30191) Fix SliceBuilderTests#testRandom failures Fix edge cases in CompositeKeyExtractorTests (#30175) Document time unit limitations for date histograms (#30177) Remove licenses missed by the migration [DOCS] Updates docker installation package details (#30110) Fix TermsSetQueryBuilder.doEquals() method (#29629) [Monitoring] Remove unhelpful Monitoring tests (#30144) [Test] Fix RenameProcessorTests.testRenameExistingFieldNullValue() (#29655) [test] include oss tar in packaging tests (#30155) TEST: Update settings should go through cluster state (#29682) Add additional shards routing info in ShardSearchRequest (#29533) Reinstate missing documentation (#28781) Clarify documentation of scroll_id (#29424) Fixes Eclipse build for sql jdbc project (#30114) Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)

jimczi added >bug :Search/Search Search-related issues that do not fall into other categories v7.0.0 v6.3.0 labels Apr 16, 2018

jimczi requested a review from s1monw April 16, 2018 13:55

s1monw requested changes Apr 19, 2018

View reviewed changes

jimczi added 4 commits April 19, 2018 15:30

Address reviews

b90716c

Rename remapShardId to shardRequestOrdinal. Compute request ordinal per shard and number of request shard per index only once.

Merge branch 'master' into bug/slice_with_routing

a16e22b

Address reviews

6fb78b3

Propagate preference and routing in the ShardSearchRequest in order to compute the shard ordinal and number of requested shard lazily on each node. The computation is done in the slice builder only when preference or routing are set on the original search request.

Merge branch 'master' into bug/slice_with_routing

2ef8f66

cosmetics

9fedb3b

s1monw reviewed Apr 20, 2018

View reviewed changes

jimczi added 5 commits April 23, 2018 13:38

send resolved routing values for the index in shard search request

8289993

Merge branch 'master' into bug/slice_with_routing

02a91a4

fix ut

ca87ce7

fix random routings value (null is not allowed)

36ac087

handle empty routing

803c88a

s1monw approved these changes Apr 24, 2018

View reviewed changes

jimczi added v6.4.0 and removed v6.3.0 labels Apr 26, 2018

jimczi merged commit 8b8c0c0 into elastic:master Apr 26, 2018

jimczi deleted the bug/slice_with_routing branch April 26, 2018 07:58

jimczi added a commit that referenced this pull request Apr 26, 2018

Adjust serialization versions after backport

752ba2f

Relates #29533

jimczi mentioned this pull request Apr 27, 2018

update the Slice bean which id and max should be accumulate in global elastic/elasticsearch-hadoop#1141

Closed

jimczi mentioned this pull request Oct 2, 2018

Connector uncorrectly determines the number of partitions elastic/elasticsearch-hadoop#1196

Closed

1 task

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add additional shards routing info in ShardSearchRequest #29533

Add additional shards routing info in ShardSearchRequest #29533

jimczi commented Apr 16, 2018 •

edited

Loading

elasticmachine commented Apr 16, 2018

s1monw left a comment

s1monw Apr 19, 2018

jimczi Apr 19, 2018

s1monw Apr 19, 2018

jimczi Apr 19, 2018

jimczi commented Apr 20, 2018

s1monw Apr 20, 2018

s1monw Apr 20, 2018

s1monw Apr 20, 2018

s1monw Apr 20, 2018

s1monw Apr 20, 2018

jimczi Apr 20, 2018

s1monw Apr 21, 2018

s1monw left a comment

Add additional shards routing info in ShardSearchRequest #29533

Add additional shards routing info in ShardSearchRequest #29533

Conversation

jimczi commented Apr 16, 2018 • edited Loading

elasticmachine commented Apr 16, 2018

s1monw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimczi commented Apr 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw left a comment

Choose a reason for hiding this comment

jimczi commented Apr 16, 2018 •

edited

Loading