Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure GroupShardsIterator is consistent across requests, to ensure consistent sorting #7698

Merged
merged 1 commit into from Sep 12, 2014

Conversation

Projects
None yet
3 participants
@s1monw
Copy link
Contributor

s1monw commented Sep 11, 2014

GroupShardsIterator is used in many places like the search execution
to determine which shards to query. This can hold shards of one index
as well as shards of multiple indices. The iteration order is used
to assign a per-request shard ID for each shard that is used as a
tie-breaker when scores are the same. Today the iteration order is
solely dependent on the HashMap iteration order which is undefined or
rather implementation dependent. This causes search requests to return
inconsistent results across requests if, for instance, different nodes
are coordinating the requests.

Simple queries like match_all may return results in arbitrary order
if pagination is used or may even return different results for the same
request even though there hasn't been a refresh call and preferences are
used.

@kimchy

This comment has been minimized.

Copy link
Member

kimchy commented Sep 11, 2014

LGTM

[CORE] Ensure GroupShardsIterator is consistent across requests
GroupShardsIterator is used in many places like the search execution
to determin which shards to query. This can hold shards of one index
as well as shards of multiple indices. The iteration order is used
to assigne a per-request shard ID for each shard that is used as a
tie-breaker when scores are the same. Today the iteration order is
soely depending on the HashMap iteration order which is undefined or
rather implementation dependent. This causes search requests to return
inconsistent results across requests if for instance different nodes
are coordinating the requests.

Simple queries like `match_all` may return results in arbitrary order
if pagination is used or may even return different results for the same
request even though there hasn't been a refresh call and preferences are
used.

@s1monw s1monw force-pushed the s1monw:consistent_shard_order branch to a3f2677 Sep 12, 2014

@s1monw s1monw merged commit a3f2677 into elastic:master Sep 12, 2014

@s1monw s1monw changed the title [CORE] Ensure GroupSahrdsIterator is consistent across requests [CORE] Ensure GroupShardsIterator is consistent across requests Sep 12, 2014

@s1monw s1monw deleted the s1monw:consistent_shard_order branch Sep 12, 2014

@clintongormley clintongormley changed the title [CORE] Ensure GroupShardsIterator is consistent across requests Internal: Ensure GroupShardsIterator is consistent across requests Sep 12, 2014

@clintongormley clintongormley changed the title Internal: Ensure GroupShardsIterator is consistent across requests Internal: Ensure GroupShardsIterator is consistent across requests, to ensure consistent sorting Sep 12, 2014

brwe added a commit to brwe/elasticsearch that referenced this pull request Sep 12, 2014

field name lookup: return List instead of Set for names matching a pa…
…ttern

The returned sets are only used for iterating. Therefore we might
as well return a list since this guaratees order.

This is the same effect as in
elastic#7698
The test SimpleIndexQueryParserTests#testQueryStringFieldsMatch
failed on openjdk 1.7.0_65 with
<jdk.map.althashing.threshold>0</jdk.map.althashing.threshold>

brwe added a commit that referenced this pull request Sep 26, 2014

field name lookup: return List instead of Set for names matching a pa…
…ttern

The returned sets are only used for iterating. Therefore we might
as well return a list since this guaratees order.

This is the same effect as in
#7698
The test SimpleIndexQueryParserTests#testQueryStringFieldsMatch
failed on openjdk 1.7.0_65 with
<jdk.map.althashing.threshold>0</jdk.map.althashing.threshold>

closes #7709

brwe added a commit that referenced this pull request Sep 26, 2014

field name lookup: return List instead of Set for names matching a pa…
…ttern

The returned sets are only used for iterating. Therefore we might
as well return a list since this guaratees order.

This is the same effect as in
#7698
The test SimpleIndexQueryParserTests#testQueryStringFieldsMatch
failed on openjdk 1.7.0_65 with
<jdk.map.althashing.threshold>0</jdk.map.althashing.threshold>

closes #7709

brwe added a commit that referenced this pull request Sep 26, 2014

field name lookup: return List instead of Set for names matching a pa…
…ttern

The returned sets are only used for iterating. Therefore we might
as well return a list since this guaratees order.

This is the same effect as in
#7698
The test SimpleIndexQueryParserTests#testQueryStringFieldsMatch
failed on openjdk 1.7.0_65 with
<jdk.map.althashing.threshold>0</jdk.map.althashing.threshold>

closes #7709

@clintongormley clintongormley changed the title Internal: Ensure GroupShardsIterator is consistent across requests, to ensure consistent sorting Ensure GroupShardsIterator is consistent across requests, to ensure consistent sorting Jun 7, 2015

mute pushed a commit to mute/elasticsearch that referenced this pull request Jul 29, 2015

field name lookup: return List instead of Set for names matching a pa…
…ttern

The returned sets are only used for iterating. Therefore we might
as well return a list since this guaratees order.

This is the same effect as in
elastic#7698
The test SimpleIndexQueryParserTests#testQueryStringFieldsMatch
failed on openjdk 1.7.0_65 with
<jdk.map.althashing.threshold>0</jdk.map.althashing.threshold>

closes elastic#7709
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.