Add heuristics to compute pre_filter_shard_size when unspecified #53873
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit changes the pre_filter_shard_size default from 128 to unspecified. This allows to apply heuristics based on the request and the target indices when deciding whether the can match phase should run or not. When unspecified, this pr runs the can match phase automatically if one of these conditions is met: * The request targets more than 128 shards. * The request contains read-only indices. * The primary sort of the query targets an indexed field. Users can opt-out from this behavior by setting the `pre_filter_shard_size` to a static value. Closes elastic#39835
jimczi
added
>enhancement
:Search/Search
|
Pinging @elastic/es-search (:Search/Search) |
jimczi
changed the title
javanna
approved these changes
Mar 20, 2020
rest-api-spec/src/main/resources/rest-api-spec/api/msearch.json
Outdated
Show resolved
Hide resolved
| When unspecified, the pre-filter phase is executed if any of these conditions is met: | ||
| - The request targets more than `128` shards. | ||
| - The request contains read-only indices. | ||
| - The primary sort of the query targets an indexed field. |
There was a problem hiding this comment.
thinking out loud: this could hit people who use CCS without minimizing roundtrips and experience latency, but given that we minimize roundtrips by default, it should be ok. and they can always increase the parameter manually should they see their CCS search slow down.
javanna
reviewed
Mar 23, 2020
| a threshold that, when exceeded, will enforce a round-trip to pre-filter search shards that cannot possibly match. | ||
| This filter phase can limit the number of shards significantly. For instance, if a date range filter is applied, then all indices (frozen or unfrozen) that do not contain documents within the date range can be skipped efficiently. | ||
| The default value for `pre_filter_shard_size` is `128` but it's recommended to set it to `1` when searching frozen indices. There is no | ||
| significant overhead associated with this pre-filter phase. |
There was a problem hiding this comment.
would it make sense to still explain and make sure that users don't mess with pre_filter_shard_size at this point? I would not want them to end up setting it when searching frozen indices. It sounds like there is never a good reason to do so?
javanna
approved these changes
Mar 23, 2020
jimczi
added a commit
to jimczi/elasticsearch
that referenced
this pull request
Mar 23, 2020
…stic#53873) This commit changes the pre_filter_shard_size default from 128 to unspecified. This allows to apply heuristics based on the request and the target indices when deciding whether the can match phase should run or not. When unspecified, this pr runs the can match phase automatically if one of these conditions is met: * The request targets more than 128 shards. * The request contains read-only indices. * The primary sort of the query targets an indexed field. Users can opt-out from this behavior by setting the `pre_filter_shard_size` to a static value. Closes elastic#39835
This was referenced Mar 23, 2020
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this pull request
Mar 23, 2020
nik9000
added a commit
that referenced
this pull request
Mar 24, 2020
jimczi
added a commit
that referenced
this pull request
Mar 24, 2020
) (#54007) This commit changes the pre_filter_shard_size default from 128 to unspecified. This allows to apply heuristics based on the request and the target indices when deciding whether the can match phase should run or not. When unspecified, this pr runs the can match phase automatically if one of these conditions is met: * The request targets more than 128 shards. * The request contains read-only indices. * The primary sort of the query targets an indexed field. Users can opt-out from this behavior by setting the `pre_filter_shard_size` to a static value. Closes #39835
|
|
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Mar 25, 2020
tlrx
added a commit
that referenced
this pull request
Mar 25, 2020
tlrx
added a commit
that referenced
this pull request
Mar 25, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit changes the pre_filter_shard_size default from 128 to unspecified.
This allows to apply heuristics based on the request and the target indices when deciding
whether the can match phase should run or not. When unspecified, this pr runs the can match phase automatically if one of these conditions is met:
Users can opt-out from this behavior by setting the
pre_filter_shard_sizeto a static value.Closes #39835