Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-sort shards based on the max/min value of the primary sort field #49092

Merged
merged 14 commits into from
Nov 21, 2019

Commits on Nov 14, 2019

  1. Pre-sort shards based on the max/min value of the primary sort field

    This change automatically pre-sort search shards on search requests that use a primary sort based on the value
    of a field. When possible, the can_match phase will extract the min/max (depending on the provided sort order) values
    of each shard and use it to pre-sort the shards prior to running the subsequent phases. This feature can be useful to
    ensure that shards that contain recent data are executed first so that intermediate merge have more chance to contain
    contiguous data (think of date_histogram for instance) but it could also be used in a follow up to early terminate sorted
    top-hits queries that don't require the total hit count. The latter could significantly speed up the retrieval of the most/least
    recent documents from time-based indices.
    I took two shortcuts here:
    * I reused the can_match phase to add the required information for the shard sort. We could instead introduce a new phase
     but it make sense to me to use the existing phase to add more informations as long as the additional ops are lightweight.
    * The shard sort is done automatically if the primary search sort is based on a field. However this sorting only makes sense
    if the range of values in each shard doesn't overlap (time-based indices sorted on timestamp for instance). We could add
    a new option to enable/disable this behavior or even add an additional `shard_sort` criteria but I also like the fact that
    users don't need to set any option to benefit from this feature.
    
    Relates elastic#49091
    jimczi committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    5f91d33 View commit details
    Browse the repository at this point in the history
  2. fix x-pack test

    jimczi committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    125b6e2 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2019

  1. Configuration menu
    Copy the full SHA
    66f1852 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2019

  1. address review

    jimczi committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    9f959dc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a7e61c2 View commit details
    Browse the repository at this point in the history
  3. cleanup

    jimczi committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    f41a8f4 View commit details
    Browse the repository at this point in the history
  4. Update server/src/main/java/org/elasticsearch/action/search/CanMatchP…

    …reFilterSearchPhase.java
    
    Co-Authored-By: Adrien Grand <jpountz@gmail.com>
    jimczi and jpountz committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    4baaf13 View commit details
    Browse the repository at this point in the history
  5. Update server/src/main/java/org/elasticsearch/action/search/CanMatchP…

    …reFilterSearchPhase.java
    
    Co-Authored-By: Adrien Grand <jpountz@gmail.com>
    jimczi and jpountz committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    39f7e47 View commit details
    Browse the repository at this point in the history
  6. Update server/src/main/java/org/elasticsearch/action/search/CanMatchP…

    …reFilterSearchPhase.java
    
    Co-Authored-By: Adrien Grand <jpountz@gmail.com>
    jimczi and jpountz committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    56195eb View commit details
    Browse the repository at this point in the history
  7. Update server/src/main/java/org/elasticsearch/action/search/CanMatchP…

    …reFilterSearchPhase.java
    
    Co-Authored-By: Adrien Grand <jpountz@gmail.com>
    jimczi and jpountz committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    4977d3f View commit details
    Browse the repository at this point in the history
  8. Update server/src/main/java/org/elasticsearch/cluster/routing/GroupSh…

    …ardsIterator.java
    
    Co-Authored-By: Adrien Grand <jpountz@gmail.com>
    jimczi and jpountz committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    3cc99a4 View commit details
    Browse the repository at this point in the history
  9. Update server/src/main/java/org/elasticsearch/search/sort/MinAndMax.java

    Co-Authored-By: Adrien Grand <jpountz@gmail.com>
    jimczi and jpountz committed Nov 20, 2019
    Configuration menu
    Copy the full SHA
    aa517ff View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2019

  1. address review

    jimczi committed Nov 21, 2019
    Configuration menu
    Copy the full SHA
    310b140 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d89dfea View commit details
    Browse the repository at this point in the history