Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allocate memory lazily in BestBucketsDeferringCollector #43339

Merged
merged 6 commits into from
Jun 19, 2019
Merged

Allocate memory lazily in BestBucketsDeferringCollector #43339

merged 6 commits into from
Jun 19, 2019

Commits on Jun 18, 2019

  1. Allocate memory lazily in BestBucketsDeferringCollector 

    While investigating memory consumption of deeply nested aggregations for elastic#43091
the memory used to keep track of the doc ids and buckets in the BestBucketsDeferringCollector
showed up as one of the main contributor. In my tests half of the memory held in the
 BestBucketsDeferringCollector is associated to segments that don't have matching docs
 in the selected buckets. This is expected on fields that have a big cardinality since each
 bucket can appear in very few segments. By allocating the builders lazily this change
 reduces the memory consumption by a factor 2 (from 1GB to 512MB), hence reducing the
impact on gcs for these volatile allocations. This commit also switches the PackedLongValues.Builder
with a RoaringDocIdSet in order to handle very sparse buckets more efficiently.

I ran all my tests on the `geoname` rally track with the following query:

````
{
    "size": 0,
    "aggs": {
        "country_population": {
            "terms": {
                "size": 100,
                "field": "country_code.raw"
            },
            "aggs": {
                "admin1_code": {
                    "terms": {
                        "size": 100,
                        "field": "admin1_code.raw"
                    },
                    "aggs": {
                        "admin2_code": {
                            "terms": {
                                "size": 100,
                                "field": "admin2_code.raw"
                            },
                            "aggs": {
                                "sum_population": {
                                    "sum": {
                                        "field": "population"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
````
    @jimczi
    jimczi committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    fd54cca View commit details
    Browse the repository at this point in the history

  2. restore the packed long builder since the roaring builder does not ac… 

    …cept duplicate
    @jimczi
    jimczi committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    d54d3da View commit details
    Browse the repository at this point in the history

  3. address review

    @jimczi
    jimczi committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    2edc7b6 View commit details
    Browse the repository at this point in the history

  4. restore delta encoding for doc id set builder

    @jimczi
    jimczi committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    ed5e964 View commit details
    Browse the repository at this point in the history

  5. address another review

    @jimczi
    jimczi committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    ecea58a View commit details
    Browse the repository at this point in the history

  6. refactor MergingBucketsDeferringCollector to be an extension of BestB… 

    …ucketsDeferringCollector
    @jimczi
    jimczi committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    ce7f447 View commit details
    Browse the repository at this point in the history