Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-20.2: sql: shrink SampleReservoir capacity on memory exhaustion #67059

Merged
merged 3 commits into from
Jun 30, 2021
Merged

release-20.2: sql: shrink SampleReservoir capacity on memory exhaustion #67059

merged 3 commits into from
Jun 30, 2021

Commits on Jun 30, 2021

  1. sql, stats: shrink sampleAggregator capacity to match child capacity 

    To avoid bias, sampleAggregator must always use a capacity <= the
capacity of each child samplerProcessor feeding it. This was always
implicitly true before, as every samplerProcessor and sampleAggregator
used the same fixed capacity. But the next few commits will give
sampleProcessors (and sampleAggregators) the ability to dynamically
shrink capacity when out of memory, meaning we now sometimes need to
resize sampleAggregator to keep this invariant true.

(Resizing sampleAggregator really means resizing the underlying
SampleReservoir, so most of the changes are there.)

This commit does not yet change any behavior, because the capacities of
samplerProcessors are still static. Next few commits will change that.

Also factor the giant closure out of TestSampleAggregator into an
external function, to make table-driven testing easier.

Release note: None
    @michae2
    michae2 committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    7eabe8e View commit details
    Browse the repository at this point in the history

  2. sql, stats: shrink SampleReservoir capacity on memory exhaustion 

    Fixes: cockroachdb#62206

Instead of returning an error when out of memory, make
SampleReservoir.SampleRow dynamically reduce capacity and retry. Fewer
samples will result in a less accurate histogram, but less accurate is
probably still better than no histogram at all, up to a point.

If the number of samples falls below a minimum threshold, then give up
and disable histogram collection, as we were doing originally, rather
than using a wildly inaccurate histogram.

Also fix some memory accounting in SampleReservoir.copyRow.

Release note (performance improvement): continue to generate histograms
when table statistics collection reaches memory limits, instead of
disabling histogram generation.
    @michae2
    michae2 committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    ef2ec87 View commit details
    Browse the repository at this point in the history

  3. sql, stats: shrink sampleAggregator capacity before histogram generation 

    SampleAggregator can also run out of memory when gathering the datums
for histogram generation. Push this datum gathering down into
SampleReservoir so that we can use the same dynamic capacity-shrinking
strategy as used in SampleRow to reclaim some memory and try again.

Release note: None
    @michae2
    michae2 committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    8f7247c View commit details
    Browse the repository at this point in the history