Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store: fix chunk pool memory leak #1512

Merged
merged 1 commit into from Sep 13, 2019
Merged

store: fix chunk pool memory leak #1512

merged 1 commit into from Sep 13, 2019

Conversation

abursavich
Copy link
Contributor

I experienced a slow drip of "key does not exist" errors, presumably due to compactions, then nearly all get_range requests began to fail with "pool exhausted" errors.

caller err
bucket.go:871 rpc error: code = Aborted desc = fetch series for block 01DM8ACH11QQQESTY0VET5PXKS: preload chunks: read range for 1: get range reader: The specified key does not exist.
bucket.go:871 rpc error: code = Aborted desc = fetch series for block 01DMA1C7D151KM6H1DGY498WWQ: preload chunks: read range for 6: allocate chunk bytes: pool exhausted

Screen Shot 2019-09-12 at 3 41 44 PM

Changes

Return the byte slice to the pool on error paths.

Verification

N/A

Signed-off-by: Andy Bursavich <abursavich@gmail.com>
Copy link
Member

@GiedriusS GiedriusS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this, we are definitely forgetting to put the contents back into the pool here! All of the other places seem OK to me.

@GiedriusS GiedriusS merged commit 42e9506 into thanos-io:master Sep 13, 2019
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

wbh1 pushed a commit to wbh1/thanos that referenced this pull request Sep 17, 2019
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants