Fix memory leak in blocks storage ingesters #2586
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
In a production cluster running the blocks storage we've seen a large portion of the inuse heap retaining memory allocate by
grpc.recvMsg()
. This is caused by the fact thatFromLabelAdaptersToLabelsWithCopy()
doesn't do a deep copy but reuse input strings (which are read-only slice pointers).In this PR we're fixing it doing a deep copy.
Credits: the root cause has been spotted and fixed by @pstibrany. I've just done double checks and opening the PR for it.
Given the same exact test scenario, this is the
recvMsg()
allocations before the fix:And this is after the fix:
Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]