pkg/perf: Prevent intermediate perfmap/jitdump entry lists #2325
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brancz
force-pushed
the
symtab-mem-usage
branch
2 times, most recently
from
63be869
to
defdd55
Compare
Previously we maintained multiple lists of perfmap and jitdump entries as well as all related strings. This caused somewhat large memory spikes when a lot of these have to be loaded frequently. Now strings are never on heap and are directly written to the optimized symtab file, and only entries are loaded into memory once so they can be sorted and deduplicated easily and then without any further intermediate lists directly written to the optimized symtab file. If this is also not sufficient the next optimization would be to read perfmap files from the back and directly write entries that are read if they don't conflict with any already written memory ranges.
brancz
force-pushed
the
symtab-mem-usage
branch
from
defdd55
to
9b6e508
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why?
Previously we maintained multiple lists of perfmap and jitdump entries as well as all related strings. This caused somewhat large memory spikes when a lot of these have to be loaded frequently.
What?
Prevent anything that doesn't have to happen on heap to go onto the heap in the first place.
How?
Now strings are never on heap and are directly written to the optimized symtab file, and only entries are loaded into memory once so they can be sorted and deduplicated easily and then without any further intermediate lists directly written to the optimized symtab file.
Test Plan
Tested locally against jitdump and perfmap workloads.
Further notes
If this is also not sufficient the next optimization would be to read perfmap files from the back and directly write entries that are read if they don't conflict with any already written memory ranges.