Reduce bugtool memory usage #17546
Merged
Reduce bugtool memory usage #17546
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
s/ethool/ethtool/ Also fix the build for other non-Linux platforms besides macOS. Signed-off-by: Tobias Klauser <tobias@cilium.io>
Avoid reallocations and GC pressure in the loop. Signed-off-by: Tobias Klauser <tobias@cilium.io>
tklauser
added
release-note/minor
|
/test |
There is no need to recompile the constant regexp for each iteration of the loop. Also, hashEncryptionKeys is called in a loop, so move the regexp compilation to a global var to avoid recompiling it. Signed-off-by: Tobias Klauser <tobias@cilium.io>
Instead of open-coding a worker pool use the existing github.com/cilium/workerpool package. Also limit the number of workers to the number of CPUs by default, which should help to reduce excessive memory usage by too many parallel goroutines at the price of potentially slightly slower bugtool report creation. If users want more parallel tasks, the number can be specified using the newly introduces `--parallel-workers` option. Signed-off-by: Tobias Klauser <tobias@cilium.io>
Converting []byte to string can cause allocation due to the fact that strings are immutable in Go. By letting execCommand return []byte instead of string we can avoid some of these allocations, i.e also the allcoation caused by further processing of that return value. Signed-off-by: Tobias Klauser <tobias@cilium.io>
tklauser
force-pushed
the
pr/tklauser/bugtool-mem-usage
branch
from
c61ff98
to
e090a65
Compare
When the command output doiesn't need to be postprocessed, the output
can be written directly to the file without buffering. This should
significantly reduce memory usage.
On my test system this reduces total RSS by about one third:
Before:
$ /usr/bin/time -f 'RSS=%MKB' cilium-bugtool
RSS=63168KB
After:
$ /usr/bin/time -f 'RSS=%MKB' cilium-bugtool
RSS=42752KB
Signed-off-by: Tobias Klauser <tobias@cilium.io>
tklauser
force-pushed
the
pr/tklauser/bugtool-mem-usage
branch
from
e090a65
to
68d1f63
Compare
michi-covalent
approved these changes
Oct 6, 2021
|
/test |
|
Multicluster test is the only one failing. That test is not required and the failure will be fixed by #17549. All other enabled tests passed, marking as |
tklauser
added
the
ready-to-merge
This was referenced Nov 30, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See individual commits and #17309 for details.
On my test system this reduces total RSS by about one third:
Before:
After:
Fixes #17309