Collect RSS and CPU usage for OVS/OVN processes. #106
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this change the new process-monitor.py script is copied into each container and started to collect RSS and CPU usage of all OVS/OVN processes. Later, log-collector will stop the monitoring and will collect results as a process-stats.json file. do.sh script then uses process-stats.py script at the data mining stage to produce html reports for master and worker nodes containing RSS an CPU usage graphs. The integration part is a bit sketchy. Alternative is to avoid modification of the ovn-tester itself and only deploy the process monitor on the physical host. This will make the code a bit cleaner, but we'll loose an easy way to add container names to process names. Also, the monitor spends 0.5 seconds per process to estimate the CPU usage. More processes to track - less accurate the results. For example, one round of measurements for 9 processes takes ~4.5 seconds, so we can easily miss the 3 second database compaction. We could run a monitor per process though, but that sounds like an overkill. In order to work properly, python3-psutil needs to be installed in the container. If it's not available, ovn-heater will still work normally, but resource usage reports will not be generated. Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
|
Corresponding ovn-fake-multinode PR: ovn-org/ovn-fake-multinode#64 |
|
Reports look like this, but they are interactive html (github doesn't allow to attach html): |
|
Request: Make it an ovscon 2022 track! |
|
Looks great to me, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this change the new process-monitor.py script is copied into
each container and started to collect RSS and CPU usage of all
OVS/OVN processes. Later, log-collector will stop the monitoring
and will collect results as a process-stats.json file. do.sh
script then uses process-stats.py script at the data mining stage
to produce html reports for master and worker nodes containing
RSS an CPU usage graphs.
The integration part is a bit sketchy. Alternative is to avoid
modification of the ovn-tester itself and only deploy the
process monitor on the physical host. This will make the code
a bit cleaner, but we'll loose an easy way to add container names
to process names. Also, the monitor spends 0.5 seconds per process
to estimate the CPU usage. More processes to track - less accurate
the results. For example, one round of measurements for 9 processes
takes ~4.5 seconds, so we can easily miss the 3 second database
compaction. We could run a monitor per process though, but that
sounds like an overkill.
In order to work properly, python3-psutil needs to be installed
in the container. If it's not available, ovn-heater will still
work normally, but resource usage reports will not be generated.
Signed-off-by: Ilya Maximets i.maximets@ovn.org