fix sys.cpu.utilization metrics for titus isolate v2 #127
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
copperlight
force-pushed
the
fix-sys.cpu.utilization
branch
2 times, most recently
from
1c58da9
to
ec0bcb1
Compare
The new 7th generation hardware sets a `cfs_quota` of `max`, which is then read as a `0` in the number vector. This was breaking utilization calculations. * Refactor code to remove all remaining traces of cgroup selection, since all Titus hosts are now on cgroupv2. * Remove extra `ifdef` macro for `TITUS_SYSTEM_SERVICE` that had no effect. * The `TITUS_NUM_CPU` environment variable is expected to always be present, and should always be used to determine the number of CPUs allocated. * The number of CPUs can be used to determine the value of `cfs_quota`, which can then be used to calculate available CPU time. * Refactor metrics collection for cpu time and cpu utilization, to reduce the number of duplicate calls and simplify the processing. * Shuffle the order of nvml library initialization, so that we can use the standard logger to report status. * Refactor the `setup-venv.sh` script, to make it easier to bootstrap new dev environments on bionic hosts.
copperlight
force-pushed
the
fix-sys.cpu.utilization
branch
from
ec0bcb1
to
f66b053
Compare
|
|
||
| auto cpu_max = read_num_vector_from_file(path_prefix_, "cpu.max"); | ||
| auto cfs_period = cpu_max[1]; | ||
| auto cfs_quota = cfs_period * num_cpu; | ||
| auto avail_cpu_time = (delta_t / cfs_period) * cfs_quota; |
There was a problem hiding this comment.
Computing the avail_cpu_time and num_cpu seems to be duplicated for the normal stats and peak. It might make sense to have functions for those to ensure they stay consistent in the future.
There was a problem hiding this comment.
Fixed in #128.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The new 7th generation hardware sets a
cfs_quotaofmax, which is then read as a0in the number vector. This was breaking utilization calculations.ifdefmacro forTITUS_SYSTEM_SERVICEthat had no effect.TITUS_NUM_CPUenvironment variable is expected to always be present, and should always be used to determine the number of CPUs allocated.cfs_quota, which can then be used to calculate available CPU time.setup-venv.shscript, to make it easier to bootstrap new dev environments on bionic hosts.