Speed up nhc_hw_gather_data function for KNL #47
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reduces number of access to /proc/cpuinfo in the nhc_hw_gather_data function. For processors like Intel Phi KNL with 256+ threads, the function was taking over 40 seconds to return. Running on a normal file, the function should only take about 4 seconds. Pre-filtering with grep shortens the function runtime further to 1.5s on a 1.4GHz (64 core, 256 thread) KNL.
This was referenced Sep 21, 2017
Closed
|
This one is great! |
|
Hi |
mej
added a commit
that referenced
this pull request
Jan 31, 2023
This branch/changeset extensively refactors `lbnl_hw.nhc` as a possible solution/improvement for the `/proc` file I/O problem (e.g., - Split `nhc_hw_gather_data()` into distinct functions per section, allowing more fine-grained control over which `procfs` and/or `sysfs` files actually need to be read and parsed in the first place; - Convert the path+filename for each file being parsed into a configuration variable that users can customize as needed; - Alter the way NHC is pulling data from each file, by using a single `read` invocation, to avoid the [`lseek()`/rebuild problems described](#43 (comment)) by @mattmix in #43. - Add support for using a cached copy of the original file (instead of reading directly from `/proc` or `/sys`) to avoid the resulting poor performance; and - Allow for the aforementioned configuration variables to specify a process substitution expression (see `bash(1)` for details) rather than a path+filename. This, in turn, permits the user to, for example, `grep` out unused lines to minimize parsing time. (For a specific example and rationale, see [this comment](#30 (comment)) from @NateCrawford with relevant performance comparisons.) - Improve the unit tests for this module; now that the hardware checks are capable of reading from a user-defined location, the unit tests feed auto-generated data into the checks via the `/dev/stdin` "file" (really a shell variable). Not only does this verify the new "user-specified custom data source" code, it also expands the "code coverage" of the unit tests. The old tests just directly assigned the test data to the right NHC-internal variables; the new tests cover the parsing code as well, not just the checks themselves. These changes are intended to fix #30, #39, #43, #47, and #118 as well as some older LANL-internal issues with [Trinity](https://www.lanl.gov/projects/trinity/specifications.php) (our Haswell/KNL-based, nineteen-thousand-node HPE/Cray XC40). And with respect to Trinity, I would be remiss were I to fail to express my sincere thanks to @grahamvh, my colleague at @lanl and one of the main sysadmins for that system, who helped me immensely in brainstorming, devising potential solutions, testing, and providing critical feedback en route toward finally getting this problem licked! Feedback on this approach is much appreciated!
|
Fixed via #121 instead; closing. NHC goes to great lengths to avoid spawning subshells or other processes wherever and whenever possible. The culprit turned out to be the way the kernel and Bash handle |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reduces number of access to /proc/cpuinfo in the nhc_hw_gather_data function. For processors like Intel Phi KNL with 256+ threads, the function was taking over 40 seconds to return. Running on a normal file, the function should only take about 4 seconds. Pre-filtering with grep shortens the function runtime further to 1.5s on a 1.4GHz (64 core, 256 thread) KNL.