flux-perilog-run: improve usefulness of logging when prolog/epilog fails #4054
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: When `flux perilog-run --exec-per-rank=CMD` fails, error messages from individual ranks are not differentiated, so it is difficult to tell on which rank/host the prolog or epilog error occurred. Additionally, the default Python logging level is too low and important "info" messages are dropped. Adjust the execution of subprocesses so that stderr is captured, and log any stderr from failed ranks. Also capture the instance hostlist before running and use this to display the failing hostname along with its rank for easier diagnosis. Finally, promote the log message regarding draining nodes from LOGGER.info to LOGGER.error, so that it is displayed by default. Fixes flux-framework#4053
Problem: No tests of `flux perilog-run` ensure that errors generated by the prolog/epilog are logged by the perilog-run script along with their hostname and rank. Add a test to ensure the above is true.
|
Thanks! Had to fix a broken |
Codecov Report
@@ Coverage Diff @@
## master #4054 +/- ##
==========================================
+ Coverage 83.45% 83.77% +0.32%
==========================================
Files 374 374
Lines 61397 54243 -7154
==========================================
- Hits 51236 45440 -5796
+ Misses 10161 8803 -1358
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When attempting to set up prolog/epilog support on a system, it was discovered that errors from remote prolog/epilog execution were being copied to the logs without an identifying rank or hostname, which made diagnosis of errors unnecessarily difficult.
This PR improves the logging of the
flux perilog-runscript when used to execute per-rankprologandepilogby including the rank and hostname in any lines of stderr generated by these scripts, along with other small improvements.