job-manager: improve handling of offline ranks in job prolog #5910
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: flux-perilog-run.py fails if there are offline ranks, but the INFO log message is suppressed by default. Promote the message to a WARNING so it is logged. Additionally fix use of plural 'ranks' when there is only one rank offline.
Problem: When a prolog fails due to offline ranks, a generic exception like "prolog exited with exit code=1" is raised after the prolog exits, but this gives the user and admins no details about why the prolog failed. Add a nonfatal exception from the job prolog when there are offline ranks detected so that a specific reason is associated with the job eventlog.
Problem: t2274-manager-perilog.t runs a test instance with the default topology, but it will be easier to manage offline rank testing if the TBON were flat. Update test_under_flux() to run a flat TBON for this test.
Problem: The handling of offline ranks from the job-manager prolog needs to be tested, but there is no easy way to force this condition (a rank goes offline between the scheduler allocating resources and the prolog execution.) Add a jobtap plugin which can be loaded before the perilog plugin that will set rank 3 offline first thing in RUN state for a job.
garlick
approved these changes
Apr 22, 2024
t/t2274-manager-perilog.t
Outdated
| test_must_fail flux job attach $id && | ||
| flux jobtap remove offline.so | ||
| ' | ||
| test_expect_success 'perilog: offline ranks are logeed by prolog' ' |
There was a problem hiding this comment.
Fixed that and I'll set MWP. Thanks
Problem: In t2274-manager-perilog.t, handling of offline ranks by flux-perilog-run.py is only tested after a job has run by manually invoking the command. However, this doesn't ensure that the script raises a nonfatal exception of the job instead of silently failing. Use the new offline.so jobtap plugin to force rank 3 offline instead of disconnecting the rank outside of any job. Since the offline.so plugin is loaded before the perilog.so plugin, this simulates a rank going offline between the scheduler assigning resources and the prolog. Ensure that a nonfatal exception is raised on the job.
grondo
force-pushed
the
prolog-and-offline
branch
from
4503eac
to
a557e58
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #5910 +/- ##
==========================================
- Coverage 83.34% 83.31% -0.03%
==========================================
Files 514 514
Lines 82901 82913 +12
==========================================
- Hits 69092 69082 -10
- Misses 13809 13831 +22
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently, jobs were consistently failing with an exception
job prolog failed with exit code=1without any other details in the logs or job eventlog. It turned out that the scheduler was handing out offline nodes, and the prolog fails in this case, but due to the use of an incorrect log level, the log message for this condition was suppressed. Also, users were confused by the lack of detail in the job exception.This PR improves prolog handling of offline ranks by
flux perilog-runcan't raise the fatal exception itself, since would then cancel the prolog, which may be running on other ranks. The nonzero exit status from the prolog will raise the fatal exception, so a job eventlog failing in this manner might look like this:which is a bit better than completely silent failure.