testsuite: fix lingering processes after make check
#5769
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
garlick
approved these changes
Mar 5, 2024
grondo
force-pushed
the
make-check-orphans
branch
from
a2540cb
to
6f45007
Compare
|
Well this change is causing hangs in CI now, so obviously this wasn't a sufficient or good workaround. More analysis needed. |
grondo
force-pushed
the
make-check-orphans
branch
from
6f45007
to
596a4cc
Compare
Problem: There are often a couple orphaned Flux instances that remain after a run of `make -j N check`. The instances appear to be blocked in rc3 with a sleep job in CLEANUP which can never exit because the scheduler module has already been unloaded. The root cause of this situation is unknown, but probably the affected tests can be improved to reduce the probability of the issue. The affected tests include t2800-jobs-recursive.t and t2802-uri-cmd.t, each of which start a nested instance of one or more levels with sleep jobs. Attempt to terminate the affected batch jobs cleanly rather than leaving it to the test instance rc3 script. This should reduce or eliminate stray processes after running make check or the individual test.
grondo
force-pushed
the
make-check-orphans
branch
from
596a4cc
to
ffa89a0
Compare
|
Ok. The hang was from a use of |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #5769 +/- ##
=======================================
Coverage 83.34% 83.34%
=======================================
Files 509 509
Lines 82485 82485
=======================================
+ Hits 68744 68747 +3
+ Misses 13741 13738 -3 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR attempts to fix or workaround an issue that I keep seeing: A couple orphaned instances that stick around after
make -j N check. A quick investigation showed that the instances all had a sleep job stuck in CLEANUP, with the scheduler unloaded, and had at least instance-level 1 (so they were batch jobs run within the test instance).I wasn't really able to make a reproducer for this issue, but terminating the affected instances more cleanly seems to resolve the problem (at least in my testing), so it seems like a good idea to do that.