.close() interchange process to release more fds #3399
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This releases two fds in the workflow process used to communicate with the interchange process, and is part of work to release more resources explicitly at shutdown rather than leaving them until process end. One test was making an out-of-API shutdown of HTEX on the basis that the DFK would not shutdown this executor when a bad state was set. That used to be true, but PR 2855, which re-arranged some shutdown behaviour, (accidentally?) changed shutdown to always happen. That is, I think, the right shutdown behaviour: if an executor wants to suppress parts of its own shutdown based on internal state, that's the business of the executor, not the DFK. This PR removes that out-of-API shutdown.
khk-globus
reviewed
Apr 25, 2024
Comment on lines
-40
to
-43
|
|
||
| # htex needs shutting down explicitly because dfk.cleanup() will not | ||
| # do that, as it is in bad state | ||
| parsl.dfk().config.executors[0].shutdown() |
There was a problem hiding this comment.
Good change ... ought we go one further and include a fixture that fails if the executor is not shutdown at the end of each test (or perhaps module)?
There was a problem hiding this comment.
this is working towards two fixtures that will fail on threads existing and fds being open...
checking that dfk.cleanup() has been called would be pretty straightforward: that flag already exists in order to try to make dfk.cleanup() idempotent. but not all tests have a dfk... or exactly one dfk...
khk-globus
approved these changes
Apr 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This releases two fds in the workflow process used to communicate with the interchange process, and is part of work to release more resources explicitly at shutdown rather than leaving them until process end.
One test was making an out-of-API shutdown of HTEX on the basis that the DFK would not shutdown this executor when a bad state was set. That used to be true, but PR #2855, which re-arranged some shutdown behaviour, (accidentally?) changed shutdown to always happen. That is, I think, the right shutdown behaviour: if an executor wants to suppress parts of its own shutdown based on internal state, that's the business of the executor, not the DFK.
This PR removes that out-of-API shutdown.
Changed Behaviour
As exposed by the changed test, shutdown behaviour is a bit peturbed: for example, situations where executor.shutdown was called multiple times that worked before might no longer work.
Type of change