fix: commands (TensorBoards, notebooks, etc.) should not be preempted [DET-4157] #1346
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mackrorysd
requested review from
aaron276h and
rb-determined-ai
as code owners
shiyuann
reviewed
Sep 25, 2020
mackrorysd
force-pushed
the
det-4157
branch
from
75779c7
to
ea3bec4
Compare
shiyuann
reviewed
Sep 28, 2020
mackrorysd
force-pushed
the
det-4157
branch
from
ea3bec4
to
b957dab
Compare
|
@mackrorysd I have left one comment. Also, can you also run E2E GPU tests for this PR before merging it? |
mackrorysd
force-pushed
the
det-4157
branch
from
5ed9380
to
e8466ae
Compare
mackrorysd
changed the title
shiyuann
reviewed
Sep 30, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Commands (TensorBoards, Notebooks), being interactive, would cause more obvious disruption to an end-user if preempted.
Implementing this requires changes in 2 aspects of the scheduler. First simply not sending the ReleaseResources message to such tasks (and closely related: commands will also ignore this unless something changes CanTerminate to true, and nothing does right now). Second, the scheduler will also take this into account when allocating slots so that a group cannot lose a slot it needs for non-preemptible tasks, so the system cannot oversubscribe itself.
Test Plan
I've added unit tests that reproduced the original problem, including subtleties discovered during implementation. Some other unit tests needed to have CanTerminate added to their task definitions since it defaults to false, which triggers the new behavior.
Commentary
This patch prevents these commands from being preempted using the existing CanTerminate flag. Because of Go's zero-value initialization, this means all tasks are now non-premptible by default. This feels backwards - I would actually prefer to replace this with a field call NonPreemptible (depending on how reviewers feel about it) and invert the logic.
I also made garbage collection non-preemptible - I'm not sure if that's desirable or not.
Checklist
docs/release-notes/.See Release Note for details.