[CORE-1441] [2.4.x] leave pipelines in STANDBY until they're FINISHED #8500
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When monitorPipeline starts, the first thing it does is put its pipeline in STANDBY[1]. Then, it listens on ciChan for commits in READY and takes the pipeline out of STANDBY if it gets any[2]. Meanwhile, another goro calls SubscribeCommit and puts any events into ciChan[3]. The issue is that the first goro discards PFS events where the commit is FINISHING, so if the PPS master restarts while a commit is in FINISHING, it'll come back up, put the pipeline in STANDBY, and then won't take it out. This leaves the commit stuck in FINISHING until compaction finishes much, much later. [1]: https://github.com/pachyderm/pachyderm/blob/8fca5b912358c15f67d8b19a32f84d2ba507571c/src/server/pps/server/monitor.go#L137 [2]: https://github.com/pachyderm/pachyderm/blob/8fca5b912358c15f67d8b19a32f84d2ba507571c/src/server/pps/server/monitor.go#L163 [3]: https://github.com/pachyderm/pachyderm/blob/8fca5b912358c15f67d8b19a32f84d2ba507571c/src/server/pps/server/monitor.go#L110
msteffen
changed the title
msteffen
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the 2.4.x version of #8499
When monitorPipeline starts, the first thing it does is put its pipeline
in STANDBY. Then, it listens on ciChan for commits in READY and takes
the pipeline out of STANDBY if it gets any. Meanwhile, another goro
calls SubscribeCommit and puts any events into ciChan.
The issue is that the first goro discards PFS events where the commit is
FINISHING, so if the PPS master restarts while a commit is in FINISHING,
it'll come back up, put the pipeline in STANDBY, and then won't take it
out. This leaves the commit stuck in FINISHING until compaction finishes
much later.
This PR changes the logic in
monitorPipelineso that if amonitorPipelinegoro's pipeline has a commit in FINISHING, the pipeline goes into RUNNING
and stays there until the commit is FINISHED.