fix: tfjob with restartPolicy=ExitCode not work #1562
Conversation
|
Hi @cheimu. Thanks for your PR. I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
cheimu
changed the title
Pull Request Test Coverage Report for Build 2030402854
💛 - Coveralls |
|
/ok-to-test |
| // the restarting condition will be removed from jobStatus by commonv1.filterOutCondition(), | ||
| // so we need to append the restarting condition back to jobStatus. | ||
| if existingRestartingCondition != nil { | ||
| err := commonutil.UpdateJobConditions(jobStatus, commonv1.JobRestarting, existingRestartingCondition.Reason, existingRestartingCondition.Message) |
There was a problem hiding this comment.
Should we reset the value of existingRestartingCondition after this line?
There was a problem hiding this comment.
Hi @richardsliu, I don't know if we could reset it. I record the existingRestartingCondition outside the loop, which is the condition and result for current reconcile(). Inside the loop, when iterate over all pods, if there are multiple workers or masters .etc, are running, and only one worker failed, by the logic inside of updateJobConditions(), restarting + running combination will make running condition overwrite restarting.
There was a problem hiding this comment.
@richardsliu is this the right way to deal with it? If one of the pods is restarting, do we mark the whole job as restarting ?
There was a problem hiding this comment.
If we do something along the lines of, would it achieve the same thing?
if condition.Type == commonv1.JobRunning || condition.Type == commonv1.JobRestarting {
restart = true
There was a problem hiding this comment.
Hi @pavanky, I would like to hear more advices: ) For now, I just follow the current convention. See https://github.com/kubeflow/training-operator/blob/master/pkg/controller.v1/tensorflow/tfjob_controller.go#L762 . Current strategy is that if one of the pods failed, then update the whole job status to failed : /
There was a problem hiding this comment.
Wouldn't that technically mean we are updating the job status to restart and incrementing the count two times for the same failure ?
There was a problem hiding this comment.
IMO, we could see it from 2 points of views.
- If jobStatus is
running, and one of roles got failed pods, then the whole job will be updated tofailed, then in next reconcile, the pods will be deleted, so jobstatus should have arestarting(don't know if I understand the code base correctly, we don't have specific restarting logic but use general reconcile logic) - Technically we just set jobstatus's field twice without sending request to apiServer, so in fact, it didn't get really updated. But yeah, current one is not elegant. 🤔
|
/retest |
|
/lgtm |
|
Hi @richardsliu , what should we do next for this pr/issue? |
|
Hi all! Just wanted to ask if there is intent to merge this PR -- our users are definitely impacted by this issue and we'd really benefit from the fix. Thanks so much! |
|
We get two LGTMs here. I think we should merge it. Sorry for the late reply. /approve. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cheimu, gaocegege The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Thanks so much for the speedy merge! |
(cherry picked from commit 9cc1cc9)
…le-cherrypick-exitcode-fix fix: tfjob with restartPolicy=ExitCode not work (kubeflow#1562)
What this PR does / why we need it:
As describe in issues linked below, the tfjob with restartPolicy=ExitCode won’t work. If workers/evaluators got exited, the whole job will be set to failed instead of restarting, which is not expected.
Need for more comments! Thank you in advance! :D
Which issue(s) this PR fixes (optional, in
Fixes #<issue number>, #<issue number>, ...format, will close the issue(s) when PR gets merged):Fixes #1560, kubeflow/common#186
Checklist: