[v1beta2] Add ActiveDeadlineSeconds and BackoffLimit

[u2takey]

1. add ActiveDeadlineSeconds to tfjob.Spec like batch.Job's JobSpec.ActiveDeadlineSeconds, which control the max active time for a job.
2. add BackoffLimitt to tfjob.Spec like batch.Job's JobSpec.BackoffLimitt, which control restart limit for a failing job.
3. add CleanPolicy to tfjob.Spec for control clean up tfjob's pod after tfjob is done(success/fail), clean up on time is very important for a cloud user.

[gaocegege]

Thanks for the issue. We have some discussions about the restartpolicy here #544

As for the clean policy, I am not sure that adding a clean policy is the solution. We have some discussions in #536

We prefer to cleanup the pods ASAP, while log is another thing that we should consider.

[u2takey]

ok thanks @gaocegege , i will add comment there.

[jlewi]

We added CleanPodPolicy to v1alpha2 and it should be in 0.2.0.

Do we still need/want ActiveDeadlineSeconds and BackoffLimit with the other changes to RestartPolicy?

Is this something we want to fix in 0.3?

We should try to nail down the API in order to get to v1.

[gaocegege]

I think we could add ActiveDeadlineSeconds and BackoffLimit, while it is not in high priority

[jlewi]

@richardsliu @johnugeorge Can we make a decision about whether to include these fields as part of the API?

[gaocegege]

I think we should add it as API, it is useful.

[johnugeorge]

/remove-area 0.4.0
/area 0.5.0

[k8s-ci-robot]

@johnugeorge: Those labels are not set on the issue: area/0.4.0

In response to this:

/remove-area 0.4.0
/area 0.5.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[richardsliu]

/assign @ChanYiLin

[ChanYiLin]

I can take this issue
Thanks!

[gaocegege]

/assign @ChanYiLin

[richardsliu]

@ChanYiLin Any update on this?

[ChanYiLin]

I have finished ActiveDeadlineSeconds feature and now thinking how to implement BackoffLimit.
According to the implementation of Job controller in K8s, they count the restart count based on the restart count of the pod/container.
However, in tf-operator, when the pod restart because of exit code, we delete the pod and recreate it using api, so there will no restart count in the pod.
I have to try another way, sorry for the late.

[richardsliu]

@ChanYiLin Is this still on track to be finished in 0.5.0? We are trying to reach code complete by 3/15.

[ChanYiLin]

Yes it can be finished this week. I am testing it.

[richardsliu]

Thanks!

[jlewi]

@ChanYiLin Any update on this?

[ChanYiLin]

@jlewi almost done
I and @richardsliu just discussed how to implement the unit test.
Also, it seems that it failed the e2e test sometimes and randomly.

[ChanYiLin]

@jlewi
the PR has been merged.