Custom CRD: Support dynamic Trial's jobs conditions

[andreyvelich]

Related: #1214.
WIP until we merge this: #1305, after that I will generate API clients, etc, but it would be great if you can start to review it.

I changed a bit API from proposal to support various success and failure conditions in CRD.
As I can see not all CRDs are using the same status approach with Conditions list as Kubeflow operators or BatchJobs.
For example, Argo uses .status.phase to define whether Node is succeeded or failed.

To handle all of these cases, I proposed approach when user defines SuccessCondition and FailureCondition in Experiment.

This condition is gjson expression which can directly point when job is succeeded or failed.

For example, for TFJob:

successCondition = "status.conditions.#(type=="Succeeded")#|#(status=="True")#"
failureCondition = "status.conditions.#(type=="Failed")#|#(status=="True")#"

For Argo Workflow:

successCondition = "status.[@this].#(phase=="Succeeded")"
failureCondition =  "status.[@this].#(phase=="Failed")"

I think this approach is extensible, since user can define any expression there when they want to fail Trial.
For example, when TFJob reaches 3 failed replicas.
status.replicaStatuses.master.[@this].#(failed=="3")

For Kubeflow operators or BatchJobs we can define these conditions in advance if we don't want to let user manually add them.

I renamed UpdateTrialStatusCondition to UpdateTrialStatusConditionDeprecated, since I had to refactor UpdateTrialStatusCondition.

In job_util.go I created TrialJobStatus which represents current Job status.

1. When Job is created/running status is Running.
2. If Job reaches successCondition status is Succeeded.
3. If Job reaches failureCondition status is Failed.

I also transfer Message and Reason from deployed Job to Trial if it is possible.

/assign @gaocegege @johnugeorge @sperlingxx
/cc @czheng94 @nielsmeima

[k8s-ci-robot]

@andreyvelich: GitHub didn't allow me to request PR reviews from the following users: czheng94, nielsmeima.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Related: #1214.
WIP until we merge this: #1305, after that I will generate API clients, etc, but it would be great if you can start to review it.

I changed a bit API from proposal to support various success and failure conditions in CRD.
As I can see not all CRDs are using the same status approach with Conditions list as Kubeflow operators or BatchJobs.
For example, Argo uses .status.phase to define whether Node is succeeded or failed.

To handle all of these cases, I proposed approach when user defines SuccessCondition and FailureCondition in Experiment.

This condition is gjson expression which can directly point when job is succeeded or failed.

For example, for TFJob:

successCondition = "status.conditions.#(type=="Succeeded")#|#(status=="True")#"
failureCondition = "status.conditions.#(type=="Failed")#|#(status=="True")#"

For Argo Workflow:

successCondition = "status.[@this].#(phase=="Succeeded")"
failureCondition =  "status.[@this].#(phase=="Failed")"

I think this approach is extensible, since user can define any expression there when they want to fail Trial.
For example, when TFJob reaches 3 failed replicas.
status.replicaStatuses.master.[@this].#(active=="1")

For Kubeflow operators or BatchJobs we can define these conditions in advance if we don't want to let user manually add them.

I renamed UpdateTrialStatusCondition to UpdateTrialStatusConditionDeprecated, since I had to refactor UpdateTrialStatusCondition.

In job_util.go I created TrialJobStatus which represents current Job status.

1. When Job is created/running status is Running.
2. If Job reaches successCondition status is Succeeded.
3. If Job reaches failureCondition status is Failed.

I also transfer Message and Reason from deployed Job to Trial if it is possible.

/assign @gaocegege @johnugeorge @sperlingxx
/cc @czheng94 @nielsmeima

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kubeflow-bot]

This change is 

[andreyvelich]

@gaocegege @johnugeorge @sperlingxx This PR is ready for review.

[andreyvelich]

/retest

[gaocegege]

/assign @johnugeorge

[gaocegege]

Can we delete it now?

[andreyvelich]

@gaocegege We can delete it after we finish Custom CRD implementation.
Controller still runs this function if successCondition and failureCondition is not set here:

katib/pkg/controller.v1beta1/trial/trial_controller.go

Line 307 in 409b3a2

r.UpdateTrialStatusConditionDeprecated(instance, deployedJob, jobCondition)

[gaocegege]

/lgtm

[johnugeorge]

Please keep track of the cleanup required to remove the deprecated functions.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johnugeorge]

/lgtm

[andreyvelich]

/retest

[andreyvelich]

/retest