Introduce Webhook Validation

[tenzen-y]

As described in #1708 (comment), we can not introduce CEL validations for fields under the .spec.replicaSpec due to the CEL cost budget violations. The root cause is that our replicaSpec is defined as a typed map, and then Kubernetes can not estimate how many maximum replicas will be held in the .spec.replicas.

Also, current validations aren't better for UX since the users can find validation errors only in the training-operator logs.

So, I would suggest introducing the webhook validations with internal certs mechanisms like Katib since validations help to find validation errors for users and the internal cert mechanisms could reduce installation costs.

/kind feature

• PyTorch: Implement webhook validations for the PyTorchJob #2035
• TensorFlow: Implement webhook validation for the TFJob #2051
• XGBoost: Implement webhook validations for the XGBoostJob #2052
• MXNet: Implement webhook warnings for the MXJob #2058
• PaddlePaddle: Implement webhook validations for the PaddleJob #2057
• MPIJob: No Plan

[tenzen-y]

/assign

[tenzen-y]

I'm wondering if we should not implement the webhook validations for the MXJob because we will remove the MXJob.

@kubeflow/wg-training-leads WDYT?

[tenzen-y]

I'm wondering if we should not implement the webhook validations for the MXJob because we will remove the MXJob.

@kubeflow/wg-training-leads WDYT?

Based on the last community call, we decided not to implement webhook validations for MXJob, but we will implement webhook warnings to notify deprecations.

[andreyvelich]

Yes, that's correct @tenzen-y!
Let's just add the warning for users via webhook.

[tenzen-y]

@andreyvelich @johnugeorge For MPIJob v1, I'm wondering if we can skip to implement webhooks.
WDYT?

[andreyvelich]

I am fine to not add them since we will be focus on V2 for the long-term

[tenzen-y]

Thank you for checking this.
So, I will skip the webhook implementations for the MPIJob v1.