Kubernetes versions to be supported in the next training operator release

[tenzen-y]

Currently, we support 3 K8s versions, such as v1.23, v1.24, and v1.25.

However, v1.23 has already reached EoL, and v1.24 will reach EoL on 2023-07-28.

So we can support v1.24-v1.26 or v1.25-v1.27 in the next training operator release.

ref: https://kubernetes.io/releases/

WDYT? @kubeflow/wg-training-leads @zw0610 @kuizhiqing

[tenzen-y]

I prefer v1.25-v1.27.

[terrytangyuan]

No objection but if it's low effort to keep 1.24 support, it would be great.

[tenzen-y]

No objection but if it's low effort to keep 1.24 support, it would be great.

If we stop supporting 1.24, we can add CEL validation (#1708). I believe that adding validations makes increase UX.

[terrytangyuan]

If that's the only benefit, I'd suggest holding off on dropping 1.24 support since many production systems are still running on that.

[tenzen-y]

If that's the only benefit, I'd suggest holding off on dropping 1.24 support since many production systems are still running on that.

It sounds good to me, and I'm ok with keeping to support 1.24. But probably, we can't support the scheduler-plugins integration in v1.24 because of #1769.

In the next release, we support essential features of the training operator in v1.24-v1.26, although we don't support the scheduler plugin integration in v1.24.

[tenzen-y]

This means users can upgrade the training operator to the next one in K8s v1.24 only if they don't use the scheduler-plugins integration.

If they want to use the latest training operator with the scheduler plugins integration, they must use K8s v1.25-v1.26.

[terrytangyuan]

Ok I am fine with dropping v1.24 support.

[tenzen-y]

Ok I am fine with dropping v1.24 support.

Thank you!
@ OWNERS Let me know if someone would like to support v1.24 :)

[johnugeorge]

IMO, supporting 1.24 would be tough. I am proposing to keep 1.25 as the minimum k8s version. The community is proposing for supporting till 1.27 for the next KF release

Ref: kubeflow/manifests#2450 (comment)

[tenzen-y]

IMO, supporting 1.24 would be tough. I am proposing to keep 1.25 as the minimum k8s version. The community is proposing for supporting till 1.27 for the next KF release

Ref: kubeflow/manifests#2450 (comment)

Thank you for letting us know!

[johnugeorge]

We can support 1.25, 1.26 and 1.27 in v1.7.0 release for being consistent

[tenzen-y]

/assign

In the first step, I'll try to bump k8s version to v1.26.

[tenzen-y]

I will create a PR to upgrade k8s to v1.27 after #1769 is done.

[tenzen-y]

I found that k8s v1.24 is blocking other tasks.

[tenzen-y]

So I will upgrade k8s version, first.