Alpha integration with prerelease of Kubeflow Trainer v2

[mimowo]

What would you like to be added:

Currently, Kubeflow Trainer v2 has prerelease, so it would be good to have an "alpha" integration to detect possible issues.

Why is this needed:

Short term to provide early feedback and make sure there are no roadblocks for the full integration in #3884

[mimowo]

cc @tenzen-y @andreyvelich @mwysokin

[tenzen-y]

This is duplicated with #3884

[mimowo]

This is duplicated with #3884

I see, I'm ok to close then, but maybe we can rename as "Alpha integration with Kubeflow Trainer v2" using the pre-release for early feedback?

[mimowo]

/retitle Alpha integration with prerelease of Kubeflow Trainer v2

[tenzen-y]

One big problem is that the TrainJob nodeSelector (.spec.podSpecOverrides[*].nodeSelector) and tolerarations (.spec.podSpecOverrides[*].tolerations) are immutable fields: https://github.com/kubeflow/trainer/blob/b71a69064fa033c2ca51f6a24dbe199b552cada9/pkg/apis/trainer/v1alpha1/trainjob_types.go#L113-L116

We need to relax this validation when TrainJob is suspended.

[mimowo]

Yes, so doing the Alpha integration may uncover more issues.

[tenzen-y]

Yes, so doing the Alpha integration may uncover more issues.

Opened: kubeflow/trainer#2679