--enable-gang-scheduling=true doesn't work for MPIJob

[cheimu]

Hi tranining-operator experts.

When I set --enable-gang-scheduling=true for training-operator
, they worked for TFJob, PyTorchJob, .etc, but it doesn't work for MPIJob.

2022-03-15T03:30:18.191Z        DEBUG   controller-runtime.manager.events       
Warning {"object": {"kind":"MPIJob","namespace":"pytorch-job-test","name":"transformer-glue-4node","uid":"593289df-28b6-4f78-aee3-d6d6a3728cce","apiVersion":"kubeflow.org/v1","resourceVersion":"728417214"}, 
"reason": "SettedPodTemplateSchedulerName", "message": "Another scheduler is specified when gang-scheduling is enabled and it will not be overwritten"}

However, the manifest that I submitted doesn't set any SchedulerName

My tranining-operator version is [training-operator:6c115f6e00e3f2c979c6aa4bf2d93906a646b99d] Could anyone give a hint or help? Thank you in advance!

[zw0610]

Hi, I double-checked with the latest master branch (1f3f29a0bbac00e0410b8fc9b0f32cdf710dda30) and was not able to reproduce the issue. After setting --enable-gang-scheduling=true, I filed the MPIJob under the example directory and was able to observe the creation of the PodGroup