Support kubeflow's MPIJob

[alculquicondor]

That is kubeflow's mpi-operator. We could have started with other custom jobs, but this one seems important enough for our audience.

They currently don't have a suspend field, so we need to add it. Then, we program the controller based on the existing kueue job-controller.

/label feature
/size L
/priority important-longterm

[k8s-ci-robot]

@alculquicondor: The label(s) /label feature cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor

In response to this:

That is kubeflow's mpi-operator. We could have started with other custom jobs, but this one seems important enough for our audience.

They currently don't have a suspend field, so we need to add it. Then, we program the controller based on the existing kueue job-controller.

/label feature
/size L
/priority important-longterm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

/kind feature

[zvonkok]

Just out of curiosity how many ranks are you going to test? Are you also looking into different distributions frameworks besides Horovod?

[denkensk]

Just out of curiosity how many ranks are you going to test?

Any suggestions? I think our first step is to choose a small number of ranks (for example 3 ) to ensure that the whole process is feasible.

Are you also looking into different distributions frameworks besides Horovod?

I think Kueue is agnostic about the framework. Whether it is Horovrd or pytorch, as long as it can be launched by MPI through mpi-operator, it is fine.

[zvonkok]

@ArangoGutierrez Help me out, what was the critical amount of ranks aka nodes where we have seen bad scaling for the MPI operator?

[alculquicondor]

Sorry, we are not planning to implement an MPIJob. We are planning to support queuing for the existing kubeflow mpi-operator https://github.com/kubeflow/mpi-operator/tree/master/v2

I think your questions fit better in that repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kerthcet]

/remove-lifecycle stale

[alculquicondor]

/lifecycle frozen

[alculquicondor]

Work on mpi-operator will initiate soon kubeflow/mpi-operator#504

The kueue size of things is currently blocked on #369

[mimowo]

/assign

[mimowo]

Starting the work under #578.

IIUC the work isn't strictly blocked on #369. Moreover, the working initial implementation of the MPI integration may help us to better abstract out the interfaces, which could happen as a follow up. Adding the interfaces prior to the MPI integration also makes sense, but we then may need to adapt them, but this may anyway happen with future framework integrations, so maybe there is no point block over another, but just try to align in the process.

[tenzen-y]

Starting the work under #578.

IIUC the work isn't strictly blocked on #369. Moreover, the working initial implementation of the MPI integration may help us to better abstract out the interfaces, which could happen as a follow up. Adding the interfaces prior to the MPI integration also makes sense, but we then may need to adapt them, but this may anyway happen with future framework integrations, so maybe there is no point block over another, but just try to align in the process.

I'm ok with either using the job interface or not since releasing v0.3 isn't blocked by this feature.
cc: @kerthcet @alculquicondor

[kerthcet]

Either, just hope to avoid repetitive work.

[alculquicondor]

I think @kerthcet and @mimowo can make progress in parallel. Please stay in touch.