Multi-pod jobs on Slurm: one allocation per gang, started by a per-site launcher

[matbun]

Distributed training and Ray clusters reach interLink as several pods (a Kubeflow Trainer JobSet, a KubeRay RayCluster). Offloaded one by one, they become separate Slurm jobs that start at different times on unrelated nodes, so a rank can wait for its peers until the walltime runs out. We run a version of the Slurm plugin that starts them as one Slurm job, and would like to know whether you want this upstream and in what shape. Draft PR: interlink-hq/interlink-slurm-plugin#173.

How it works:

• The pods of one job carry Kueue's pod-group keys (label kueue.x-k8s.io/pod-group-name, annotation kueue.x-k8s.io/pod-group-total-count), written by whoever groups them: Kueue, a policy engine or the user. The virtual kubelet is unchanged and knows nothing about gangs.
• The plugin buffers the members until the group is complete, then submits one sbatch --nodes=N. Each pod keeps its own status and log.
• How the job starts is site knowledge (modules, srun or mpirun, PMIx, fabric libraries, NCCL settings), so it lives in one bash file per site, mounted into the plugin (SiteScriptPath). A pod annotated interlink.eu/launcher: torchrun is started by the site's torchrun-launcher function, which starts every rank itself. Without the annotation each pod runs its own command on its own node.
• An optional Kueue admission check asks the plugin's POST /estimate when Slurm would start the job (sbatch --test-only), so Kueue admits it only when the site can run it soon and does not hold quota while it waits.

PyTorch DDP, MPI, TensorFlow, Ray Train and Ray Tune have run this way on 2 nodes with 4 H100 each at one site. The logs show NCCL on InfiniBand between the nodes: 134 GB/s all-reduce bus bandwidth, against 4.1 GB/s with InfiniBand disabled.

Questions for you:

1. Is gang support welcome in the Slurm plugin, or should it stay outside?
2. Is Kueue's pod-group label the right contract, or should the plugin also read Volcano's?
3. Should the site file be part of the plugin config in the Helm chart?
4. The admission check calls the plugin directly, since the interLink API server does not proxy /estimate. Would you add it to the API, or keep it a side channel?

[matbun]

Update: we split this so that the plugin does not have to buffer pods when the caller can send the whole group at once.

Why a pod group: a distributed job is a set of pods that only makes sense when they start together, and every layer that schedules gangs already describes it that way. Kubernetes itself adds Workload and PodGroup to scheduling.k8s.io (KEP-4671, alpha in 1.35). Volcano and the coscheduling plugin each have a PodGroup with minMember (Volcano, scheduler-plugins). Kueue admits a job as a Workload made of PodSets, and plain pods as a group named by kueue.x-k8s.io/pod-group-name (Kueue). Handing interLink the group in one call lets the plugin turn it into one allocation, whichever of these formed it, without guessing when the last pod has arrived.

• feat: /createPodGroup, the pods of one pod group in a single call #606 adds POST /createPodGroup to the API server: a caller that holds every pod of a group sends them in one call, each prepared as for /create. It is optional, and the virtual kubelet path does not change.
• feat: gang job for a pod group sent whole, and POST /estimate interlink-slurm-plugin#174 is the gang job behind it: one sbatch --nodes=N for the group, started by the site's launchers, plus POST /estimate. No buffering.
• feat: buffer pod groups that /create receives pod by pod interlink-slurm-plugin#173 now holds only the buffer, for groups that the virtual kubelet sends pod by pod with Kueue's pod-group keys. They end up in the same gang job.

The first two can be reviewed and merged without the third. If you would rather not have gangs on the virtual kubelet path, #173 can stay out.