Skip to content

fix(spur-k8s): give SPUR_PEER_NODES resolvable pod names - #807

Open
pre wants to merge 1 commit into
ROCm:mainfrom
silogen:pr/peer-dns
Open

fix(spur-k8s): give SPUR_PEER_NODES resolvable pod names#807
pre wants to merge 1 commit into
ROCm:mainfrom
silogen:pr/peer-dns

Conversation

@pre

@pre pre commented Sep 2, 2026

Copy link
Copy Markdown

Related:

Motivation

In Pod mode SPUR_PEER_NODES is unusable, so no distributed workload can start.

Every Kubernetes node registers the single operator address as its agent address.
The peer list the controller dispatches therefore holds that one address repeated
once per node, and the variable reads
"<operator-ip>:6818,<operator-ip>:6818,<operator-ip>:6818". An MPI or torchrun
launcher that reads it addresses the operator three times and never reaches a
peer. A single node job shows the same value, so it was never usable in Pod mode.

Technical Details

The Pods of a multi-node job already resolve each other. launch_job sets the Pod
hostname to the target node name and the subdomain to the job's headless Service,
so <node>.<job-service>.<namespace>.svc.cluster.local is a working name for each
peer.

SPUR_PEER_NODES now carries those names, in nodelist order, so index N is the
peer whose SPUR_NODE_RANK is N. A small headless_peer_dns helper builds them.

A single node job has no headless Service and therefore no name to publish. The
variable is left unset rather than holding an address that resolves to nothing,
so a launcher fails loudly instead of hanging on a dead address.

This also corrects the message the operator returns when no SpurJob carries the
requested job id. It said "agent unreachable", although the operator is running and
answering. It now says that Pod mode launches only a SpurJob, and that a job from
sbatch or spur submit has nothing for the operator to launch.

Related:

Test Plan

  1. Run a three node job in Pod mode and read SPUR_PEER_NODES inside every Pod.
  2. Resolve each name from inside a Pod and compare against the real Pod IPs.
  3. Run a single node job and confirm the variable is absent.
  4. Unit tests for the name builder, including the nodelist ordering.

Test Result

Environment: three node RKE2 cluster on cloud VMs, 8 vCPU and 96 GiB each, no GPU.

  • All three Pods of a three node job read the same list of three names, in nodelist
    order. Each name resolved from inside a Pod to the Pod IP of the matching node.
  • Before the change the same job read one address three times.
  • A single node job left SPUR_PEER_NODES unset.
  • cargo clippy --workspace --exclude spur-ffi --all-targets --locked reports
    nothing; cargo test --locked passes 3528 tests, including three new ones for
    the name builder.

Submission Checklist

In Pod mode every Kubernetes node registers the one operator address as
its agent address, so the peer list the controller dispatches held that
address repeated once per node. SPUR_PEER_NODES therefore read
"<operator-ip>:6818,<operator-ip>:6818,<operator-ip>:6818" and no
distributed workload could reach a peer with it. A single node job showed
the same value, so it was never usable.

The Pods of a multi-node job already resolve each other: launch_job sets
the Pod hostname to the target node and the subdomain to the job's
headless Service. SPUR_PEER_NODES now carries those names, in nodelist
order, so index N is the peer whose SPUR_NODE_RANK is N.

A single node job has no headless Service and therefore no name to
publish, so the variable is left unset instead of holding an address that
resolves to nothing.

Also correct the message the operator returns when no SpurJob carries the
job id. It said "agent unreachable", although the operator is running and
reachable; it now says that Pod mode launches only a SpurJob, and that a
job from sbatch or spur submit has nothing to launch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant