fix(spur-k8s): give SPUR_PEER_NODES resolvable pod names - #807
Open
pre wants to merge 1 commit into
Open
Conversation
In Pod mode every Kubernetes node registers the one operator address as its agent address, so the peer list the controller dispatches held that address repeated once per node. SPUR_PEER_NODES therefore read "<operator-ip>:6818,<operator-ip>:6818,<operator-ip>:6818" and no distributed workload could reach a peer with it. A single node job showed the same value, so it was never usable. The Pods of a multi-node job already resolve each other: launch_job sets the Pod hostname to the target node and the subdomain to the job's headless Service. SPUR_PEER_NODES now carries those names, in nodelist order, so index N is the peer whose SPUR_NODE_RANK is N. A single node job has no headless Service and therefore no name to publish, so the variable is left unset instead of holding an address that resolves to nothing. Also correct the message the operator returns when no SpurJob carries the job id. It said "agent unreachable", although the operator is running and reachable; it now says that Pod mode launches only a SpurJob, and that a job from sbatch or spur submit has nothing to launch.
pre
requested review from
biluriuday,
sajmera-pensando,
sgopinath1,
shiv-tyagi and
yansun1996
as code owners
September 2, 2026 12:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related:
Motivation
In Pod mode
SPUR_PEER_NODESis unusable, so no distributed workload can start.Every Kubernetes node registers the single operator address as its agent address.
The peer list the controller dispatches therefore holds that one address repeated
once per node, and the variable reads
"<operator-ip>:6818,<operator-ip>:6818,<operator-ip>:6818". An MPI or torchrunlauncher that reads it addresses the operator three times and never reaches a
peer. A single node job shows the same value, so it was never usable in Pod mode.
Technical Details
The Pods of a multi-node job already resolve each other.
launch_jobsets the Podhostname to the target node name and the subdomain to the job's headless Service,
so
<node>.<job-service>.<namespace>.svc.cluster.localis a working name for eachpeer.
SPUR_PEER_NODESnow carries those names, in nodelist order, so index N is thepeer whose
SPUR_NODE_RANKis N. A smallheadless_peer_dnshelper builds them.A single node job has no headless Service and therefore no name to publish. The
variable is left unset rather than holding an address that resolves to nothing,
so a launcher fails loudly instead of hanging on a dead address.
This also corrects the message the operator returns when no
SpurJobcarries therequested job id. It said "agent unreachable", although the operator is running and
answering. It now says that Pod mode launches only a
SpurJob, and that a job fromsbatchorspur submithas nothing for the operator to launch.Related:
Test Plan
SPUR_PEER_NODESinside every Pod.Test Result
Environment: three node RKE2 cluster on cloud VMs, 8 vCPU and 96 GiB each, no GPU.
order. Each name resolved from inside a Pod to the Pod IP of the matching node.
SPUR_PEER_NODESunset.cargo clippy --workspace --exclude spur-ffi --all-targets --lockedreportsnothing;
cargo test --lockedpasses 3528 tests, including three new ones forthe name builder.
Submission Checklist