feat: add one-node Slurm runtime

[nabinchha]

Summary

Adds the one-node allocation-local runtime for resolved Slurm plans. It verifies persisted launch intent, launches server/endpoint/client steps, publishes monotonic state, proxies model traffic, and owns complete cleanup through focused runtime collaborators.

Related Issue

Part of #868 (868#2).

Merge Order and Dependencies

• The feat: add Slurm state writer #908 state-writer prerequisite has merged into feat/slurm-execution. This branch is rebased directly onto that merged base.
• The client step targets the worker owned by Implement the allocation-local Data Designer client worker #876. A real allocation E2E remains blocked until that worker is available in the client image.

Changes

Added

• A deterministic, content-addressed runtime bundle with restrictive atomic publication.
• One-node allocation preflight for scheduler identity, GPU shape, attempt ownership, artifact digests, writable mounts, and ports.
• Shell-free srun construction, process-group supervision, monotonic state publication, and a bounded loopback least-connections proxy.
• An entrypoint-owned TerminationSignalCoordinator for handler installation, spawn deferral, cleanup blocking, and deferred delivery.
• Explicit persisted RESTARTING readiness and execution-scoped restrictive log directories keyed by the restart revision.

Changed

• Decomposed controller, supervisor, log ownership, step construction, record publication, and entrypoint workflows into short, single-purpose helpers.
• Bundled the complete data_designer.slurm package in a deterministic source manifest and import-checked every manifested module after extraction.
• Reused the public plan/state validation boundary for runtime completion and immutable winner finalization.
• Made the supervisor the sole owner of child-process cleanup and required its signal coordinator explicitly.
• Extracted runtime-archive publication behind a focused collaborator with concurrency-safe recovery.

Fixed

• Resumed requeued RUNNING attempts through an explicit restart epoch while preserving monotonic readiness revisions.
• Reset stale READY/PUBLISHED state before restart preflight so observers do not see unavailable services as ready.
• Prevented requeues from colliding with fixed O_EXCL log filenames by isolating every execution's logs.
• Bound each server step with one Slurm mask_gpu covering its assigned GPUs without manually forwarding CUDA_VISIBLE_DEVICES.
• Tried each eligible proxy backend at most once and passed the final 429/Retry-After response to the caller.
• Deferred termination across child spawn/registration without calling main-thread-only signal APIs from the supervisor.
• Published STOPPED only after every managed process was confirmed exited.
• Preserved CLIENT_FAILED classification for partial or failed client results before candidate-manifest loading.
• Removed the compute-node runtime's dependency on an ambient data-designer-slurm installation.

Attention Areas

Reviewers: Please pay special attention to:

• runtime/controller.py — lifecycle ordering, explicit restart publication, and terminal-state publication.
• runtime/logs.py — execution-scoped path binding and restrictive descriptor-safe creation.
• runtime/signals.py — centralized signal ownership and deferred delivery.
• runtime/supervisor.py — process ownership and cleanup confirmation.
• runtime/bundle.py — deterministic complete-package source packaging.
• integration.py — shared plan-aware candidate validation.

Testing

• make check-slurm
• make test-slurm — 1,183 passed after rebasing onto merged feat: add Slurm state writer #908
• make test-slurm-wheel-install — CLI overhead -0.001s
• Focused bundle, supervisor, controller, restart-readiness, and integration regressions
• Allocation E2E — blocked on Implement the allocation-local Data Designer client worker #876 and access to a Slurm allocation

Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs: N/A — this implements the existing Implement Slurm submission and allocation runtime #868 runtime lane

---

Description updated with AI

[greptile-apps]

Greptile Summary

The PR adds an allocation-local one-node Slurm runtime and completes the previously requested restart recovery behavior.

• Stages and verifies a deterministic runtime bundle for compute-node execution.
• Orchestrates preflight, server, endpoint, client, supervision, signal handling, and cleanup.
• Resumes persisted RUNNING attempts through a monotonic RESTARTING readiness epoch with execution-scoped logs.
• Extends candidate validation and readiness reconciliation for runtime completion and restart transitions.

Confidence Score: 5/5

The PR appears safe to merge because the previously reported restart failures are addressed and no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/data-designer-slurm/src/data_designer/slurm/runtime/controller.py	Adds allocation lifecycle orchestration and now reloads persisted readiness before publishing a monotonic restart epoch for requeued running attempts.
packages/data-designer-slurm/src/data_designer/slurm/state/readiness.py	Adds the explicit RESTARTING state and validates reset deployment snapshots consistently.
packages/data-designer-slurm/src/data_designer/slurm/state/reconciliation.py	Allows nonterminal readiness states to enter a reset restart epoch while preserving terminal-state restrictions and revision monotonicity.
packages/data-designer-slurm/src/data_designer/slurm/runtime/supervisor.py	Centralizes shell-free process launch, required-process checks, termination, and cleanup confirmation.
packages/data-designer-slurm/src/data_designer/slurm/runtime/bundle.py	Builds and atomically publishes a deterministic, restrictive allocation runtime archive.
packages/data-designer-slurm/src/data_designer/slurm/integration.py	Splits candidate and winner validation while retaining identity, count, path, digest, and chronology checks.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Load persisted attempt] --> B{Attempt state}
    B -->|SUBMITTED or PENDING| C[Run allocation preflight]
    B -->|RUNNING| D[Load persisted readiness]
    D --> E[Publish next-revision RESTARTING state]
    E --> C
    C --> F[Mark attempt RUNNING]
    F --> G[Launch servers]
    G --> H[Probe backends]
    H --> I[Launch and probe endpoints]
    I --> J[Publish READY]
    J --> K[Run client and validate candidate]
    K --> L[Clean up managed processes]
    L --> M[Publish STOPPED]
    M --> N[Persist terminal attempt]
    C -->|failure| O[Publish failure state]
    G -->|failure| O
    H -->|failure| O
    I -->|failure| O
    K -->|failure| O
    O --> L

Loading

_{Reviews (11): Last reviewed commit: "fix: make allocation restarts explicit" | Re-trigger Greptile}

[greptile-apps]

Restart resets readiness revision

When the runtime restarts an attempt already in RUNNING state with a persisted readiness snapshot, _readiness remains unset and this code publishes revision 1 again. SlurmStateWriter.write_readiness rejects that non-monotonic update, causing the otherwise executable attempt to enter the failure path instead of resuming.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/data-designer-slurm/src/data_designer/slurm/runtime/controller.py
Line: 335-336

Comment:
**Restart resets readiness revision**

When the runtime restarts an attempt already in `RUNNING` state with a persisted readiness snapshot, `_readiness` remains unset and this code publishes revision 1 again. `SlurmStateWriter.write_readiness` rejects that non-monotonic update, causing the otherwise executable attempt to enter the failure path instead of resuming.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[nabinchha]

Fixed in a5ece02. A requeued RUNNING attempt now loads its persisted readiness snapshot before launching, continues at the next revision, and suppresses PENDING/STARTING publications that would regress an already advanced readiness state. Regressions cover persisted STARTING and READY snapshots and verify revision 7 resumes at revision 8.

[andreatnvidia]

[P3] One thing that may get awkward as this runtime grows: glob("*.py") only includes modules directly under runtime/. Moving code into a subpackage for the multi-node work would behave normally in the checkout but leave those files out of the allocation bundle. It might be worth packaging a dedicated runtime package, or recursively collecting an explicit manifest and checking that the extracted bundle imports cleanly.

[nabinchha]

Fixed in e9d254a. Runtime source collection is now recursive and deterministic, and the archive contains an explicit runtime-sources.txt manifest. Tests stage a synthetic nested package, import it from the extracted bundle, and import every module listed by the real bundle manifest. The isolated wheel-install check also passes.

[andreatnvidia]

[P3] We now have the candidate rules in a few places: here, ClientResult, and PlanStateValidator.validate_finalization_chain. That makes resume or winner-policy changes easy to update on one side but miss on another, so allocation success and winner finalization could drift apart. Could PlanStateValidator expose a shared validate_client_candidate(...) step that both paths call?

[nabinchha]

Fixed in e9d254a. PlanStateValidator.validate_client_candidate(...) is now the authoritative cross-record check for identities, counts, resume policy, dataset/manifest location, digest, and timestamps. Both runtime result loading and validate_finalization_chain(...) call it; the runtime boundary still classifies partial/failed client outcomes as CLIENT_FAILED before candidate loading. Added direct pre-terminal and boundary-order regressions.

[andreatnvidia]

[P3] The signal handling is spread across the entrypoint and supervisor now: one installs handlers, another temporarily replaces them, and cleanup masks them again. It works for the current main-thread flow, but start() fails if launch work later moves to a background thread, and future signal changes would need coordinated edits in several places. A small signal coordinator owned by the entrypoint and passed into the supervisor would keep this easier to reason about.

[nabinchha]

Fixed in e9d254a. TerminationSignalCoordinator is created by the entrypoint and passed explicitly into StepSupervisor. It now owns handler installation, deferred delivery, and cleanup-time blocking; the supervisor no longer installs or replaces handlers itself. A regression launches start() on a worker thread, and the spawn-race test confirms deferred termination is replayed only after registration and cleanup owns the child.

[andreatnvidia]

[P2] I think there’s still a deployment dependency hiding here. The archive contains only the runtime modules, while the package shim and entrypoint load SlurmStateWriter and the other contracts from whatever data-designer-slurm installation happens to be available on the compute node. An older or missing installation will fail before the runtime starts, and the same bundle digest could execute against different state code. Do we guarantee the exact wheel on every node? Otherwise, could the bundle include or verify the required package version?

[nabinchha]

Fixed in 823c77d. The content-addressed archive now carries the complete data_designer.slurm Python package—including state, contracts, planning, client, and runtime—instead of extending an ambient installation. The bundle digest therefore pins the exact Slurm implementation used on the compute node. Tests assert the package contents and import every module from the extracted manifest; make test-slurm (1,173 tests), make check-slurm, and the isolated wheel-install suite all pass.

[greptile-apps]

Requeued attempts are terminally failed

When Slurm reinvokes an array task whose persisted attempt is already RUNNING, the deterministic attempt directory causes the same manifest to be loaded, but this validation rejects it before launching any steps and the controller persists FAILED. No production path in this package creates the replacement attempt required for recovery, so the shard remains unexecuted.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/data-designer-slurm/src/data_designer/slurm/runtime/controller.py
Line: 282-289

Comment:
**Requeued attempts are terminally failed**

When Slurm reinvokes an array task whose persisted attempt is already `RUNNING`, the deterministic attempt directory causes the same manifest to be loaded, but this validation rejects it before launching any steps and the controller persists `FAILED`. No production path in this package creates the replacement attempt required for recovery, so the shard remains unexecuted.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[nabinchha]

Fixed in a5ece02. RUNNING is now an executable recovery state for the same scheduler-owned attempt: the controller reloads nonterminal readiness, restarts the allocation-owned preflight/server/endpoint/client process sequence, and preserves monotonic readiness through completion. FAILED or STOPPED readiness remains non-restartable. The STARTING and READY requeue regressions complete successfully and the full Slurm suite passes (1,178 tests).

[andreatnvidia]

[P1] One recovery edge still looks like it will block real requeues. A RUNNING attempt keeps the same attempt directory, and SubprocessStepRunner creates fixed log names with O_EXCL. Once a step has created its .out or .err files, restarting it hits FileExistsError before anything can relaunch. The requeue test uses the fake runner, so it misses this path. Could each execution use its own log directory, or could the existing artifacts be handled explicitly before restarting?

[nabinchha]

Fixed in 69574fa. Every controller execution now binds all preflight, server, endpoint, and generation steps to logs/execution-<initial-readiness-revision>/, while the restrictive O_EXCL creation rule remains intact. Requeues therefore preserve prior logs and create a fresh namespace. The production subprocess regression now launches the same step ID successfully in two execution directories.

[andreatnvidia]

[P2] There’s also a window where the persisted status can be misleading. If the previous snapshot is READY, the PENDING and STARTING updates are skipped while the old processes are gone and the replacement services are still booting. Anyone reading state continues to see READY/PUBLISHED during that restart. Could recovery use an execution epoch or another transition that keeps readiness accurate without losing monotonic revisions?

[nabinchha]

Fixed in 69574fa. A requeued RUNNING attempt now publishes an explicit next-revision RESTARTING snapshot before allocation preflight. That snapshot resets ready backends to zero and endpoint publication to pending, so observers no longer see stale READY/PUBLISHED state while services are absent. Transition validation allows this reset only from nonterminal readiness through RESTARTING, and subsequent revisions remain strictly monotonic.

[andreatnvidia]

Nice work tightening this up. The explicit restart state and execution-scoped logs cleanly handle the requeue cases we discussed, and the overall runtime flow looks solid now. LGTM!