Release v1.0.5

A large bug-fix and hardening release driven by two rounds of deep code
review, with end-to-end validation on El Capitan-class (MI300A/Flux)
systems.

GPU assignment and the torchrun trampoline

Workers no longer initialize HIP/CUDA at import time with all GPUs
visible, which pinned every rank on a node to physical GPU 0; each
rank now sees only its granted devices, and the memory-fraction cap
applies to the device actually selected.
The trampoline puts the user's own code on sys.path (sibling imports
work under --local, subdirectories, and custom launch dirs), destroys
the process group on every exit path, publishes rank identity itself,
and only passes device_id to init_process_group for accelerators, so
CPU/gloo jobs start again.

Correctness of generated launches

Job exit codes propagate through Scheduler.launch to the CLI exit
status; SIGINT kills the scheduler child instead of orphaning it.
User-controlled values are shell-quoted in generated scripts, closing
breakage and injection holes.
Each launch picks a unique rendezvous port instead of a hardcoded one.
Ephemeral blocking launches stream output, are supervised, and expand
their CLI environment in-process; non-blocking immutable scripts now
receive the full launcher environment.

Scheduler backends and system profiles

LSF: bsub flags are proper argv tokens, submit-only flags no longer
leak into jsrun, get_job_id is implemented, and --smpiargs quoting is
fixed. Sierra CPU-affinity binding follows the requested procs/node,
and the rzansel hostname typo means RZAnsel is detected again.
Scheduler argument dicts and System env lists are per-instance, so
state no longer leaks between jobs in one process; SystemParams is
copied before overrides; scheduler autodetection and -p scheduler=
are honored.
The RCCL/NCCL environment derives from torch's bundled ROCm rather
than ROCM_PATH, NCCL_NET is set only when the libfabric plugin is
actually present, unversioned ROCM_PATH no longer crashes, and El
Capitan MIOpen cache paths tolerate an unset TMPDIR. Adds a Corona
profile and a ROCm 7.x PyTorch behavior variable.

CLI and packaging

Torchrun passthrough arguments respect the launcher/script boundary,
handle abbreviated flags, and validate identically across rendezvous
protocols; module execution via -m is supported; -x/--xargs accepts
the schedulers' real dashed keys; missing commands and same-file
--out/--err fail with clean errors; resource-validation failures are
visible and actionable.
GPU vendor libraries moved from install_requires to [rocm]/[cuda]
extras, so wheels no longer bake in the build machine's hardware; a
new [rocm-auto] extra pins amdsmi to the installing machine's ROCm
release for source installs.
Launch directory names include a UUID for uniqueness; documentation
that contradicted actual behavior was corrected.

Tests and CI

Extensive new regression coverage (GPU visibility, argument
boundaries, state leaks, packaging reproducibility, docs consistency);
several tautological or shadowed tests now assert what they meant to;
torch is installed on every CI leg and its absence fails instead of
skipping; the minimum Python for CI was raised.

Navigation Menu