You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
autotrainer.train_step(model, loss_fn, xb, yb, opt, scaler=...). The
all-in-one companion to prepare(optimize=True): one call per step does the
full AMP dance in the correct order - move the batch to the model's device,
forward under autocast (bf16 on modern GPUs, fp16 otherwise, no-op on CPU),
compute the loss, backward + optimizer.step() + scaler.update() (a plain backward()/step() when no scaler), and zero the grads. Returns the detached
loss for logging. Closes the last "not fully automatic" gap: prepare()
can't wrap your loop, so AMP used to be a hand-written, order-sensitive
snippet users had to copy; train_step() runs it for you. Touches nothing
about your recipe (lr/loss/schedule/optimizer). scaler is optional (leave
it None on CPU / bf16 GPUs); autocast=False opts the forward out of mixed
precision while keeping the backward/step/zero bookkeeping. Backward runs
outside autocast (the documented-correct scoping). The AMP path now also has
end-to-end tests (CPU: loss actually decreases through the helper; cuda:
the same on-device, plus a forced-enabled fp16 scaler exercising the real
scaling arithmetic) - previously only the pieces were unit-tested.
autotrainer.TrainingMonitor: training triage - explains numerical
failures in plain language. The health companion to the perf monitors:
where BottleneckMonitor/ThroughputMonitor answer "is it fast?", this
answers "is it healthy?". Call monitor.step(loss, model=..., optimizer=..., scaler=...) once per step (after backward()) and it flags, once each,
with the fix: non-finite loss (lr too high / bad inputs, with a concrete
lr/10 suggestion when the optimizer is passed), loss divergence, fp16
gradient overflow (persistent AMP-scaler backoff -> "switch to bf16 via autocast_context()"), and non-finite / spiking / vanishing gradient norms.
Gradient checks are skipped while an AMP scaler is active (grads are still
scaled there, so the scaler-backoff signal owns fp16 health). report()
prints a one-line all-clear or a recap. Everything but loss is optional;
it only observes - never touches lr/loss/schedule/optimizer - and is
opt-in with zero overhead when unused. Fully unit-tested on CPU (synthetic
losses, a fake scaler, hand-set gradients).
tune()/fit() search a much wider training recipe. The default space
now covers the LR schedule (cosine/onecycle/constant) and its warmup
fraction, gradient clipping (max_norm), and - for classification -
label smoothing, on top of the original lr / weight-decay / optimizer /
batch-size. The default is now task-aware: it's built from the model and
the inferred loss (CNNs lean on the SGD + 1cycle recipe with a higher LR
band; everything else defaults to a lower band with cosine/constant), so
trials are spent where they pay off. DEFAULT_SPACE stays as the maximal
superset of every searchable knob. A searched scheduler/grad_clip/ label_smoothing trains identically in the short trials and the phase-2
retrain via a shared _make_scheduler helper. Trials are always scored
with the plain unsmoothed loss so label-smoothing candidates stay
comparable. Does NOT search the architecture (no NAS) or the loss family
(still inferred).
ASHA (successive-halving) is the default tuning pruner, replacing the
median pruner. Most trials get a small budget and only survivors are
promoted - the multi-fidelity strategy that keeps the now-wider search
affordable. min_resource=1, so a trial is never pruned before its second
epoch (single-epoch trials behave exactly as before). Override with tune(..., pruner=...).
lr is coupled to batch_size when both are searched: the applied LR
is scaled toward the trial's batch (linear for SGD, square-root for
Adam-family, relative to a batch of 32) so the search doesn't burn trials
rediscovering that relationship. best_params still records the un-scaled
searched LR. Toggle with tune(..., lr_scaling="none") / fit(..., lr_scaling="none"); applied consistently across the search and
the phase-2 retrain.
fit(..., test_loader=...): an honest held-out generalization estimate.
The final (best-epoch) model is scored on a loader the search never saw;
the number is printed and stored on rank 0's study.user_attrs["test_loss"]. Guards against reading too much into a val
loss that a wide search has implicitly optimized against. The (model, best_params, study) return shape is unchanged.
prepare() auto-launches multi-GPU workers. On a box with ≥2 visible
GPUs, a bare python train.py now distributes across all of them with no
launcher: the first prepare() call detects it's a fresh parent process
(no RANK/WORLD_SIZE env set), spawns one worker per GPU via the same
spawn+supervisor machinery the autotrainer run CLI uses, and the parent sys.exits without entering the training loop. Each worker re-enters the
script with RANK set and proceeds through prepare() normally (DDP wrap,
distributed sampler). Closes the second vision gap: "drop a script in and
it auto-detects single AND multi-GPU." Spawn only fires when all three hold
(no RANK set, not under SLURM, detect() reports local_multi_gpu), so
it never double-spawns under srun or loops on already-launched workers.
Opt out with prepare(..., auto_launch=False). Each spawned child is
pinned to its own GPU via per-child CUDA_VISIBLE_DEVICES (the torchrun
pattern) so workers don't race on set_device.
prepare(..., static_graph=True): when distributed (DDP path), enables
DDP's static_graph=True plus gradient_as_bucket_view=True - free
wins when the computation graph is the same every iteration (static graph
skips per-iteration graph-recording overhead after the first step;
bucketing lowers peak memory). Both are opt-in because they have
correctness implications when the graph genuinely changes (conditional
execution, varying depth). No-op on single-device and the FSDP path. Does
NOT touch lr / loss / schedule / optimizer.
prepare(..., find_unused_parameters=True): forwards DDP's find_unused_parameters=True for models that don't touch every param each
step (e.g. conditional branches). Mutually exclusive with static_graph
(torch forbids the combination; prepare raises a clear ValueError).
autotrainer.ThroughputMonitor: throughput (samples/sec), peak GPU memory,
and a rough model-FLOPS-utilization (MFU) estimate. The companion to BottleneckMonitor - where that answers "am I waiting on the loader?"
(a ratio), this answers "how fast am I going, and is that fast relative to
the GPU's peak?" (absolute numbers). MFU is opt-in (pass model_flops);
the GPU-peak denominator comes from a small spec table (A100/H100/common
consumer parts) or is reported as None for unknown hardware rather than
invented. Bookkeeping is unit-tested on CPU; the real-GPU memory read is
gated on the cuda marker. MFU is a diagnostic, not a benchmark - it
assumes a matmul-heavy model.
Multi-rank FSDP test coverage: the FSDP wrap + use_orig_params param-
addressability path now runs against a real process group in test_distributed.py (CPU-gloo), closing the gap that the single-process
no-op test left open. The full sharded fwd+bwd+step is gated on >= 2
usable GPUs (torch 2.13 FSDP won't run a forward with CPU params when cuda.is_available() is True), and is left to a multi-GPU runner.
timeout-minutes: 30 on the test-cuda CI job, so an in-execution hang
(driver crash, stuck NCCL collective, deadlocked test) fails loudly after
30 minutes instead of running silently. Note this does not cover the
"queued forever because the runner is offline" case - the timeout only
starts once a runner picks the job up.
.github/dependabot.yml: weekly dependency-update checks for the Python
(pip) ecosystem and the GitHub Actions versions used in CI.
autotrainer.configure_nccl(): sets NCCL_SOCKET_IFNAME to the
default-route interface when unset, so multi-node SLURM jobs don't hang
or crawl because NCCL guessed the wrong (or loopback) network interface.
Non-clobbering (setdefault semantics, like configure_scratch); an
optional debug=True turns on NCCL_DEBUG=INFO and prints a hint when
detection fails (no ip binary, Windows host, stripped container) instead
of a silent hang. Detection is isolated in _detect_primary_interface()
(fails closed -> None); the env-var logic is fully unit-tested via
monkeypatch. Real multi-node validation remains a 1.0-gate item.
scripts/provision-runner-python.ps1: automates the self-hosted GPU
runner's system-wide Python install (the test-cuda CI job's env.PYTHON
points at C:\Python<ver>, which the NETWORK SERVICE runner account
must be able to read). Idempotent: re-running with the same -Version is
a no-op; changing -Version re-provisions and reminds the operator to
update env.PYTHON in ci.yml. Requires admin + uv; RUNNER_SETUP.md
now points at it as the recommended path.
prepare(auto_bs=True) now prints a one-line note when the forward-only
sweep is used (no loss_fn): "auto_bs running forward-only (no loss_fn);
pass loss_fn for a larger batch size". Without a loss the sweep measures
only activations+params, not grads + optimizer state, so the picked size
is safe but smaller than a real fwd+bwd sweep would allow; the note
surfaces why instead of leaving the user to find it in the docstring.
Silent when a loss_fn is given (the sweep is then real).
Changed
prepare() now defaults optimize=True (was False). On a GPU this
turns on the free-win bundle by default — TF32, cudnn.benchmark for CNNs, num_workers/pin_memory/persistent_workers defaults on bare loaders,
and AMP — without touching lr/loss/schedule/optimizer. This closes the
"drop a script in and it runs GPU-optimized" gap: a bare prepare(model, loader) no longer leaves the GPU wins on the table. The
bundle is a documented no-op on CPU (every flag gates on a visible CUDA
device), so CPU-only callers are unaffected. Pass optimize=False to opt
out. When the bundle fires on a GPU with AMP on, prepare() now also
prints the exact two-line autocast_context() + GradScaler() snippet to
add to the training loop (we can't wrap an arbitrary loop from inside prepare; the helpers are no-ops on CPU so the snippet is safe verbatim). auto() and fit() inherit the new default automatically.
Repository-hygiene cleanup (no behavior change, no public-API change):
Refreshed stale version references: SECURITY.md supported-versions
table now reflects the current release line (0.11.x, was 0.7.x);
the bug-report issue template placeholder updated to 0.11.0.
Normalized GitHub URL casing to the canonical OriAlpha/Autotrainer
across README.md, CONTRIBUTING.md, SECURITY.md, pyproject.toml,
and issue templates (GitHub redirects case-insensitively, but the
canonical form is now used everywhere).
RUNNER_SETUP.md generalized: removed maintainer/machine-specific
details (GPU model, driver version, runner name, fixed paths) so it
reads as a reusable guide for any contributor setting up a self-hosted
GPU runner.
NEXT_STEPS.md moved out of the published repo (it was an internal
engineering backlog); it is no longer tracked. Maintainers keep it
locally under the gitignored docs/internal/.
fitting.py split: the phase-1 search and checkpoint helpers
(_unwrap, _sync_from_rank0, _journal_storage, _parallel_search, _save_checkpoint, _load_checkpoint, _CHECKPOINT_FORMAT) moved to a
new _fit_search.py, leaving fitting.py (431 -> 319 LOC) as the
orchestrator + phase-2 training loop. Pure move, no behavior change; the
names are re-exported from fitting.py so existing imports keep working.
Prepares the ground for the roadmap's training-triage features by
separating the two phases before layering more onto either.
Optimize-path CUDA test stub consolidated: the per-attribute _pretend_cuda helper (patched is_available/device_count/set_device/ Tensor.to individually) is replaced by a shared pretend_cuda conftest
fixture that patches the complete public torch.cuda surface prepare() reads (CUDA_OPTIMIZE_SURFACE), on the real module object so
torch's internal torch.cuda reads still resolve. A new guard test
(test_optimize.py::TestCudaSurfaceCoverage) scans the source and fails
loudly if prepare()/_optimize/utils reach for a public torch.cuda
attribute the fixture doesn't stub - closing the "silently exercises the
CPU path on a CPU-only box" gap that bit PR #1. No behavior change;
pure test-hygiene improvement.
Removed
Breaking:tune() no longer accepts the train_loader=/val_loader=
keyword aliases (deprecated in 0.10). Passing them now raises TypeError
pointing at the replacement train=/val= names. The soak period elapsed;
these names were misleading for estimator inputs (which take arrays, not
loaders).
Fixed
ThroughputMonitor MFU computation crashed on Python 3.9: it used zip(..., strict=True), which is PEP 618 (3.10+ runtime syntax). The
project's requires-python and ruff target-version both target 3.9,
so the 3.9 CI cell raised TypeError: zip() takes no keyword arguments
in the three MFU tests. The two deques are appended in lockstep
(step_time + tick fire once per step), so the strict check was only a
defensive assertion; replaced with a plain zip() plus a comment
documenting the lockstep invariant.
The multi-rank FSDP wrap tests (test_fsdp_wraps_with_orig_params_over_process_group, test_fsdp_with_cpu_offload_wraps) failed on torch >= 2.13 in CPU-gloo CI
with "FSDP needs a non-CPU accelerator device, but no accelerator device is
detected". On those builds FSDP() refuses even the wrap when no CUDA/XPU
device is visible - and the harness deliberately hides CUDA (CUDA_VISIBLE_DEVICES="")
so two ranks don't fight over one GPU. The worker now catches that specific
refusal and emits a SKIP reason=fsdp-needs-accelerator line, so the test
skips cleanly on builds where torch forbids the CPU wrap while still running
the real assertion on builds where torch allows it (and where a usable GPU
exists, the existing cuda-marked full-step test covers the rest).
Examples now run standalone on the spawn start method (Windows/macOS):
every script in examples/ wraps its executable code in a main() behind if __name__ == "__main__":. Without it, prepare()/fit() setting num_workers > 0 (and sklearn's joblib/loky) spawned workers that
re-imported the module and re-ran training, crashing with "DataLoader worker
exited unexpectedly". autotrainer run and srun were already unaffected
(they run the script as __main__).