Skip to content

autotrainer 0.12.0

Choose a tag to compare

@OriAlpha OriAlpha released this 27 Jul 06:46
· 22 commits to main since this release

[0.12.0] - 2026-07-27

Added

  • autotrainer.train_step(model, loss_fn, xb, yb, opt, scaler=...). The
    all-in-one companion to prepare(optimize=True): one call per step does the
    full AMP dance in the correct order - move the batch to the model's device,
    forward under autocast (bf16 on modern GPUs, fp16 otherwise, no-op on CPU),
    compute the loss, backward + optimizer.step() + scaler.update() (a plain
    backward()/step() when no scaler), and zero the grads. Returns the detached
    loss for logging. Closes the last "not fully automatic" gap: prepare()
    can't wrap your loop, so AMP used to be a hand-written, order-sensitive
    snippet users had to copy; train_step() runs it for you. Touches nothing
    about your recipe (lr/loss/schedule/optimizer). scaler is optional (leave
    it None on CPU / bf16 GPUs); autocast=False opts the forward out of mixed
    precision while keeping the backward/step/zero bookkeeping. Backward runs
    outside autocast (the documented-correct scoping). The AMP path now also has
    end-to-end tests (CPU: loss actually decreases through the helper; cuda:
    the same on-device, plus a forced-enabled fp16 scaler exercising the real
    scaling arithmetic) - previously only the pieces were unit-tested.
  • autotrainer.TrainingMonitor: training triage - explains numerical
    failures in plain language.
    The health companion to the perf monitors:
    where BottleneckMonitor/ThroughputMonitor answer "is it fast?", this
    answers "is it healthy?". Call monitor.step(loss, model=..., optimizer=..., scaler=...) once per step (after backward()) and it flags, once each,
    with the fix: non-finite loss (lr too high / bad inputs, with a concrete
    lr/10 suggestion when the optimizer is passed), loss divergence, fp16
    gradient overflow (persistent AMP-scaler backoff -> "switch to bf16 via
    autocast_context()"), and non-finite / spiking / vanishing gradient norms.
    Gradient checks are skipped while an AMP scaler is active (grads are still
    scaled there, so the scaler-backoff signal owns fp16 health). report()
    prints a one-line all-clear or a recap. Everything but loss is optional;
    it only observes - never touches lr/loss/schedule/optimizer - and is
    opt-in with zero overhead when unused. Fully unit-tested on CPU (synthetic
    losses, a fake scaler, hand-set gradients).
  • tune()/fit() search a much wider training recipe. The default space
    now covers the LR schedule (cosine/onecycle/constant) and its warmup
    fraction, gradient clipping (max_norm), and - for classification -
    label smoothing, on top of the original lr / weight-decay / optimizer /
    batch-size. The default is now task-aware: it's built from the model and
    the inferred loss (CNNs lean on the SGD + 1cycle recipe with a higher LR
    band; everything else defaults to a lower band with cosine/constant), so
    trials are spent where they pay off. DEFAULT_SPACE stays as the maximal
    superset of every searchable knob. A searched scheduler/grad_clip/
    label_smoothing trains identically in the short trials and the phase-2
    retrain via a shared _make_scheduler helper. Trials are always scored
    with the plain unsmoothed loss so label-smoothing candidates stay
    comparable. Does NOT search the architecture (no NAS) or the loss family
    (still inferred).
  • ASHA (successive-halving) is the default tuning pruner, replacing the
    median pruner. Most trials get a small budget and only survivors are
    promoted - the multi-fidelity strategy that keeps the now-wider search
    affordable. min_resource=1, so a trial is never pruned before its second
    epoch (single-epoch trials behave exactly as before). Override with
    tune(..., pruner=...).
  • lr is coupled to batch_size when both are searched: the applied LR
    is scaled toward the trial's batch (linear for SGD, square-root for
    Adam-family, relative to a batch of 32) so the search doesn't burn trials
    rediscovering that relationship. best_params still records the un-scaled
    searched LR. Toggle with tune(..., lr_scaling="none") /
    fit(..., lr_scaling="none"); applied consistently across the search and
    the phase-2 retrain.
  • fit(..., test_loader=...): an honest held-out generalization estimate.
    The final (best-epoch) model is scored on a loader the search never saw;
    the number is printed and stored on rank 0's
    study.user_attrs["test_loss"]. Guards against reading too much into a val
    loss that a wide search has implicitly optimized against. The
    (model, best_params, study) return shape is unchanged.
  • prepare() auto-launches multi-GPU workers. On a box with ≥2 visible
    GPUs, a bare python train.py now distributes across all of them with no
    launcher: the first prepare() call detects it's a fresh parent process
    (no RANK/WORLD_SIZE env set), spawns one worker per GPU via the same
    spawn+supervisor machinery the autotrainer run CLI uses, and the parent
    sys.exits without entering the training loop. Each worker re-enters the
    script with RANK set and proceeds through prepare() normally (DDP wrap,
    distributed sampler). Closes the second vision gap: "drop a script in and
    it auto-detects single AND multi-GPU." Spawn only fires when all three hold
    (no RANK set, not under SLURM, detect() reports local_multi_gpu), so
    it never double-spawns under srun or loops on already-launched workers.
    Opt out with prepare(..., auto_launch=False). Each spawned child is
    pinned to its own GPU via per-child CUDA_VISIBLE_DEVICES (the torchrun
    pattern) so workers don't race on set_device.
  • prepare(..., static_graph=True): when distributed (DDP path), enables
    DDP's static_graph=True plus gradient_as_bucket_view=True - free
    wins when the computation graph is the same every iteration (static graph
    skips per-iteration graph-recording overhead after the first step;
    bucketing lowers peak memory). Both are opt-in because they have
    correctness implications when the graph genuinely changes (conditional
    execution, varying depth). No-op on single-device and the FSDP path. Does
    NOT touch lr / loss / schedule / optimizer.
  • prepare(..., find_unused_parameters=True): forwards DDP's
    find_unused_parameters=True for models that don't touch every param each
    step (e.g. conditional branches). Mutually exclusive with static_graph
    (torch forbids the combination; prepare raises a clear ValueError).
  • autotrainer.ThroughputMonitor: throughput (samples/sec), peak GPU memory,
    and a rough model-FLOPS-utilization (MFU) estimate. The companion to
    BottleneckMonitor - where that answers "am I waiting on the loader?"
    (a ratio), this answers "how fast am I going, and is that fast relative to
    the GPU's peak?" (absolute numbers). MFU is opt-in (pass model_flops);
    the GPU-peak denominator comes from a small spec table (A100/H100/common
    consumer parts) or is reported as None for unknown hardware rather than
    invented. Bookkeeping is unit-tested on CPU; the real-GPU memory read is
    gated on the cuda marker. MFU is a diagnostic, not a benchmark - it
    assumes a matmul-heavy model.
  • Multi-rank FSDP test coverage: the FSDP wrap + use_orig_params param-
    addressability path now runs against a real process group in
    test_distributed.py (CPU-gloo), closing the gap that the single-process
    no-op test left open. The full sharded fwd+bwd+step is gated on >= 2
    usable GPUs (torch 2.13 FSDP won't run a forward with CPU params when
    cuda.is_available() is True), and is left to a multi-GPU runner.
  • timeout-minutes: 30 on the test-cuda CI job, so an in-execution hang
    (driver crash, stuck NCCL collective, deadlocked test) fails loudly after
    30 minutes instead of running silently. Note this does not cover the
    "queued forever because the runner is offline" case - the timeout only
    starts once a runner picks the job up.
  • .github/dependabot.yml: weekly dependency-update checks for the Python
    (pip) ecosystem and the GitHub Actions versions used in CI.
  • autotrainer.configure_nccl(): sets NCCL_SOCKET_IFNAME to the
    default-route interface when unset, so multi-node SLURM jobs don't hang
    or crawl because NCCL guessed the wrong (or loopback) network interface.
    Non-clobbering (setdefault semantics, like configure_scratch); an
    optional debug=True turns on NCCL_DEBUG=INFO and prints a hint when
    detection fails (no ip binary, Windows host, stripped container) instead
    of a silent hang. Detection is isolated in _detect_primary_interface()
    (fails closed -> None); the env-var logic is fully unit-tested via
    monkeypatch. Real multi-node validation remains a 1.0-gate item.
  • scripts/provision-runner-python.ps1: automates the self-hosted GPU
    runner's system-wide Python install (the test-cuda CI job's env.PYTHON
    points at C:\Python<ver>, which the NETWORK SERVICE runner account
    must be able to read). Idempotent: re-running with the same -Version is
    a no-op; changing -Version re-provisions and reminds the operator to
    update env.PYTHON in ci.yml. Requires admin + uv; RUNNER_SETUP.md
    now points at it as the recommended path.
  • prepare(auto_bs=True) now prints a one-line note when the forward-only
    sweep is used (no loss_fn): "auto_bs running forward-only (no loss_fn);
    pass loss_fn for a larger batch size". Without a loss the sweep measures
    only activations+params, not grads + optimizer state, so the picked size
    is safe but smaller than a real fwd+bwd sweep would allow; the note
    surfaces why instead of leaving the user to find it in the docstring.
    Silent when a loss_fn is given (the sweep is then real).

Changed

  • prepare() now defaults optimize=True (was False). On a GPU this
    turns on the free-win bundle by default — TF32, cudnn.benchmark for CNNs,
    num_workers/pin_memory/persistent_workers defaults on bare loaders,
    and AMP — without touching lr/loss/schedule/optimizer. This closes the
    "drop a script in and it runs GPU-optimized" gap: a bare
    prepare(model, loader) no longer leaves the GPU wins on the table. The
    bundle is a documented no-op on CPU (every flag gates on a visible CUDA
    device), so CPU-only callers are unaffected. Pass optimize=False to opt
    out. When the bundle fires on a GPU with AMP on, prepare() now also
    prints the exact two-line autocast_context() + GradScaler() snippet to
    add to the training loop (we can't wrap an arbitrary loop from inside
    prepare; the helpers are no-ops on CPU so the snippet is safe verbatim).
    auto() and fit() inherit the new default automatically.
  • Repository-hygiene cleanup (no behavior change, no public-API change):
    • Refreshed stale version references: SECURITY.md supported-versions
      table now reflects the current release line (0.11.x, was 0.7.x);
      the bug-report issue template placeholder updated to 0.11.0.
    • Normalized GitHub URL casing to the canonical OriAlpha/Autotrainer
      across README.md, CONTRIBUTING.md, SECURITY.md, pyproject.toml,
      and issue templates (GitHub redirects case-insensitively, but the
      canonical form is now used everywhere).
    • RUNNER_SETUP.md generalized: removed maintainer/machine-specific
      details (GPU model, driver version, runner name, fixed paths) so it
      reads as a reusable guide for any contributor setting up a self-hosted
      GPU runner.
    • NEXT_STEPS.md moved out of the published repo (it was an internal
      engineering backlog); it is no longer tracked. Maintainers keep it
      locally under the gitignored docs/internal/.
  • fitting.py split: the phase-1 search and checkpoint helpers
    (_unwrap, _sync_from_rank0, _journal_storage, _parallel_search,
    _save_checkpoint, _load_checkpoint, _CHECKPOINT_FORMAT) moved to a
    new _fit_search.py, leaving fitting.py (431 -> 319 LOC) as the
    orchestrator + phase-2 training loop. Pure move, no behavior change; the
    names are re-exported from fitting.py so existing imports keep working.
    Prepares the ground for the roadmap's training-triage features by
    separating the two phases before layering more onto either.
  • Optimize-path CUDA test stub consolidated: the per-attribute
    _pretend_cuda helper (patched is_available/device_count/set_device/
    Tensor.to individually) is replaced by a shared pretend_cuda conftest
    fixture that patches the complete public torch.cuda surface
    prepare() reads (CUDA_OPTIMIZE_SURFACE), on the real module object so
    torch's internal torch.cuda reads still resolve. A new guard test
    (test_optimize.py::TestCudaSurfaceCoverage) scans the source and fails
    loudly if prepare()/_optimize/utils reach for a public torch.cuda
    attribute the fixture doesn't stub - closing the "silently exercises the
    CPU path on a CPU-only box" gap that bit PR #1. No behavior change;
    pure test-hygiene improvement.

Removed

  • Breaking: tune() no longer accepts the train_loader=/val_loader=
    keyword aliases (deprecated in 0.10). Passing them now raises TypeError
    pointing at the replacement train=/val= names. The soak period elapsed;
    these names were misleading for estimator inputs (which take arrays, not
    loaders).

Fixed

  • ThroughputMonitor MFU computation crashed on Python 3.9: it used
    zip(..., strict=True), which is PEP 618 (3.10+ runtime syntax). The
    project's requires-python and ruff target-version both target 3.9,
    so the 3.9 CI cell raised TypeError: zip() takes no keyword arguments
    in the three MFU tests. The two deques are appended in lockstep
    (step_time + tick fire once per step), so the strict check was only a
    defensive assertion; replaced with a plain zip() plus a comment
    documenting the lockstep invariant.
  • The multi-rank FSDP wrap tests (test_fsdp_wraps_with_orig_params_over_process_group,
    test_fsdp_with_cpu_offload_wraps) failed on torch >= 2.13 in CPU-gloo CI
    with "FSDP needs a non-CPU accelerator device, but no accelerator device is
    detected". On those builds FSDP() refuses even the wrap when no CUDA/XPU
    device is visible - and the harness deliberately hides CUDA (CUDA_VISIBLE_DEVICES="")
    so two ranks don't fight over one GPU. The worker now catches that specific
    refusal and emits a SKIP reason=fsdp-needs-accelerator line, so the test
    skips cleanly on builds where torch forbids the CPU wrap while still running
    the real assertion on builds where torch allows it (and where a usable GPU
    exists, the existing cuda-marked full-step test covers the rest).
  • Examples now run standalone on the spawn start method (Windows/macOS):
    every script in examples/ wraps its executable code in a main() behind
    if __name__ == "__main__":. Without it, prepare()/fit() setting
    num_workers > 0 (and sklearn's joblib/loky) spawned workers that
    re-imported the module and re-ran training, crashing with "DataLoader worker
    exited unexpectedly". autotrainer run and srun were already unaffected
    (they run the script as __main__).