You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
prepare(model, loader, opt, optimize=True): the GPU optimization layer
the original thesis promised - detect the hardware, set it up for
throughput, leave the user's hyperparameters alone. When optimize=True
and CUDA is available, prepare() applies:
cudnn.benchmark = True for CNNs (free win on fixed input shape).
TF32 on Ampere+ (cuda.matmul.allow_tf32, cudnn.allow_tf32) - a
~2-3x matmul speedup that ships disabled for legacy reproducibility
reasons.
Loader defaults on bare DataLoader(...) calls: num_workers
(capped at 8, sharded by world size), pin_memory=True, persistent_workers=True. User-set values are never overridden.
AMP implied (use autotrainer.autocast_context() + GradScaler() as
before); pass amp=False to opt out.
Every decision is printed; nothing in this path touches lr, loss,
schedule, or optimizer choice. No-op on CPU and when optimize=False,
so existing callers see no change.
prepare(..., auto_bs=True, loss_fn=...): grow the loader's batch size
until OOM then back off one step. Uses the user's loss_fn for an
accurate forward+backward measurement; without it the sweep is
forward-only (conservative). The discovered size rebuilds the loader;
lr and schedule are NOT changed - pair with accumulate() to scale the
step to the new effective batch.
prepare(..., max_bs=N): ceiling for the auto_bs sweep (default 4096).
prepare(..., compile=True, compile_mode=): wrap the model with torch.compile() before any DDP wrap. Order matters - compiling the
unwrapped module then DDP-wrapping is the documented-supported path;
the reverse causes graph breaks on the .module indirection. No-op on
CPU and torch < 2.0. On compile failure (dynamic shapes the backend
can't handle, missing Triton on Windows, etc.) falls back to the
uncompiled model with a warning rather than crashing the run. compile_mode: default | reduce-overhead (CUDA graphs) | max-autotune (kernel search). Does not touch lr/loss/schedule.
prepare(..., fsdp=True): wrap with FullyShardedDataParallel instead
of DDP when distributed. FSDP shards params/grads/optimizer state
across ranks - the path for models too large to fit on one GPU (DDP
replicates and OOMs). Uses use_orig_params=True so the user's
optimizer keeps working unchanged. On single-process (world_size==1)
or torch < 2.0, fsdp=True is a no-op with a warning. Does not touch
lr/loss/schedule/optimizer.
prepare(..., cpu_offload=True): when paired with fsdp=True, enables CPUOffload(offload_params=True) - moves FSDP-sharded params to CPU and
brings them to GPU only for the forward/backward. Trades throughput for
the ability to train models that OOM even when sharded across ranks.
Ignored with a warning on the DDP path (no built-in CPU param offload)
and on single-process (world_size == 1). Does not touch lr/loss/schedule.
Training-loop helpers (autotrainer.zero_grad, eval_mode, train_mode, accumulate): the small things users forget inside the loop. zero_grad uses set_to_none=True; eval_mode/train_mode are
context managers that restore the prior mode (kills the classic
"forgot to flip back to train() after eval" bug); accumulate(opt, steps=N)
handles gradient accumulation with optional GradScaler integration.
None touch lr / loss / schedule / optimizer choice.
autotrainer.BottleneckMonitor: the cheapest piece of the roadmap's
training-triage theme. Sample per-step data-load vs compute time and
print a plain-language warning when the dataloader dominates the GPU
("raise num_workers / pin_memory / prefetch"). Opt-in; zero overhead
when not constructed.
autotrainer.node_scratch() and autotrainer.configure_scratch(): SLURM
node-local scratch ergonomics. node_scratch() returns $TMPDIR
(per-job, per-node, auto-cleaned under SLURM; system temp elsewhere),
suffixed with the SLURM job id so concurrent jobs don't collide. configure_scratch() sets TORCHINDUCTOR_CACHE_DIR to it (so torch.compile kernel cache doesn't hit NFS) and warns when the scratch
looks like it's on a network filesystem (NFS/Lustre/GPFS/Panasas). Call
once at the top of your training script, before any torch.compile.
CUDA CI: new cuda pytest marker selects GPU-required tests; CPU jobs
run -m "not cuda" to skip them cleanly. A test-cuda CI job runs the cuda-marked subset on a self-hosted GPU runner (runs-on: [self-hosted, gpu]), catching the class of CUDA-path bug that CPU-only CI cannot
(the device_count() crash, the _pretend_cuda stub defect, and the /nfs PermissionError were all found by running on a real GPU). See RUNNER_SETUP.md for one-time runner registration, and NEXT_STEPS.md
for the engineering backlog that came out of this work.
autotrainer.cuda_device(): canonical helper for picking the CUDA
device-or-CPU based on device_count() > 0. Centralizes the check
previously duplicated (inconsistently) across prepare, _ensure_process_group, find_lr, _find_lr_synced, and tune.
Fixed
Public autotrainer.prepare() now forwards keyword arguments to the torch
backend. Previously the dispatcher in __init__.py called the backend
with only 3 positional args, so every prepare(optimize=True), prepare(compile=True), prepare(fsdp=True), prepare(auto_bs=True),
etc. raised TypeError: prepare() got an unexpected keyword argument 'optimize' - the entire Tier 1-3 feature surface was unreachable from the
public API. The CI tests passed only because they imported the backend
directly. Added a regression test class (TestPublicDispatcherForwardsKwargs)
that imports the public package and exercises the dispatcher path users
actually take.
CUDA device selection now gates on torch.cuda.device_count() > 0, not
just is_available(), centralized in the new autotrainer.cuda_device()
helper. The previous check was True whenever the driver was present,
even when CUDA_VISIBLE_DEVICES="" hid every GPU - so set_device(local_rank)
crashed with "invalid device ordinal" on driver-present, GPU-hidden
boxes (e.g. the CPU-gloo distributed tests on a 1-GPU dev machine). All
four device-pick sites (prepare, _ensure_process_group, find_lr, _find_lr_synced, tune) now share the one helper.