autotrainer 0.18.0

[0.18.0] - 2026-10-22

Added

• Support for dictionary and tuple batches in autotrainer.train(): PyTorch training now natively parses complex batch structures (e.g. HuggingFace datasets yielding {"input_ids": ..., "labels": ...} or multi-input tuples (x1, x2)) via split_xy() and forwards them cleanly to model forward methods via robust_forward().
• DDP model.no_sync() optimization in autotrainer.accumulate(): Intermediate micro-batches during gradient accumulation now bypass redundant cross-GPU gradient all-reductions using model.no_sync(). An explicit sync: bool | None = None parameter on _Accumulator.backward() allows callers to force synchronization on tail micro-batches when dataset lengths are not multiples of accumulation steps.
• CI example validation and linting: Added examples/ to GitHub Actions workflow for ruff check and ruff format --check, keeping all runnable example scripts strictly linted.

Fixed

• Keras EarlyStopping without validation data: When training without validation data, autotrainer.train() now sets EarlyStopping(monitor="loss") instead of the default val_loss, eliminating noisy runtime warnings and ensuring early stopping triggers reliably.
• Scikit-learn 1.8+ LogisticRegression deprecation warning: Setting n_jobs on LogisticRegression (and any custom subclasses evaluated via MRO) is now skipped in sklearn_backend.prepare(), preventing deprecation FutureWarning while preserving multithreading configuration on all other estimators (e.g., RandomForestClassifier).
• Robust batch unpacking in train(): Replaced hardcoded (xb, yb) unpacking with split_xy(batch) across dataloader iterations, preventing ValueError or key-iteration string errors on non-2-tuple dataloaders.

autotrainer 0.17.0

[0.17.0] - 2026-10-07

Added

• Test suite expansion & quality gates: Overall test coverage expanded to 86.5%
  (coverage floor raised to 80% in pyproject.toml), adding exhaustive unit coverage
  across cli, __init__, detect, auto_optim, tf_backend, and ui endpoints.

Fixed

• A .json save_path on an sklearn-API booster wrote a pickle. The
  estimator branch of train() always called joblib.dump, so
  train(XGBClassifier(...), X, y, save_path="model.json") - what
  examples/xgboost_example.py does, and what the example docstrings
  advertise - produced a joblib pickle named .json that xgboost could not
  load back. A .json/.ubj path on an estimator with its own save_model
  now uses that native writer; everything else (LightGBM's wrapper included,
  which has no save_model) still pickles as before.
• train(optimizer=...) and auto(optimizer=...) discarded a passed
  optimizer. Only a name ("adamw" / "sgd") was ever honored: an
  optimizer instance went to _choose_optimizer as if it were a name, came
  back verbatim, and _build_optimizer then compared it to "sgd" and
  constructed a fresh AdamW - so a passed Adam(lr=7e-3) silently trained as
  AdamW at whatever the LR range test returned. Instances are now used as-is,
  the same passthrough _make_loss already did for a loss instance, and the
  lr on the passed optimizer is respected unless lr= is also given (in which
  case it is applied to that optimizer rather than one of the two being
  dropped). examples/pytorch_ddp.py advertised this and never got it.
• DataLoader workers ignored the SLURM allocation. The worker count was
  sized from the node's physical core count (psutil.cpu_count), which does
  not respect cgroups or cpusets. A job granted 2 CPUs on a 128-core node got
  the cap of 8 workers per rank - four times its whole allocation - and they
  thrashed against the cgroup. Worker sizing now uses SLURM_CPUS_PER_TASK
  and falls back to os.sched_getaffinity, matching what the sklearn and
  boosting backends already did for n_jobs. All three now share
  utils.available_cpus(), so they can no longer disagree about the
  allocation.

Added

• channels_last for conv nets under prepare(optimize=True), gated on
  AMP: measured +28% per step on a small conv net (RTX 5070), rising to +34%
  when train_step() also converts the input. Gated deliberately - the same
  conversion measured 3x slower in fp32/TF32, where cuDNN's fast paths are
  NCHW. Applied only when every convolution is a Conv2d, since the layout is
  4D. Reported in the optimize: line like every other change.
• Fused optimizers in auto() on CUDA: +11-12% per step on ordinary nets,
  and over 2x when the optimizer step dominates. Only for the optimizer
  auto() builds; one you pass in is never modified.
• autotrainer doctor now checks the two things that silently halve
  throughput on a cluster: the per-rank CPU budget (with the num_workers it
  implies, and a warning when one CPU has to feed a GPU) and whether scratch
  looks networked. It also warns when SLURM_CPUS_PER_TASK is unset inside a
  job.

Changed

• The Web UI's markup moved out of ui.py into
  src/autotrainer/templates/. ui.py was 4,629 lines, of which 3,707 were
  two HTML documents held as string literals - the export report as an
  f-string, so every one of its 122 CSS brace pairs was doubled and none of it
  could be linted or highlighted as HTML. The templates are now .html files
  filled by _render() at request time, and ui.py is 954 lines of server
  code. Rendered output is byte-identical; both files ship in the wheel and
  the sdist.
• auto() builds the optimizer after prepare() has placed the model, so the
  params are on the GPU in time for the fused kernels. _make_optimizer split
  into _choose_optimizer (the decision, on the raw model) and
  _build_optimizer (the construction, after placement). Both are internal;
  the inferred recipe is unchanged.

Removed

• Hyperparameter search (fit(), tune()) is gone. Autotrainer is the
  execution layer: it makes your recipe run correctly and fast on the
  hardware you have. Searching for a better recipe is a different job, and
  Optuna and Ray Tune already do it well - use either one and call
  prepare() inside the objective. What this removes:
  • autotrainer.fit() and autotrainer.tune(), and the autotrainer.tuning,
    autotrainer.tuning_estimator, autotrainer.metrics, autotrainer.fitting
    and autotrainer._fit_search modules.
  • The metric= selection knob (accuracy/f1/auc/r2), the Optuna
    journal-file parallel search, and the phase-1/phase-2 checkpoint format.
  • autotrainer.augment_batch() and the autotrainer.augment module; it
    existed to give the search a scalar regularizer to tune.
  • sanity.overlap() - the train/val leak check. Nothing takes both a train
    and a validation loader any more, so it was unreachable.
  • The tune extra (pip install "autotrainer[tune]"). Optuna is no longer
    a dependency of any extra, including all.
• docs/guide/fit.md, examples/pytorch_fit.py, examples/pytorch_tune.py.

Changed

• autotrainer.train() moved from autotrainer.fitting to
  autotrainer.training (both are internal modules; autotrainer.train
  is unchanged).
• New One-line training guide covering
  train(), auto(), and the data checks, replacing docs/guide/fit.md.
  Preemption (autotrainer.preempt) now documented in the scaling guide.

Unaffected: prepare(), auto(), train(), train_step(), the launcher,
SLURM support, the monitors, the trackers, and the UI.

autotrainer 0.16.0

[0.16.0] - 2026-08-20

Added

• Hardware Telemetry Curves & Resource Diagnostics (#hwChart):
  • Live system resource meter footer with green pulse status indicator in sidebar.
  • Dedicated hardware telemetry card displaying GPU VRAM (GB / %), GPU model name, and Host RAM / CPU utilization curves over training steps and epochs.
  • REST endpoint GET /api/hardware with cross-platform memory and GPU detection.
• Run Tagging, Star Favoriting & Sidebar Organization:
  • Star favorite toggle (⭐) on individual runs with persistent metadata state.
  • Sidebar filter tabs: [ All ] and [ ⭐ Starred ] with glow accents.
  • Inline run tag chip management: + Tag creator, ✕ tag pill deletion, and dynamic sidebar tag filter pills bar.
  • REST endpoints: POST /api/runs/<run_id>/tags and POST /api/runs/<run_id>/favorite.
• Hyperparameter & Metric Parallel Coordinates Chart (#parallelCanvas):
  • High-performance Canvas multi-axis parallel coordinates visualization in Compare Mode.
  • Visualizes complex relationships and trajectories across Learning Rate -> Batch Size -> First Loss -> Training Loss -> Validation Loss across multiple selected runs.
• Run Annotations & Markdown Notes Drawer (#notesCard):
  • Embedded Markdown drawer with Edit and live Preview rendering tabs.
  • Auto-persisted to both run.json and <run_dir>/notes.md on disk.
  • REST endpoint POST /api/runs/<run_id>/notes.
• 1-Click Run Deletion & Archive:
  • Clean deletion of experiment runs and log directories from disk with modal confirmation.
  • REST endpoints DELETE /api/runs/<run_id> and POST /api/runs/<run_id>/archive.
• Refined KPI Stat Cards:
  • Displays FIRST LOSS, TRAINING LOSS, VALIDATION LOSS, and EPOCHS with glowing accents.

Security

The Web UI added in 0.15.0 served a filesystem-mutating API with no
authentication on every network interface. Each item below was confirmed by
exploiting a running server, and is now pinned by a test in
tests/test_ui_security.py.

• Unauthenticated remote directory deletion. run_id was joined onto the
  logs root with no containment check, so DELETE /api/runs/../<path>
  resolved outside it and reached shutil.rmtree. A live server answered
  {"success": true} and the directory was gone. All six lookups — plus a
  seventh inlined in _get_run_data and two more behind the HTML/Markdown
  export writes, which made it an arbitrary-write primitive too — now go
  through one _safe_run_dir() that rejects separators and anything
  resolving outside a configured root.
• autotrainer ui bound 0.0.0.0. On a shared SLURM login node that
  published rename/archive/delete to every user who could route to the port.
  It now binds 127.0.0.1; --host widens it deliberately and prints a
  warning naming what is exposed.
• No authentication. A session token is now generated per run and printed
  as part of the URL, carried afterwards by an HttpOnly; SameSite=Strict
  cookie (also accepted as ?token= or X-Autotrainer-Token) and compared
  with secrets.compare_digest. --no-token opts out. SameSite plus an
  Origin check on mutating verbs closes the CSRF hole that let any page a
  user visited drive the delete endpoint.
• POST /api/sources accepted any absolute path, turning the dashboard
  into a filesystem browser — a home directory listed 45 "runs". It now
  requires a directory that actually contains run folders.
• Stored XSS via run tags. Tags were interpolated raw into innerHTML,
  including inside onclick="removeTagFromCurrentRun('${tag}')". Tags and
  usernames are escaped, escapeHtml now covers quotes, and the tag remove
  control binds a listener instead of building a JS string.

Fixed

• AutotrainerCallback.on_epoch_end() crashed on its own defaults.
  Omitting train_loss — normal for a callback that only has a validation
  number — reached float(None) and raised TypeError, killing the run it
  was meant to be observing. log_epoch now treats every metric as optional
  and reads its three call shapes through one code path.
• optimize=True silently disabled per-epoch reshuffling. prepare()
  wraps a sharded loader in _AutoEpochDataLoader so
  DistributedSampler.set_epoch() advances on its own, but the optimize
  loader-defaults step and the auto_bs sweep each build a fresh
  DataLoader, which the wrapper did not survive. set_epoch was then never
  called and every epoch replayed one fixed permutation on every rank —
  silently, with nothing raised. It fired on the most common configuration
  there is: multi-GPU, optimize=True (the default), and an untuned loader,
  since build_loader_defaults() only has work to do when num_workers is 0
  or pin_memory is off. The wrapper is now applied once, after every
  rebuild. Loaders the user sharded themselves are still passed through
  unwrapped, and shuffle=False still means a fixed order.
• train() synced the GPU twice per step. loss.item() was called once
  for the running total and again for the summary; each is a device sync, so
  the loop stalled twice per iteration in the entry point whose selling point
  is throughput. Read once, reused.

Changed

• Docs now describe the auto-managed epoch shuffling that 0.14.0 added.
  docs/guide/gpu-optimization.md still showed the pre-0.14.0 log line
  (- call autotrainer.set_epoch(loader, epoch) ...) where the code prints
  - epoch shuffling auto-managed, and the guides still presented the manual
  call as required. They now say it is optional for loaders prepare()
  sharded, and — worth knowing — that the automatic counter overwrites an
  explicit epoch number, so resuming at a specific epoch needs the loader you
  sharded yourself. set_epoch's own docstring says the same.
• Removed the suhas and suhas-goravale-siddaramu optional-dependency
  groups from pyproject.toml. Both were ["psutil"] and shipped as real
  installable extras (pip install autotrainer[suhas]) in the published
  metadata. The autotrainer suhas CLI credits command is unrelated and
  stays.
• ThroughputMonitor is now in __all__. It was documented in
  docs/guide/monitors.md with a worked example while absent from __all__,
  which the README declares the boundary of the stable public API — so the
  docs taught something the policy called internal. Also added to the README's
  entry-point table. tests/test_docs.py now checks this direction too; it
  only ever verified that __all__ members appear in the docs, never that
  documented names are exported.

autotrainer 0.14.1

[0.14.1] - 2026-08-05

Fixed

• prepare() auto-launch no longer kills notebook kernels. Auto-launch
  re-executes sys.argv once per GPU and exits the parent, but only checked
  RANK, SLURM_JOB_ID and detect(). In a notebook on a multi-GPU box
  argv[0] is the kernel launcher, so it spawned N launchers and killed the
  kernel mid-cell with nothing to diagnose; python -m pkg was re-executed as
  a bare file path, losing the package context. It now requires a plain
  python script.py invocation and explains itself when it stands down.
• The summary reports what autotrainer did, not what it guessed. The
  "Active Optimizations" list was derived from global torch flags and env
  vars, which cannot distinguish a setting autotrainer made from one the user
  made. It claimed a multi-worker pinned-memory DataLoader pipeline for
  loaders left at num_workers=0; a "Weight Decay Exclude" norm/bias
  param-group split that autotrainer has never performed, for any nonzero
  weight_decay; "Grad Clipping" from optimizer.defaults["max_norm"], which
  no torch optimizer defines; "SLURM Node Scratch" from SLURM_JOB_ID rather
  than from configure_scratch() running; and FlashAttention/bf16, which are
  a torch default and a hardware capability. prepare(), configure_nccl()
  and configure_scratch() now record what they applied, and the report
  renders Autotrainer Applied from that record alone. Facts observed but
  not caused move to a new Environment Detected section. weight_decay
  now appears on the optimizer recipe line, reported rather than claimed.
• atexit no longer destroys the process group. Teardown ran during
  interpreter shutdown, which is unreliable and duplicates torch's own
  cleanup; the exit hook now only prints. Explicit
  finish(cleanup_dist=True) is unchanged.
• A second run in one process gets its own summary. finish() now
  releases the global tracker, instead of leaving a reported tracker that
  silently suppressed the next run's summary. fit() still emits one summary
  for the whole run rather than reporting the search and dropping the final
  training.
• train() reports its LR schedule. The scheduler inferred by auto()
  was never handed to the summary, so the LR Schedule line was blank on the
  one path that always has one.
• train() dispatches by framework. Routing probed for .fit / .forward
  attributes, so any object with a fit() method was sent to the sklearn
  backend and everything else fell through to the PyTorch path, failing deep
  inside it. It now uses the same module-prefix rule as prepare() and
  tune(), raises a clear TypeError for models it cannot route, and sends
  boosting estimators to the boosting backend's thread config.

autotrainer 0.14.0

[0.14.0] - 2026-07-31

Added

• Unified autotrainer.train() 1-Line API. Single-function execution across PyTorch, Scikit-Learn, XGBoost, LightGBM, CatBoost, and TensorFlow/Keras. Automatically infers loss/optimizer/scheduler, configures thread/replica batch scaling, executes training, serializes model checkpoints, and reports performance summary.

• Model Checkpoint save_path= Parameter. Added save_path parameter to train(), fit(), and tune(), enabling direct model serialization (.pt, .joblib, .keras, .json) without requiring separate save0() or joblib.dump() code lines.

• Automatic Summary & Process Group Cleanup. train(), fit(), and tune() now automatically trigger finish(checkpoint=save_path) on exit, eliminating separate autotrainer.finish() calls.

• autotrainer.finish(checkpoint=...) & SummaryTracker API across all backends. One-line post-training summary reporter and cleanup helper across all supported frameworks (torch, sklearn, xgboost/lightgbm, and tensorflow/keras). Displays formatted summary box with cluster hardware, topology, VRAM usage %, training throughput (samples/sec), loss metrics, active optimizations, and health triage, then automatically destroys process groups on exit.

• Model Parameter Count & Executive Summary Formatting. SummaryTracker now calculates total trainable model parameters (Model Parameters), formats learning rates cleanly (lr=4.21e-05), detects active LR schedulers (LR Schedule), and renders Autotrainer Active Optimizations cleanly after Artifacts & Checkpoint.

• Top-level autotrainer.log_epoch() and autotrainer.step() functions. Exported in __all__ for recording epoch metrics and step losses directly into the active summary.

• Auto-managed DDP Epoch Shuffling (_AutoEpochDataLoader). autotrainer.prepare() now auto-wraps DataLoaders to advance DistributedSampler.set_epoch(epoch) automatically on every epoch loop iteration, eliminating the need for explicit autotrainer.set_epoch() calls.

• Dynamic Plain-English Optimizations Catalog. Summary engine dynamically detects active optimizations across 25 supported methods (TF32, AMP, FlashAttention SDPA, cuDNN benchmark, DDP, FSDP, CPU parameter offloading, CUDA allocator tuning, DataLoader workers, SLURM scratch, etc.) and prints human-readable explanations.

• Framework Diagnostics in autotrainer doctor. Enhanced autotrainer doctor to print exact version strings for installed ML libraries (torch v2.12.1+cpu, sklearn v1.9.0, lightgbm v4.6.0, etc.).

• CI Example Test Job (test-examples). Added automated workflow job running all 8 example scripts (pytorch_auto, pytorch_fit, pytorch_tune, pytorch_ddp, pytorch_optimize, sklearn_example, xgboost_example, tensorflow_example) on PRs and releases.

• Internal Architecture Catalog & Market Positioning. Documented all 25 optimization methods in docs/internal/architecture.md and positioning vs PyTorch Lightning/Accelerate in docs/internal/market-positioning.md.

• Unit Test Coverage. Added tests/test_summary.py covering SummaryTracker, finish(), and dynamic optimization text detection. Updated pretend_cuda fixture in tests/conftest.py with max_memory_allocated.

• Prerelease channel. A version that is not exactly X.Y.Z (an rc, a,
  b, .dev, or .post suffix) is now published to TestPyPI and marked as a
  prerelease on GitHub, instead of going to production PyPI. This exists so the
  SLURM validation runbook can pip install and exercise a real wheel before
  the version number is spent - PyPI versions are immutable, so validating
  against a production release means any defect found costs a new version. The
  routing tests for == 'false' rather than == 'true', so only an explicit
  "this is stable" reaches PyPI; anything unexpected goes to TestPyPI.

• Release notes for a prerelease fall back to the [Unreleased] changelog
  section, since an RC has no section of its own. An empty or missing section
  is still a hard failure.

• cuda-watchdog CI job. When the self-hosted GPU runner is offline,
  test-cuda sits queued indefinitely and GPU coverage silently drops to zero
  while the run still looks green - timeout-minutes does not help, because it
  only starts counting once a runner accepts the job. The watchdog notices and
  annotates the run.

• tests/test_docs.py: checks that relative doc links and heading anchors
  resolve, that everything in __all__ is mentioned in the docs, and that no
  message promises a version that has not shipped. Every one of these
  corresponds to a real defect found by hand in this release.

• .github/scripts/ is now linted and format-checked in CI; it writes the
  release notes and was previously unchecked.

Fixed

• The TypeError raised for the removed tune(train_loader=...)/val_loader=
  aliases said they "were removed in 1.0". 1.0 has not shipped - they were
  removed in 0.12. The message pointed users at a release that does not exist.
• Excluded yanked PyPI package build==1.5.1 from pyproject.toml dev dependencies (build!=1.5.1) and updated uv.lock.

Changed

• Automatically register get_active_summary() when backend entry points (prepare() and scope()) are invoked.
• Documentation restructured. The README was 551 lines with 59% of it under a
  single ## Use heading nested four levels deep; it is now a ~220-line
  entry point (pitch, install, quickstart, an entry-point table, and how
  autotrainer differs from Lightning / Accelerate / Ray / Optuna), with the
  reference material moved to docs/guide/. Also fixes two sections titled
  "auto batch size" 244 lines apart covering different features, and a
  paragraph that continued the quickstart 220 lines after it was interrupted.
• docs/README.md claimed to list everything in autotrainer.__all__ but
  omitted TrainingMonitor (added 0.12) and augment_batch (added 0.13).

autotrainer 0.13.0

[0.13.0] - 2026-07-28

Added

• metric= on tune() and fit(): select on the number you actually care
  about. The torch path could only ever optimize validation loss - it
  picked the recipe, pruned trials, early-stopped, and chose the best epoch on
  it - while the sklearn-estimator path has scored on accuracy/R² through
  scoring= since 0.10. Loss is a proxy, and it comes apart from the goal
  exactly where the search space is widest: a regularized recipe (label
  smoothing, augmentation) is less confident, so val cross-entropy can bottom
  out and start climbing from overconfidence while val accuracy is still
  improving - and a loss-driven patience stops there and restores a
  less-accurate checkpoint. Pass metric="accuracy", "f1", "auc", "r2",
  or a callable(model, loader) -> float (with direction= if lower is
  better). Training still always uses the loss; this changes only what runs are
  scored by. Default is "loss", so existing behavior is unchanged.
• autotrainer.metrics. Pure-torch implementations of the built-ins, so
  scoring on a metric doesn't make scikit-learn a hard dependency. F1 is
  macro-averaged over the classes present in the targets (the one to reach for
  on imbalanced data, where accuracy flatters a majority-class predictor); AUC
  is rank-based with average ranks for ties, which a naive implementation gets
  wrong whenever a saturated model emits identical logits, and its multiclass
  one-vs-rest form ranks on softmax probabilities rather than raw logits (they
  differ: softmax's per-sample denominator makes it non-monotone in a single
  logit column). All four agree with scikit-learn to 1e-9 across 1500
  randomized cross-checks.
• Preemption handling in fit(checkpoint=...). The resumable checkpoint
  has existed since 0.11, but nothing caught the scheduler's warning shot, so a
  job preempted mid-epoch still threw that epoch away - and a job preempted
  during phase 1 threw away the entire search. fit() now watches
  SIGUSR1/SIGTERM (SLURM's --signal=B:USR1@120) and stops at the next epoch
  boundary after the checkpoint is written, so the requeued job resumes at the
  following epoch. Phase 1 is journaled to <checkpoint>.study, so an
  interrupted search resumes with its completed trials instead of restarting.
  Both are armed only when checkpoint= is set.
• Data sanity checks in auto() and tune(). The failures that aren't the
  framework's fault and don't announce themselves: un-normalized or non-finite
  inputs (which look like a bad learning rate), class imbalance (which looks
  like a model that "works" at 95%), constant inputs or targets, and - the
  expensive one - a train/val split that overlaps, which doesn't fail at all,
  it just reports an excellent validation score that every trial then optimizes
  against. The batches were already being peeked to infer the loss, so the
  checks are nearly free, and they run before the LR range test and the first
  trial rather than after an allocation is spent. Overlap is exact when both
  loaders are Subsets of one dataset, and otherwise compares a bounded sample
  of rows byte-for-byte (skipped below 4 features, where identical rows arise
  by chance). Warnings only - nothing is changed for you. sanity=False turns
  them off; tune() also gates them on verbose.
• autotrainer.sanity. The checks themselves (report() / overlap()),
  returning message strings rather than printing, so they can be called
  directly against your own loaders.
• autotrainer.preempt. The watcher behind it (watch() / preempted()),
  usable in hand-written loops. The handler only sets a flag - it never raises
  or exits - so your loop chooses a stopping point where its state is
  consistent.
• epochs is now a searched hyperparameter. Training length is one of the
  largest levers on final quality, and every trial previously ran on one fixed
  horizon. Each trial now trains for its own searched budget and anneals its
  LR schedule over exactly that budget, so a recipe that wants a short, fast
  anneal and one that wants a long, gentle one are each evaluated the way they
  would actually run. The range is 1..epochs_per_trial, so adding the knob
  widens what is searched without making any trial cost more than it did
  before; raise epochs_per_trial to widen it.
• aug_strength: searchable image augmentation for CNNs. A single scalar
  dials a fixed flip + cutout policy from off to standard, and it enters the
  default space only when conv layers are detected (the policy is
  image-specific, so searching it elsewhere would burn trials on a dimension
  that changes nothing). 0.0 is in range, so "no augmentation" stays a
  candidate. The winning strength carries into fit()'s phase-2 retrain -
  dropping it there would retrain a different, unregularized recipe.
  Augmentation is applied to inputs on the training path only; validation and
  test passes always see clean data.
• autotrainer.augment_batch(x, strength). The primitive behind
  aug_strength, exported for use in hand-written loops. Pure torch (no new
  torchvision dependency), target-preserving, never mutates its input, and a
  no-op on anything that isn't a float NCHW batch - so it can sit in a loop
  that also trains tabular or text models.

Changed

• ASHA now prunes on normalized rungs instead of raw epoch indices.
  Required by the epochs knob: with per-trial budgets, reporting at the raw
  epoch index would pit a long-budget trial (still mid-anneal at epoch 1)
  against a short-budget one that had already finished annealing, pruning the
  long trial for converging later rather than worse. Trials now report at a
  fixed number of rungs measured as fraction-of-own-schedule-completed, which
  compares like with like. Custom pruner= overrides are unaffected.
• fit()'s held-out test number is stored under test_score. It is no
  longer necessarily a loss. test_loss is still set as well when the metric
  is the loss, so existing readers keep working.
• Checkpoint format version 2 (adds metric/direction). fit() refuses
  to resume a checkpoint written under a different metric rather than compare
  an accuracy against a stored loss and restore the wrong epoch. Version-1
  checkpoints are rejected with the usual "delete to start fresh" message.

autotrainer 0.12.0

[0.12.0] - 2026-07-27

Added

• autotrainer.train_step(model, loss_fn, xb, yb, opt, scaler=...). The
  all-in-one companion to prepare(optimize=True): one call per step does the
  full AMP dance in the correct order - move the batch to the model's device,
  forward under autocast (bf16 on modern GPUs, fp16 otherwise, no-op on CPU),
  compute the loss, backward + optimizer.step() + scaler.update() (a plain
  backward()/step() when no scaler), and zero the grads. Returns the detached
  loss for logging. Closes the last "not fully automatic" gap: prepare()
  can't wrap your loop, so AMP used to be a hand-written, order-sensitive
  snippet users had to copy; train_step() runs it for you. Touches nothing
  about your recipe (lr/loss/schedule/optimizer). scaler is optional (leave
  it None on CPU / bf16 GPUs); autocast=False opts the forward out of mixed
  precision while keeping the backward/step/zero bookkeeping. Backward runs
  outside autocast (the documented-correct scoping). The AMP path now also has
  end-to-end tests (CPU: loss actually decreases through the helper; cuda:
  the same on-device, plus a forced-enabled fp16 scaler exercising the real
  scaling arithmetic) - previously only the pieces were unit-tested.
• autotrainer.TrainingMonitor: training triage - explains numerical
  failures in plain language. The health companion to the perf monitors:
  where BottleneckMonitor/ThroughputMonitor answer "is it fast?", this
  answers "is it healthy?". Call monitor.step(loss, model=..., optimizer=..., scaler=...) once per step (after backward()) and it flags, once each,
  with the fix: non-finite loss (lr too high / bad inputs, with a concrete
  lr/10 suggestion when the optimizer is passed), loss divergence, fp16
  gradient overflow (persistent AMP-scaler backoff -> "switch to bf16 via
  autocast_context()"), and non-finite / spiking / vanishing gradient norms.
  Gradient checks are skipped while an AMP scaler is active (grads are still
  scaled there, so the scaler-backoff signal owns fp16 health). report()
  prints a one-line all-clear or a recap. Everything but loss is optional;
  it only observes - never touches lr/loss/schedule/optimizer - and is
  opt-in with zero overhead when unused. Fully unit-tested on CPU (synthetic
  losses, a fake scaler, hand-set gradients).
• tune()/fit() search a much wider training recipe. The default space
  now covers the LR schedule (cosine/onecycle/constant) and its warmup
  fraction, gradient clipping (max_norm), and - for classification -
  label smoothing, on top of the original lr / weight-decay / optimizer /
  batch-size. The default is now task-aware: it's built from the model and
  the inferred loss (CNNs lean on the SGD + 1cycle recipe with a higher LR
  band; everything else defaults to a lower band with cosine/constant), so
  trials are spent where they pay off. DEFAULT_SPACE stays as the maximal
  superset of every searchable knob. A searched scheduler/grad_clip/
  label_smoothing trains identically in the short trials and the phase-2
  retrain via a shared _make_scheduler helper. Trials are always scored
  with the plain unsmoothed loss so label-smoothing candidates stay
  comparable. Does NOT search the architecture (no NAS) or the loss family
  (still inferred).
• ASHA (successive-halving) is the default tuning pruner, replacing the
  median pruner. Most trials get a small budget and only survivors are
  promoted - the multi-fidelity strategy that keeps the now-wider search
  affordable. min_resource=1, so a trial is never pruned before its second
  epoch (single-epoch trials behave exactly as before). Override with
  tune(..., pruner=...).
• lr is coupled to batch_size when both are searched: the applied LR
  is scaled toward the trial's batch (linear for SGD, square-root for
  Adam-family, relative to a batch of 32) so the search doesn't burn trials
  rediscovering that relationship. best_params still records the un-scaled
  searched LR. Toggle with tune(..., lr_scaling="none") /
  fit(..., lr_scaling="none"); applied consistently across the search and
  the phase-2 retrain.
• fit(..., test_loader=...): an honest held-out generalization estimate.
  The final (best-epoch) model is scored on a loader the search never saw;
  the number is printed and stored on rank 0's
  study.user_attrs["test_loss"]. Guards against reading too much into a val
  loss that a wide search has implicitly optimized against. The
  (model, best_params, study) return shape is unchanged.
• prepare() auto-launches multi-GPU workers. On a box with ≥2 visible
  GPUs, a bare python train.py now distributes across all of them with no
  launcher: the first prepare() call detects it's a fresh parent process
  (no RANK/WORLD_SIZE env set), spawns one worker per GPU via the same
  spawn+supervisor machinery the autotrainer run CLI uses, and the parent
  sys.exits without entering the training loop. Each worker re-enters the
  script with RANK set and proceeds through prepare() normally (DDP wrap,
  distributed sampler). Closes the second vision gap: "drop a script in and
  it auto-detects single AND multi-GPU." Spawn only fires when all three hold
  (no RANK set, not under SLURM, detect() reports local_multi_gpu), so
  it never double-spawns under srun or loops on already-launched workers.
  Opt out with prepare(..., auto_launch=False). Each spawned child is
  pinned to its own GPU via per-child CUDA_VISIBLE_DEVICES (the torchrun
  pattern) so workers don't race on set_device.
• prepare(..., static_graph=True): when distributed (DDP path), enables
  DDP's static_graph=True plus gradient_as_bucket_view=True - free
  wins when the computation graph is the same every iteration (static graph
  skips per-iteration graph-recording overhead after the first step;
  bucketing lowers peak memory). Both are opt-in because they have
  correctness implications when the graph genuinely changes (conditional
  execution, varying depth). No-op on single-device and the FSDP path. Does
  NOT touch lr / loss / schedule / optimizer.
• prepare(..., find_unused_parameters=True): forwards DDP's
  find_unused_parameters=True for models that don't touch every param each
  step (e.g. conditional branches). Mutually exclusive with static_graph
  (torch forbids the combination; prepare raises a clear ValueError).
• autotrainer.ThroughputMonitor: throughput (samples/sec), peak GPU memory,
  and a rough model-FLOPS-utilization (MFU) estimate. The companion to
  BottleneckMonitor - where that answers "am I waiting on the loader?"
  (a ratio), this answers "how fast am I going, and is that fast relative to
  the GPU's peak?" (absolute numbers). MFU is opt-in (pass model_flops);
  the GPU-peak denominator comes from a small spec table (A100/H100/common
  consumer parts) or is reported as None for unknown hardware rather than
  invented. Bookkeeping is unit-tested on CPU; the real-GPU memory read is
  gated on the cuda marker. MFU is a diagnostic, not a benchmark - it
  assumes a matmul-heavy model.
• Multi-rank FSDP test coverage: the FSDP wrap + use_orig_params param-
  addressability path now runs against a real process group in
  test_distributed.py (CPU-gloo), closing the gap that the single-process
  no-op test left open. The full sharded fwd+bwd+step is gated on >= 2
  usable GPUs (torch 2.13 FSDP won't run a forward with CPU params when
  cuda.is_available() is True), and is left to a multi-GPU runner.
• timeout-minutes: 30 on the test-cuda CI job, so an in-execution hang
  (driver crash, stuck NCCL collective, deadlocked test) fails loudly after
  30 minutes instead of running silently. Note this does not cover the
  "queued forever because the runner is offline" case - the timeout only
  starts once a runner picks the job up.
• .github/dependabot.yml: weekly dependency-update checks for the Python
  (pip) ecosystem and the GitHub Actions versions used in CI.
• autotrainer.configure_nccl(): sets NCCL_SOCKET_IFNAME to the
  default-route interface when unset, so multi-node SLURM jobs don't hang
  or crawl because NCCL guessed the wrong (or loopback) network interface.
  Non-clobbering (setdefault semantics, like configure_scratch); an
  optional debug=True turns on NCCL_DEBUG=INFO and prints a hint when
  detection fails (no ip binary, Windows host, stripped container) instead
  of a silent hang. Detection is isolated in _detect_primary_interface()
  (fails closed -> None); the env-var logic is fully unit-tested via
  monkeypatch. Real multi-node validation remains a 1.0-gate item.
• scripts/provision-runner-python.ps1: automates the self-hosted GPU
  runner's system-wide Python install (the test-cuda CI job's env.PYTHON
  points at C:\Python<ver>, which the NETWORK SERVICE runner account
  must be able to read). Idempotent: re-running with the same -Version is
  a no-op; changing -Version re-provisions and reminds the operator to
  update env.PYTHON in ci.yml. Requires admin + uv; RUNNER_SETUP.md
  now points at it as the recommended path.
• prepare(auto_bs=True) now prints a one-line note when the forward-only
  sweep is used (no loss_fn): "auto_bs running forward-only (no loss_fn);
  pass loss_fn for a larger batch size". Without a loss the sweep measures
  only activations+params, not grads + optimizer state, so the picked size
  is safe but smaller than a real fwd+bwd sweep would allow; the note
  surfaces why instead of leaving the user to find it in the docstring.
  Silent when a loss_fn is given (the sweep is then real).

Changed

• prepare() now defaults optimize=True (was False). On a GPU this
  turns on the free-win bundle by default — TF32, cudnn.benchmark for CNNs,
  num_workers/pin_memory/persistent_workers defaults on bare loaders,
  and AM...

autotrainer 0.11.0

[0.11.0] - 2026-07-22

Added

• prepare(model, loader, opt, optimize=True): the GPU optimization layer
  the original thesis promised - detect the hardware, set it up for
  throughput, leave the user's hyperparameters alone. When optimize=True
  and CUDA is available, prepare() applies:
  • cudnn.benchmark = True for CNNs (free win on fixed input shape).
  • TF32 on Ampere+ (cuda.matmul.allow_tf32, cudnn.allow_tf32) - a
    ~2-3x matmul speedup that ships disabled for legacy reproducibility
    reasons.
  • Loader defaults on bare DataLoader(...) calls: num_workers
    (capped at 8, sharded by world size), pin_memory=True,
    persistent_workers=True. User-set values are never overridden.
  • AMP implied (use autotrainer.autocast_context() + GradScaler() as
    before); pass amp=False to opt out.
    Every decision is printed; nothing in this path touches lr, loss,
    schedule, or optimizer choice. No-op on CPU and when optimize=False,
    so existing callers see no change.
• prepare(..., auto_bs=True, loss_fn=...): grow the loader's batch size
  until OOM then back off one step. Uses the user's loss_fn for an
  accurate forward+backward measurement; without it the sweep is
  forward-only (conservative). The discovered size rebuilds the loader;
  lr and schedule are NOT changed - pair with accumulate() to scale the
  step to the new effective batch.
• prepare(..., max_bs=N): ceiling for the auto_bs sweep (default 4096).
• prepare(..., compile=True, compile_mode=): wrap the model with
  torch.compile() before any DDP wrap. Order matters - compiling the
  unwrapped module then DDP-wrapping is the documented-supported path;
  the reverse causes graph breaks on the .module indirection. No-op on
  CPU and torch < 2.0. On compile failure (dynamic shapes the backend
  can't handle, missing Triton on Windows, etc.) falls back to the
  uncompiled model with a warning rather than crashing the run.
  compile_mode: default | reduce-overhead (CUDA graphs) |
  max-autotune (kernel search). Does not touch lr/loss/schedule.
• prepare(..., fsdp=True): wrap with FullyShardedDataParallel instead
  of DDP when distributed. FSDP shards params/grads/optimizer state
  across ranks - the path for models too large to fit on one GPU (DDP
  replicates and OOMs). Uses use_orig_params=True so the user's
  optimizer keeps working unchanged. On single-process (world_size==1)
  or torch < 2.0, fsdp=True is a no-op with a warning. Does not touch
  lr/loss/schedule/optimizer.
• prepare(..., cpu_offload=True): when paired with fsdp=True, enables
  CPUOffload(offload_params=True) - moves FSDP-sharded params to CPU and
  brings them to GPU only for the forward/backward. Trades throughput for
  the ability to train models that OOM even when sharded across ranks.
  Ignored with a warning on the DDP path (no built-in CPU param offload)
  and on single-process (world_size == 1). Does not touch lr/loss/schedule.
• Training-loop helpers (autotrainer.zero_grad, eval_mode, train_mode,
  accumulate): the small things users forget inside the loop.
  zero_grad uses set_to_none=True; eval_mode/train_mode are
  context managers that restore the prior mode (kills the classic
  "forgot to flip back to train() after eval" bug); accumulate(opt, steps=N)
  handles gradient accumulation with optional GradScaler integration.
  None touch lr / loss / schedule / optimizer choice.
• autotrainer.BottleneckMonitor: the cheapest piece of the roadmap's
  training-triage theme. Sample per-step data-load vs compute time and
  print a plain-language warning when the dataloader dominates the GPU
  ("raise num_workers / pin_memory / prefetch"). Opt-in; zero overhead
  when not constructed.
• autotrainer.node_scratch() and autotrainer.configure_scratch(): SLURM
  node-local scratch ergonomics. node_scratch() returns $TMPDIR
  (per-job, per-node, auto-cleaned under SLURM; system temp elsewhere),
  suffixed with the SLURM job id so concurrent jobs don't collide.
  configure_scratch() sets TORCHINDUCTOR_CACHE_DIR to it (so
  torch.compile kernel cache doesn't hit NFS) and warns when the scratch
  looks like it's on a network filesystem (NFS/Lustre/GPFS/Panasas). Call
  once at the top of your training script, before any torch.compile.
• CUDA CI: new cuda pytest marker selects GPU-required tests; CPU jobs
  run -m "not cuda" to skip them cleanly. A test-cuda CI job runs the
  cuda-marked subset on a self-hosted GPU runner (runs-on: [self-hosted, gpu]), catching the class of CUDA-path bug that CPU-only CI cannot
  (the device_count() crash, the _pretend_cuda stub defect, and the
  /nfs PermissionError were all found by running on a real GPU). See
  RUNNER_SETUP.md for one-time runner registration, and NEXT_STEPS.md
  for the engineering backlog that came out of this work.
• autotrainer.cuda_device(): canonical helper for picking the CUDA
  device-or-CPU based on device_count() > 0. Centralizes the check
  previously duplicated (inconsistently) across prepare,
  _ensure_process_group, find_lr, _find_lr_synced, and tune.

Fixed

• Public autotrainer.prepare() now forwards keyword arguments to the torch
  backend. Previously the dispatcher in __init__.py called the backend
  with only 3 positional args, so every prepare(optimize=True),
  prepare(compile=True), prepare(fsdp=True), prepare(auto_bs=True),
  etc. raised TypeError: prepare() got an unexpected keyword argument 'optimize' - the entire Tier 1-3 feature surface was unreachable from the
  public API. The CI tests passed only because they imported the backend
  directly. Added a regression test class (TestPublicDispatcherForwardsKwargs)
  that imports the public package and exercises the dispatcher path users
  actually take.
• CUDA device selection now gates on torch.cuda.device_count() > 0, not
  just is_available(), centralized in the new autotrainer.cuda_device()
  helper. The previous check was True whenever the driver was present,
  even when CUDA_VISIBLE_DEVICES="" hid every GPU - so set_device(local_rank)
  crashed with "invalid device ordinal" on driver-present, GPU-hidden
  boxes (e.g. the CPU-gloo distributed tests on a 1-GPU dev machine). All
  four device-pick sites (prepare, _ensure_process_group, find_lr,
  _find_lr_synced, tune) now share the one helper.

autotrainer 0.10.0

Changed (breaking - final API adjustments before 1.0)

• auto() now ALWAYS returns a 5-tuple
  (model, dataloader, optimizer, loss_fn, scheduler); with
  schedule=False the scheduler is None instead of the tuple shrinking
  to 4 items. Unpack the fifth value (it was easy to forget which shape
  you were getting).
• tune()'s data parameters are now train/val (they take DataLoaders
  for PyTorch models and (X, y) tuples for estimators, so _loader
  names were misleading). train_loader=/val_loader= keyword calls
  still work but emit a DeprecationWarning; removal in 1.0. Positional
  calls are unaffected.
• The public API is now explicit: autotrainer.__all__ lists it, and a
  deprecation policy in CONTRIBUTING.md covers everything in it from 1.0
  onward. Submodules and _-prefixed helpers are internal.

Added

• fit() checkpoints now carry a format_version; incompatible or
  pre-0.10 checkpoints are rejected with a clear error instead of being
  silently misread.
• The pdoc API reference is published to GitHub Pages
  (https://orialpha.github.io/Autotrainer/) on every push to main.

autotrainer 0.9.0

Added

• autotrainer.fit(model, train_loader, val_loader): one-call orchestrator
  that composes tuning and distribution. Phase 1 runs the Optuna recipe
  search; phase 2 retrains the winner from the model's original initial
  weights through prepare() (DDP + DistributedSampler when launched
  distributed) with a warmup+cosine schedule, mixed precision, and early
  stopping on the validation loss, restoring the best epoch's weights
  before returning (model, best_params, study).
• Parallel hyperparameter search in fit(): when launched distributed,
  phase-1 trials are split across ALL ranks through a shared Optuna
  journal-file study (study_storage=, default
  .autotrainer_study_<jobid>.log), one trial per process on its own GPU.
  Samplers are seeded per rank; the winning recipe and inferred loss are
  broadcast so every rank trains the same configuration. tune() gained
  storage=/study_name= to join a shared study.
• fit(checkpoint=...): preemption-safe checkpointing. Rank 0 atomically
  writes the full training state (current + best weights, optimizer,
  scheduler, recipe, early-stop counters) after every phase-2 epoch; if
  the file exists when fit() starts, the tuning phase is skipped and
  training resumes where it died - a requeued SLURM job rerunning the same
  script just continues.
• tune() now supports sklearn-API estimators (scikit-learn,
  XGBoost/LightGBM sklearn wrappers): pass (X, y) tuples instead of
  DataLoaders. Curated default search spaces ship for XGBoost, LightGBM,
  random forests/extra trees, gradient boosting, logistic regression,
  ridge/lasso/elastic-net, and SVM; other estimators take a custom
  space=. Scoring defaults to the estimator's own .score() (override
  with scoring=), thread counts follow the SLURM allocation, and the
  user's estimator object is never fitted or mutated. fit() raises a
  helpful TypeError pointing to tune() for non-PyTorch models.
• autotrainer.set_epoch(loader, epoch): call at every epoch start so the
  DistributedSampler installed by prepare() reshuffles each epoch
  (without it, every epoch sees the same order). No-op for non-distributed
  loaders; prepare() now prints a reminder when it installs the sampler.
• AUTOTRAINER_TIMEOUT env var (seconds): overrides the
  torch.distributed collective timeout, for when one rank's phase runs
  much longer than the others' (e.g. an uneven trial split in fit()).
• Real 2-rank distributed tests (gloo on CPU) in the regular suite and CI:
  sampler sharding, LR-broadcast parity, and bit-identical fit() weights
  across ranks.

Changed

• mypy now runs in strict mode (a 1.0 roadmap item): all public and
  internal signatures are fully annotated; optional-framework objects stay
  Any via follow_imports = "skip".

Fixed

• Local multi-GPU launches now rendezvous on a free OS-assigned port
  instead of always 29500, so two autotrainer run jobs on one machine no
  longer collide. An explicit AUTOTRAINER_PORT still pins the port, and
  SLURM keeps the fixed default (all nodes must agree up front).
• _gpu_count(): CUDA_VISIBLE_DEVICES now only restricts the detected
  GPU count instead of being trusted blindly - CUDA_VISIBLE_DEVICES=0 on
  a GPU-less machine no longer reports a phantom GPU (which sent the
  launcher into single-GPU CUDA mode on CPU boxes).
• prepare() and tune() no longer discard user DataLoader settings when
  rebuilding loaders: pin_memory (was forced on CUDA), timeout,
  worker_init_fn, generator, persistent_workers, and prefetch_factor
  are now carried over. prepare() also honors shuffle=False
  (SequentialSampler) instead of always shuffling, passes through loaders
  that already have a DistributedSampler, and raises a clear TypeError
  for batch_sampler=/IterableDataset loaders (previously a crash or
  silent data duplication across ranks) - and it validates the loader
  before joining the process group, so a bad loader fails fast instead of
  hanging the other ranks.
• The inferred BCE loss is now directly usable on the user's batches: it
  accepts the integer (N,) targets that triggered its selection (plain
  BCEWithLogitsLoss requires float targets shaped like the (N, 1)
  logits and raised on the very next batch). Applies to loss="bce"
  overrides too.
• auto() under DDP now runs the LR range test on rank 0 only and
  broadcasts the result. Previously every rank swept independently -
  wasted work, and differently-shuffled loaders could land each rank on a
  different LR, silently desynchronizing the DDP replicas.