Skip to content

autotrainer 0.13.0

Choose a tag to compare

@OriAlpha OriAlpha released this 28 Jul 05:59

[0.13.0] - 2026-07-28

Added

  • metric= on tune() and fit(): select on the number you actually care
    about.
    The torch path could only ever optimize validation loss - it
    picked the recipe, pruned trials, early-stopped, and chose the best epoch on
    it - while the sklearn-estimator path has scored on accuracy/R² through
    scoring= since 0.10. Loss is a proxy, and it comes apart from the goal
    exactly where the search space is widest: a regularized recipe (label
    smoothing, augmentation) is less confident, so val cross-entropy can bottom
    out and start climbing from overconfidence while val accuracy is still
    improving - and a loss-driven patience stops there and restores a
    less-accurate checkpoint. Pass metric="accuracy", "f1", "auc", "r2",
    or a callable(model, loader) -> float (with direction= if lower is
    better). Training still always uses the loss; this changes only what runs are
    scored by. Default is "loss", so existing behavior is unchanged.
  • autotrainer.metrics. Pure-torch implementations of the built-ins, so
    scoring on a metric doesn't make scikit-learn a hard dependency. F1 is
    macro-averaged over the classes present in the targets (the one to reach for
    on imbalanced data, where accuracy flatters a majority-class predictor); AUC
    is rank-based with average ranks for ties, which a naive implementation gets
    wrong whenever a saturated model emits identical logits, and its multiclass
    one-vs-rest form ranks on softmax probabilities rather than raw logits (they
    differ: softmax's per-sample denominator makes it non-monotone in a single
    logit column). All four agree with scikit-learn to 1e-9 across 1500
    randomized cross-checks.
  • Preemption handling in fit(checkpoint=...). The resumable checkpoint
    has existed since 0.11, but nothing caught the scheduler's warning shot, so a
    job preempted mid-epoch still threw that epoch away - and a job preempted
    during phase 1 threw away the entire search. fit() now watches
    SIGUSR1/SIGTERM (SLURM's --signal=B:USR1@120) and stops at the next epoch
    boundary after the checkpoint is written, so the requeued job resumes at the
    following epoch. Phase 1 is journaled to <checkpoint>.study, so an
    interrupted search resumes with its completed trials instead of restarting.
    Both are armed only when checkpoint= is set.
  • Data sanity checks in auto() and tune(). The failures that aren't the
    framework's fault and don't announce themselves: un-normalized or non-finite
    inputs (which look like a bad learning rate), class imbalance (which looks
    like a model that "works" at 95%), constant inputs or targets, and - the
    expensive one - a train/val split that overlaps, which doesn't fail at all,
    it just reports an excellent validation score that every trial then optimizes
    against. The batches were already being peeked to infer the loss, so the
    checks are nearly free, and they run before the LR range test and the first
    trial rather than after an allocation is spent. Overlap is exact when both
    loaders are Subsets of one dataset, and otherwise compares a bounded sample
    of rows byte-for-byte (skipped below 4 features, where identical rows arise
    by chance). Warnings only - nothing is changed for you. sanity=False turns
    them off; tune() also gates them on verbose.
  • autotrainer.sanity. The checks themselves (report() / overlap()),
    returning message strings rather than printing, so they can be called
    directly against your own loaders.
  • autotrainer.preempt. The watcher behind it (watch() / preempted()),
    usable in hand-written loops. The handler only sets a flag - it never raises
    or exits - so your loop chooses a stopping point where its state is
    consistent.
  • epochs is now a searched hyperparameter. Training length is one of the
    largest levers on final quality, and every trial previously ran on one fixed
    horizon. Each trial now trains for its own searched budget and anneals its
    LR schedule over exactly that budget, so a recipe that wants a short, fast
    anneal and one that wants a long, gentle one are each evaluated the way they
    would actually run. The range is 1..epochs_per_trial, so adding the knob
    widens what is searched without making any trial cost more than it did
    before; raise epochs_per_trial to widen it.
  • aug_strength: searchable image augmentation for CNNs. A single scalar
    dials a fixed flip + cutout policy from off to standard, and it enters the
    default space only when conv layers are detected (the policy is
    image-specific, so searching it elsewhere would burn trials on a dimension
    that changes nothing). 0.0 is in range, so "no augmentation" stays a
    candidate. The winning strength carries into fit()'s phase-2 retrain -
    dropping it there would retrain a different, unregularized recipe.
    Augmentation is applied to inputs on the training path only; validation and
    test passes always see clean data.
  • autotrainer.augment_batch(x, strength). The primitive behind
    aug_strength, exported for use in hand-written loops. Pure torch (no new
    torchvision dependency), target-preserving, never mutates its input, and a
    no-op on anything that isn't a float NCHW batch - so it can sit in a loop
    that also trains tabular or text models.

Changed

  • ASHA now prunes on normalized rungs instead of raw epoch indices.
    Required by the epochs knob: with per-trial budgets, reporting at the raw
    epoch index would pit a long-budget trial (still mid-anneal at epoch 1)
    against a short-budget one that had already finished annealing, pruning the
    long trial for converging later rather than worse. Trials now report at a
    fixed number of rungs measured as fraction-of-own-schedule-completed, which
    compares like with like. Custom pruner= overrides are unaffected.
  • fit()'s held-out test number is stored under test_score. It is no
    longer necessarily a loss. test_loss is still set as well when the metric
    is the loss, so existing readers keep working.
  • Checkpoint format version 2 (adds metric/direction). fit() refuses
    to resume a checkpoint written under a different metric rather than compare
    an accuracy against a stored loss and restore the wrong epoch. Version-1
    checkpoints are rejected with the usual "delete to start fresh" message.