Skip to content

Releases

v0.2.2

Choose a tag to compare

@github-actions github-actions released this 04 Nov 14:16
9a631ca

ReactantNitro v0.2.2

Diff since v0.2.1

strict for checkpoints with extra or missing leaves

Nitro(e; weights = source, strict = false) loads a source whose parameter tree differs from the model's, as PyTorch's load_state_dict(strict = False) does. Parameters are matched by keypath: one the source lacks keeps the model's fresh initialization, and one the model lacks is dropped. Both always warn, listing the leaves.

n = Nitro(MyExp(; head = true); weights = :latest => :best, strict = false)

strict = true is the default and refuses such a source, naming the leaves and pointing at strict = false. A parameter held at different sizes refuses either way. resume refuses strict = false, and a partial load refuses restore_optimizer = true, since the record's optimizer state follows its own leaves. strict applies to every weights source: a path, run => :best | :latest, run => Average(...), and another Nitro.

The nitro_train, nitro_validate, nitro_evaluate, nitro_predict and nitro_export tools take strict as well.

Changes

  • The layer state st is now compared with the model's when a checkpoint loads. A leaf that does not fit, or that only one side holds, warns, naming it as st.<keypath>, and keeps the model's fresh initialization. This applies under either strict setting. Before, st was replaced wholesale from the record without a check.
  • A weights = other::Nitro warm start now matches the two trees by keypath rather than by position.
  • check_record_tree_compatible is replaced by restore_trees.

Fixes

  • Manual mode now counts the step body in t_step, so data_wait_frac is no longer logged as 1.0 every epoch and the data wait warning no longer fires spuriously.

v0.2.1

Choose a tag to compare

@github-actions github-actions released this 27 Oct 17:26
27b35f6

ReactantNitro v0.2.1

Diff since v0.2.0

MLUtils.DataLoader with num_workers now uses its worker processes

A split that is a DataLoader with num_workers > 0 now runs one producer over the loader's own iterate, so MLUtils' worker processes build the batches. Before, the framework fanned out over getobs in its own process and the workers never started.

MLUtils delivers worker batches as they finish, so the order of an epoch does not reproduce. Setup therefore refuses such a split unless it states that:

train = PrefetchIterator(DataLoader(data; batchsize, num_workers = 4); ordered = false)

The binding report shows the split as prefetch: 1 producer, ...; the source parallelizes itself. A source type of your own can opt into the same path by defining ReactantNitro.delegates_parallelism(source) = true.

When using Revise.jl, treat num_workers with care The training process stays on the code a run started with, but worker processes do not: Distributed calls each batch request with invokelatest, so they run the newest definition of a function. If Revise is loaded, saving an edit to getobs, an augmentation or a collate function changes the batches of a run that is already training, from the next batch on, while forward and loss stay on the old code. Do not edit the data pipeline during a run that uses num_workers. The issue is tracked here: MLUtils.jl#267.

Fixes

  • An integer parallel on a DataLoader (a worker-task count) raised a TypeError at setup on the single-producer path.

Compat

  • The MLUtils floor is now 0.4.12.
  • Every stdlib dependency declares compat "1".

Docs

  • A new Data loading page covers DataLoader setup, when to use num_workers, sharing data with worker processes, how a split is driven, and writing a source of your own.

Closed issues:

  • External Dataloader (#56)

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 21 Oct 03:19
5ec6538

ReactantNitro v0.2.0

Diff since v0.1.13

Breaking changes

  • One monitor for ranking. Set monitor(e) = :metric => :mode (or the Nitro keyword monitor) once. Checkpointer, TopKCheckpointer, EarlyStopping and the new Plateau follow it unless given both their own metric and mode. Giving only one of the two is now an error.
  • No default when it might not exist. monitor defaults to :loss => :min only when the experiment defines no metrics or sets eval_loss = true. Otherwise a checkpointer with a Best selector, EarlyStopping or Plateau fails at setup until monitor is declared.
  • val_loss is now loss. Metric names carry no split. The framework loss is loss on every split, and :val_loss is accepted with a warning until 1.0.0. Manifests from before 0.2.0 are read as loss, so old runs keep ranking.
  • History column renamed. In history(nitro) the train loss column is now train_loss, and h.loss is the validation loss.
  • EarlyStopping() used outside a Nitro has no metric until setup fills it in. Give it metric and mode to call should_stop directly.

New

  • Plateau(inner; factor, patience, ...): PyTorch's ReduceLROnPlateau for any schedule key. Its state is saved with the checkpoint, so a resume continues it.
  • loss may return (l, terms). Each term is logged as loss/<term> on train, validation and test (#48 - @benjwweber)
  • eval_loss = true reports the loss beside user metrics on eval splits. A key reported by both is an error.

Fixed

  • Resuming from a mid-run checkpoint no longer loses one epoch of early-stopping patience.

v0.1.13

Choose a tag to compare

@github-actions github-actions released this 20 Oct 19:30
aaf711b

ReactantNitro v0.1.13

Diff since v0.1.12

Weights-only loads accept checkpoints from before a GraphConst field was added

A GraphConst field added to an experiment after a checkpoint was written no longer counts as a configuration change while it holds its declared default, for resume and weights alike. By convention a new field's default reproduces the behaviour from before it existed, so older checkpoints stay loadable.

A weights load whose GraphConst fields differ from the record's now warns with the diff instead of refusing. It builds the program from the current experiment and reuses only the parameters, so a changed field cannot mix two programs. A resume still refuses, since it continues the recorded program.

Fix: a checkpoint from a different architecture was not caught on restore

Every restore now checks the record's parameter tree against the freshly built model, by keypath and size, before the record's parameters replace it. The error names the first mismatching leaves with both sizes. The existing permutation check built its layout from the record's own tree, so a record from a different architecture got past it.

v0.1.12

Choose a tag to compare

@github-actions github-actions released this 18 Oct 22:23
d1dd9c7

ReactantNitro v0.1.12

Diff since v0.1.11

Fix: SAM no longer skips the leaves weight decay skips

SAM excluded the leaves no_decay excludes (biases and norm affines by default) from its perturbation and from ‖g‖. That tie to weight decay was never intended: Foret et al. perturb every parameter. SAM now perturbs every leaf by default, and the new no_perturb(e, keypath, x) hook chooses what it leaves alone, independently of no_decay. To keep the 0.1.11 behaviour:

ReactantNitro.no_perturb(e::MyExp, ks, x) = default_no_decay(ks, x)

A SAM run that relied on the old exclusion takes a slightly different trajectory from 0.1.12 on.

Fix: a resume through the Kaimon tools opened a new logged run

The tools wrap the experiment's logger to record live metrics, and the wrapper did not forward logger_state or reattach!. A run started through nitro_train therefore wrote checkpoints with no resumption state, and resuming it, through the tools or not, started a new run in the backend (a new Comet experiment, for instance) instead of continuing the original. Both now forward, and the record names the wrapped logger's type, so a record written through the tools resumes outside them and the reverse. This dates to 0.1.0. Checkpoints written through the tools before 0.1.12 hold no state, so resuming one still starts a new logged run.

Adaptive SAM

ASAM(rho; eta = 0.01f0) (Kwon et al. 2021) scales the perturbation by each parameter's own size, ε = rho * T² g / ‖T g‖ with T = |w| + eta, so the sharpness it measures is invariant to rescaling the weights:

ReactantNitro.grad_rule(::MyExp) = ASAM(0.5f0)

adaptive_perturb(e, keypath, x) chooses which leaves are scaled: rank-2 and higher by default, with biases, norm affines and scalars unscaled as under SAM. rho is relative, so typical values are about ten times SAM's, and it schedules and logs as SAM's does. eta is not schedulable, so an unqualified eta schedule key still means the learning rate. Cost, sam/gap and what is reported are as for SAM. A custom PerturbRule written against the four-argument perturbation is unchanged.

Exponential moving average of the weights

EMA(beta; warmup = true, every = 1), set through the new ema accessor or keyword, keeps a device copy of the weights averaged after each optimizer step, in the automatic loop and in manual mode:

ReactantNitro.ema(::MyExp) = EMA(0.999)

Under eval_weights = :ema, the default with an EMA, validation, checkpoint selection, restore_best, predict and export read the average; eval_weights = :raw evaluates the raw weights. Training and parameters always use the raw weights. Checkpoints keep the average in an optional ema field, a resume continues it, and checkpoint_info reports has_ema. load_ema = true loads a source's average as the weights of a new handle, from a record, an Average of records, or another Nitro. The train line logs ema/beta. Records written before 0.1.12 load as before, with no EMA.

Docs

ASAM, no_perturb and EMA on the Optimization page; EMA in checkpoints, resume and averaging on the Checkpoints page; the new keywords and hooks on the Binding cheat sheet; sam/gap and ema/beta on the Logging page.

Tests

ASAM is checked against a hand-computed reference, and equals SAM when no leaf is adaptive. no_perturb is checked to exclude leaves and to perturb every leaf by default. The EMA is checked against a hand-computed trajectory with warmup, at beta = 0 and beta = 1, across a checkpoint and resume (two epochs equal one plus one), and for what evaluation and load_ema read. ASAM and the EMA also run on a two-device mesh.

v0.1.11

Choose a tag to compare

@github-actions github-actions released this 17 Oct 21:09
77efa60

ReactantNitro v0.1.11

Diff since v0.1.10

Fix: scheduled optimizer fields other than eta were never applied

The automatic loop looked up only eta each step, so a scheduled rho, lambda or other rule field was accepted, listed in the binding report, and had no effect. Every scheduled field now reaches its group's rule, with lambda applied as eta_g * lambda(t).

A Level 2 optimizer(e, group, hp) had three related gaps, also fixed:

  • hp lacked the scheduled fields at setup, so Momentum(hp.eta, hp.rho) failed before training
  • a factory that ignored a scheduled field passed silently; setup now probes where each one lands, refuses one that lands nowhere, and the binding report shows where each landed
  • a value computed from hp, such as 2 * hp.eta, was baked into the optimizer program at its first value, so the schedule froze. This dates to 0.1.0

In manual mode, a path key now overrides the bare key for its subtree whatever order they are written in.

Schedule values are logged

Every varying scalar schedule is logged on the train line, keyed schedule/<field> for a Device field and schedule/<group>/<field> for a rule field as applied. autolog_schedules = false turns this off. log_every_n_steps (default 1) writes the per-step lines on every nth step only, for fast runs and hosted loggers.

Sharpness-aware minimization

grad_rule(e) chooses the gradient each micro-batch contributes: SingleGrad() by default, or SAM(rho):

ReactantNitro.grad_rule(::MyExp) = SAM(0.05f0)
ReactantNitro.schedules(::MyExp) = (; rho = _ -> t -> 0.05f0 * min(1, t / 1000))

SAM runs the existing gradient program twice per micro-batch, at ps and at ps + rho * g / ‖g‖, so it adds no gradient compile and leaves the optimizer program unchanged. The loss, layer state and train metrics come from the unperturbed pass, and the train line adds sam/gap. Under accum > 1 the perturbation is per micro-batch, which is m-sharpness with m the micro-batch size. Leaves no_decay excludes are not perturbed. Manual mode supports SingleGrad only.

Schedule binding: the grad namespace and bind

schedules gains two reserved namespaces. grad holds gradient rule fields such as SAM's rho. bind holds names of your choosing that match nothing: a Level 2 optimizer(e, group, hp) and the new grad_rule(e, hp) read them as hp.bind.<name> and place them. An unqualified key resolves to the one namespace it names; a key that names several is refused, and the error shows the qualified forms and bind.

Checkpoints: Checkpointer and selectors

Checkpointer(; keep) replaces TopKCheckpointer. keep is a set of selectors, Best(k), Last(n),Epochs(r) and Every(n), and the run keeps their union:

Checkpointer(; keep = (Best(3), Last(10)), metric = :val_loss, mode = :min)

The default, (Best(3), Last(1)), is the previous behaviour. An epoch is written only if some selectorwould keep it.

TopKCheckpointer(; k, keep_latest, ...) is deprecated, warns once per session, and is supported until at least 1.0.0. It builds the equivalent Checkpointer, which writes identical files. TopKCheckpointer is now a function rather than a type, so code that dispatches on it or checks isa TopKCheckpointer must use Checkpointer, the type the default checkpointer now is. ckpt.k and ckpt.keep_latest remain readable.

Checkpoint averaging

weights takes an average of one run's checkpoints:

Nitro(e; weights = :latest => Average(Last(10)))
Nitro(e; weights = :latest => Average(Best(5; among = Last(20))))
average_checkpoints("runs/my_run", :latest => Average(Last(10)); save = "runs/my_run/avg10.jld2")

Parameters are averaged uniformly; layer state comes from the best picked checkpoint when the selector ranks and the latest otherwise. restore_best = Average(...) ends train! holding the run's own average. An averaged record carries averaged_from (the files, epochs and weights it averages), shown by checkpoint_info and recorded in export provenance. Best(1) and Last(1) are the same sources as :bestand :latest. Averaging across runs is refused for now.

The Kaimon gate tools take the same sources as strings, written as Julia (weights = "latest => Average(Last(10))", restore_best = "Average(Last(10))"), parsed against a whitelist and never evaluated.

Other fixes

  • EarlyStopping counters are per handle and restored on resume: a second handle no longer inherits the first run's counters, and a resume continues its patience
  • A logger without run_id or run_url no longer fails at the first checkpoint, and a stop from request_stop! reaches the logger as :requested
  • An export echo whose name matches a forward output is refused instead of silently replacing it

Docs

A Checkpoints page; SAM on the Optimization page, including how it interacts with gradient accumulation; grad and bind on the Schedules page; and a precision pass over the guides and docstrings.

Tests

SAM is checked against a hand-computed step, and bitwise against the default rule at rho = 0 with dropout, RAdam and accum = 2, on one and two devices. TopKCheckpointer and the equivalent Checkpointer write identical files. Averages are checked against hand-computed means, and every selector round-trips through the string grammar.

v0.1.10

Choose a tag to compare

@github-actions github-actions released this 16 Oct 20:52
69f3743

ReactantNitro v0.1.10

Diff since v0.1.9

Fix: program outputs on a multi-device mesh

With n_devs > 1, validation, predict and some training paths failed when a value leaving a compiled program was either an input returned unchanged or a lazy reshape/view wrapper:

  • MethodError: no method matching (ConcretePJRTArray{T, N})(::Tuple{AsyncBuffer, AsyncBuffer}, ...) for an unchanged input
  • TypeError: in new, expected ConcretePJRTArray{T, N, 1}, got a value of type ConcretePJRTArray{T, N, 2} for a wrapper

Both come from Reactant's PJRT output codegen (EnzymeAD/Reactant.jl#3488 for the unchanged input, EnzymeAD/Reactant.jl#2989 for wrappers). ReactantNitro now works around them at its program boundary:

  • program_out materializes reshape/view wrappers and passes every returned array through optimization_barrier, so each output leaves a program as a fresh array with its sharding intact.
  • forward outputs in the eval program (validate, evaluate, predict) now go through program_out/program_in, like layer state and metrics already did.

A computed output costs nothing; an input returned unchanged costs one device copy. Single-device runs are unaffected. A user struct other than a Tuple, NamedTuple, Pair or array wrapper still cannot leave a program on a mesh until EnzymeAD/Reactant.jl#2989 is fixed upstream.

Tests

multi_device outputs (two forced host devices) covers an input returned unchanged, a reshape of an input, reshape and view of a computed array, and a layer state array that never changes, through training with :host metrics, validation and predict, compared against n_devs = 1.

v0.1.9

Choose a tag to compare

@github-actions github-actions released this 08 Oct 00:19
9f486e7

ReactantNitro v0.1.9

Diff since v0.1.8

New features

  • Metric modes. A validation metric is now written value => mode. The mode can be a count (summed, then divided at the end), :sum, :max, :min or :concat. :max and :min are elementwise across batches (#41)
  • :concat joins a value along its last axis across the split, so finalize_metrics gets one entry per sample. This makes rank-based metrics exact, such as AUROC, average precision, Spearman and quantiles. The last axis must equal the batch's real sample count, and the joined values are aligned across keys, though not in dataset order. A :concat array that finalize_metrics returns without reducing it is refused, naming its path (#41)
  • Groups. A key can hold a NamedTuple or Tuple of related values. A mode on the group covers every value inside it, and values in a group can have different modes: (; score = p => :concat, n_pos = count(y) => :sum). finalize_metrics receives the same shape with the modes removed. A value with no mode or with two modes is refused, and the error names its path (#41)
  • Modes and groups work under both :host and :device metric residency (#41)

Bug fixes

  • Training with n_devs > 1 failed in three places, and none of them had a test (#42):
    • fire_monitors called rank on the device mesh
    • A ReactantRNG replicated on a mesh failed after the first gradient step
    • Metric pairs returned from the eval program on a mesh failed in the same way. RNG state and metric pairs now leave compiled programs as untyped surrogates and are rebuilt after the call

Behavior changes

  • (value, count) and (value, nothing) still work at the top of a key. nothing now means :sum. Any mode outside the list above is refused on the first batch (#41)
  • A ReactantRNG placed on a mesh uses the PHILOX algorithm, because XLA's GPU SPMD partitioner miscompiles the default rng_bit_generator. The RNG algorithm is now part of the compile-cache key (#42)

Documentation

  • The Metrics page is rewritten around value => mode. It traces one split end to end and covers exact R² from sums, exact AUROC from :concat, binned AUROC from :sum, and which mode each kind of metric needs. The tutorial, binding sheet, metrics docstring and MNIST example use the new syntax (#41)

Infrastructure

  • Multi-device regression tests train on 2 forced CPU devices in a subprocess as part of the default suite. They cover dropout masks, including across a resume, and BatchNorm global statistics compared with n_devs = 1 (#42)

v0.1.8

Choose a tag to compare

@github-actions github-actions released this 07 Oct 15:26
470c2f5

ReactantNitro v0.1.8

Diff since v0.1.7

New features

  • TensorFlow SavedModel export. export_model(n, TFSavedModel(); ...) writes a SavedModel for TF Serving and the platforms built on it (SageMaker, Vertex AI, KServe). Each traced program is one XlaCallModule op wrapping the StableHLO, the form JAX's native serialization produces. The backend is a PythonCall extension: using PythonCall with tensorflow importable enables it (#40)
  • One signature per entry of batch_sizes (serving_b{N}, with serving_default the first), all sharing one copy of the weights. Tensors are batch-first: the Julia shape (W, H, C, N) is the TF shape [N, C, H, W] (#40)
  • TFSavedModel(; platform, stablehlo_version, call_module_version). The defaults ("CUDA", v"1.5.0", 9) load on TF 2.17 and 2.20; stablehlo_version = v"0.14.0" with call_module_version = 8 loads on TF 2.15. All three releases were verified by running every signature against the eager program (#40)
  • A program containing a custom_call is refused, and so is a model declaring export_client_outputs, since a SavedModel cannot ship model.jl. Any other postprocess is dropped with a warning. Provenance and the working-tree patch are written to assets.extra/ (#40)
  • compile_options now checks the fields of xla_debug_options, xla_executable_build_options and xla_compile_options against XLA's protos when Nitro is constructed, rather than failing at the first compile (#39)

Behavior changes

  • site_provenance now has a default for every backend, ReactantNitro.git_provenance(root), which collects the commit, tree hash, branch, dirty flag, remote and the working-tree patch. Previously a backend without its own method raised an error when provenance_root was given. ReactantServerBundle still uses ReactantServerExport.collect_provenance (#40)

Documentation

  • A Compile options page, including autotuning (#39)
  • A TensorFlow SavedModel section in the Export guide (#40)

Dependencies

  • Added PythonCall = "0.9.25" as a weak dependency and Dates as a dependency (#40)

Infrastructure

  • A separate SavedModel CI job installs TensorFlow 2.20 through CondaPkg and runs the end-to-end test in test/savedmodel. It runs only when the export code changes and is not cached, so the Python environment never evicts the other jobs' caches (#40)

v0.1.7

Choose a tag to compare

@github-actions github-actions released this 07 Oct 06:18
39e3e9c

ReactantNitro v0.1.7

Diff since v0.1.6

New features

  • Precise invalidation, on by default. Each compile now records every method specialization its trace actually went through, and the compile cache poisons an entry when any of them is redefined, deleted, or shadowed by a more specific method (for example from a package extension loaded after compiling). This covers dependencies the inference-based closure could not see: methods that dispatch only on traced types (such as LuxLib's traced mean_var), Reactant overlays, callees behind runtime dispatch, and helpers inlined several calls deep. Drift reports name the method that changed. Reactant itself is not patched and installing the hook invalidates nothing; tracing time is unchanged within measurement noise (#38)
  • Set the precise_invalidation preference to false to fall back to the inference closure. The fallback is also used automatically, with one warning, if Reactant's internals ever stop matching what the hook expects (#38)

Bug fixes

  • A program compiled in an older world than the latest (for example after @eval followed by a compile in the same function) is now recognized as compiled against stale definitions and is not reused (#38)

Documentation

  • Removed the claim that a custom ChainRules rrule is a cache coverage gap. Reactant differentiates at the MLIR level through Enzyme-JAX and never consults ChainRules, so the backward pass has no Julia methods to guard (#38)

Dependencies

  • Added Preferences = "1" (#38)

Closed issues:

  • Address method revise invalidation blind spots (#1)