Repository navigation
Releases
v0.2.2
ReactantNitro v0.2.2
strict for checkpoints with extra or missing leaves
Nitro(e; weights = source, strict = false) loads a source whose parameter tree differs from the model's, as PyTorch's load_state_dict(strict = False) does. Parameters are matched by keypath: one the source lacks keeps the model's fresh initialization, and one the model lacks is dropped. Both always warn, listing the leaves.
n = Nitro(MyExp(; head = true); weights = :latest => :best, strict = false)strict = true is the default and refuses such a source, naming the leaves and pointing at strict = false. A parameter held at different sizes refuses either way. resume refuses strict = false, and a partial load refuses restore_optimizer = true, since the record's optimizer state follows its own leaves. strict applies to every weights source: a path, run => :best | :latest, run => Average(...), and another Nitro.
The nitro_train, nitro_validate, nitro_evaluate, nitro_predict and nitro_export tools take strict as well.
Changes
- The layer state
stis now compared with the model's when a checkpoint loads. A leaf that does not fit, or that only one side holds, warns, naming it asst.<keypath>, and keeps the model's fresh initialization. This applies under eitherstrictsetting. Before,stwas replaced wholesale from the record without a check. - A
weights = other::Nitrowarm start now matches the two trees by keypath rather than by position. check_record_tree_compatibleis replaced byrestore_trees.
Fixes
- Manual mode now counts the step body in
t_step, sodata_wait_fracis no longer logged as 1.0 every epoch and the data wait warning no longer fires spuriously.
v0.2.1
ReactantNitro v0.2.1
MLUtils.DataLoader with num_workers now uses its worker processes
A split that is a DataLoader with num_workers > 0 now runs one producer over the loader's own iterate, so MLUtils' worker processes build the batches. Before, the framework fanned out over getobs in its own process and the workers never started.
MLUtils delivers worker batches as they finish, so the order of an epoch does not reproduce. Setup therefore refuses such a split unless it states that:
train = PrefetchIterator(DataLoader(data; batchsize, num_workers = 4); ordered = false)The binding report shows the split as prefetch: 1 producer, ...; the source parallelizes itself. A source type of your own can opt into the same path by defining ReactantNitro.delegates_parallelism(source) = true.
When using Revise.jl, treat num_workers with care The training process stays on the code a run started with, but worker processes do not: Distributed calls each batch request with invokelatest, so they run the newest definition of a function. If Revise is loaded, saving an edit to getobs, an augmentation or a collate function changes the batches of a run that is already training, from the next batch on, while forward and loss stay on the old code. Do not edit the data pipeline during a run that uses num_workers. The issue is tracked here: MLUtils.jl#267.
Fixes
- An integer
parallelon aDataLoader(a worker-task count) raised aTypeErrorat setup on the single-producer path.
Compat
- The MLUtils floor is now 0.4.12.
- Every stdlib dependency declares compat
"1".
Docs
- A new Data loading page covers
DataLoadersetup, when to usenum_workers, sharing data with worker processes, how a split is driven, and writing a source of your own.
Closed issues:
- External Dataloader (#56)
v0.2.0
ReactantNitro v0.2.0
Breaking changes
- One
monitorfor ranking. Setmonitor(e) = :metric => :mode(or theNitrokeywordmonitor) once.Checkpointer,TopKCheckpointer,EarlyStoppingand the newPlateaufollow it unless given both their ownmetricandmode. Giving only one of the two is now an error. - No default when it might not exist.
monitordefaults to:loss => :minonly when the experiment defines nometricsor setseval_loss = true. Otherwise a checkpointer with aBestselector,EarlyStoppingorPlateaufails at setup untilmonitoris declared. val_lossis nowloss. Metric names carry no split. The framework loss islosson every split, and:val_lossis accepted with a warning until 1.0.0. Manifests from before 0.2.0 are read asloss, so old runs keep ranking.- History column renamed. In
history(nitro)the train loss column is nowtrain_loss, andh.lossis the validation loss. EarlyStopping()used outside aNitrohas no metric until setup fills it in. Give itmetricandmodeto callshould_stopdirectly.
New
Plateau(inner; factor, patience, ...): PyTorch'sReduceLROnPlateaufor any schedule key. Its state is saved with the checkpoint, so a resume continues it.lossmay return(l, terms). Each term is logged asloss/<term>on train, validation and test (#48 - @benjwweber)eval_loss = truereports the loss beside usermetricson eval splits. A key reported by both is an error.
Fixed
- Resuming from a mid-run checkpoint no longer loses one epoch of early-stopping patience.
v0.1.13
ReactantNitro v0.1.13
Weights-only loads accept checkpoints from before a GraphConst field was added
A GraphConst field added to an experiment after a checkpoint was written no longer counts as a configuration change while it holds its declared default, for resume and weights alike. By convention a new field's default reproduces the behaviour from before it existed, so older checkpoints stay loadable.
A weights load whose GraphConst fields differ from the record's now warns with the diff instead of refusing. It builds the program from the current experiment and reuses only the parameters, so a changed field cannot mix two programs. A resume still refuses, since it continues the recorded program.
Fix: a checkpoint from a different architecture was not caught on restore
Every restore now checks the record's parameter tree against the freshly built model, by keypath and size, before the record's parameters replace it. The error names the first mismatching leaves with both sizes. The existing permutation check built its layout from the record's own tree, so a record from a different architecture got past it.
v0.1.12
ReactantNitro v0.1.12
Fix: SAM no longer skips the leaves weight decay skips
SAM excluded the leaves no_decay excludes (biases and norm affines by default) from its perturbation and from ‖g‖. That tie to weight decay was never intended: Foret et al. perturb every parameter. SAM now perturbs every leaf by default, and the new no_perturb(e, keypath, x) hook chooses what it leaves alone, independently of no_decay. To keep the 0.1.11 behaviour:
ReactantNitro.no_perturb(e::MyExp, ks, x) = default_no_decay(ks, x)A SAM run that relied on the old exclusion takes a slightly different trajectory from 0.1.12 on.
Fix: a resume through the Kaimon tools opened a new logged run
The tools wrap the experiment's logger to record live metrics, and the wrapper did not forward logger_state or reattach!. A run started through nitro_train therefore wrote checkpoints with no resumption state, and resuming it, through the tools or not, started a new run in the backend (a new Comet experiment, for instance) instead of continuing the original. Both now forward, and the record names the wrapped logger's type, so a record written through the tools resumes outside them and the reverse. This dates to 0.1.0. Checkpoints written through the tools before 0.1.12 hold no state, so resuming one still starts a new logged run.
Adaptive SAM
ASAM(rho; eta = 0.01f0) (Kwon et al. 2021) scales the perturbation by each parameter's own size, ε = rho * T² g / ‖T g‖ with T = |w| + eta, so the sharpness it measures is invariant to rescaling the weights:
ReactantNitro.grad_rule(::MyExp) = ASAM(0.5f0)adaptive_perturb(e, keypath, x) chooses which leaves are scaled: rank-2 and higher by default, with biases, norm affines and scalars unscaled as under SAM. rho is relative, so typical values are about ten times SAM's, and it schedules and logs as SAM's does. eta is not schedulable, so an unqualified eta schedule key still means the learning rate. Cost, sam/gap and what is reported are as for SAM. A custom PerturbRule written against the four-argument perturbation is unchanged.
Exponential moving average of the weights
EMA(beta; warmup = true, every = 1), set through the new ema accessor or keyword, keeps a device copy of the weights averaged after each optimizer step, in the automatic loop and in manual mode:
ReactantNitro.ema(::MyExp) = EMA(0.999)Under eval_weights = :ema, the default with an EMA, validation, checkpoint selection, restore_best, predict and export read the average; eval_weights = :raw evaluates the raw weights. Training and parameters always use the raw weights. Checkpoints keep the average in an optional ema field, a resume continues it, and checkpoint_info reports has_ema. load_ema = true loads a source's average as the weights of a new handle, from a record, an Average of records, or another Nitro. The train line logs ema/beta. Records written before 0.1.12 load as before, with no EMA.
Docs
ASAM, no_perturb and EMA on the Optimization page; EMA in checkpoints, resume and averaging on the Checkpoints page; the new keywords and hooks on the Binding cheat sheet; sam/gap and ema/beta on the Logging page.
Tests
ASAM is checked against a hand-computed reference, and equals SAM when no leaf is adaptive. no_perturb is checked to exclude leaves and to perturb every leaf by default. The EMA is checked against a hand-computed trajectory with warmup, at beta = 0 and beta = 1, across a checkpoint and resume (two epochs equal one plus one), and for what evaluation and load_ema read. ASAM and the EMA also run on a two-device mesh.
v0.1.11
ReactantNitro v0.1.11
Fix: scheduled optimizer fields other than eta were never applied
The automatic loop looked up only eta each step, so a scheduled rho, lambda or other rule field was accepted, listed in the binding report, and had no effect. Every scheduled field now reaches its group's rule, with lambda applied as eta_g * lambda(t).
A Level 2 optimizer(e, group, hp) had three related gaps, also fixed:
hplacked the scheduled fields at setup, soMomentum(hp.eta, hp.rho)failed before training- a factory that ignored a scheduled field passed silently; setup now probes where each one lands, refuses one that lands nowhere, and the binding report shows where each landed
- a value computed from
hp, such as2 * hp.eta, was baked into the optimizer program at its first value, so the schedule froze. This dates to 0.1.0
In manual mode, a path key now overrides the bare key for its subtree whatever order they are written in.
Schedule values are logged
Every varying scalar schedule is logged on the train line, keyed schedule/<field> for a Device field and schedule/<group>/<field> for a rule field as applied. autolog_schedules = false turns this off. log_every_n_steps (default 1) writes the per-step lines on every nth step only, for fast runs and hosted loggers.
Sharpness-aware minimization
grad_rule(e) chooses the gradient each micro-batch contributes: SingleGrad() by default, or SAM(rho):
ReactantNitro.grad_rule(::MyExp) = SAM(0.05f0)
ReactantNitro.schedules(::MyExp) = (; rho = _ -> t -> 0.05f0 * min(1, t / 1000))SAM runs the existing gradient program twice per micro-batch, at ps and at ps + rho * g / ‖g‖, so it adds no gradient compile and leaves the optimizer program unchanged. The loss, layer state and train metrics come from the unperturbed pass, and the train line adds sam/gap. Under accum > 1 the perturbation is per micro-batch, which is m-sharpness with m the micro-batch size. Leaves no_decay excludes are not perturbed. Manual mode supports SingleGrad only.
Schedule binding: the grad namespace and bind
schedules gains two reserved namespaces. grad holds gradient rule fields such as SAM's rho. bind holds names of your choosing that match nothing: a Level 2 optimizer(e, group, hp) and the new grad_rule(e, hp) read them as hp.bind.<name> and place them. An unqualified key resolves to the one namespace it names; a key that names several is refused, and the error shows the qualified forms and bind.
Checkpoints: Checkpointer and selectors
Checkpointer(; keep) replaces TopKCheckpointer. keep is a set of selectors, Best(k), Last(n),Epochs(r) and Every(n), and the run keeps their union:
Checkpointer(; keep = (Best(3), Last(10)), metric = :val_loss, mode = :min)The default, (Best(3), Last(1)), is the previous behaviour. An epoch is written only if some selectorwould keep it.
TopKCheckpointer(; k, keep_latest, ...) is deprecated, warns once per session, and is supported until at least 1.0.0. It builds the equivalent Checkpointer, which writes identical files. TopKCheckpointer is now a function rather than a type, so code that dispatches on it or checks isa TopKCheckpointer must use Checkpointer, the type the default checkpointer now is. ckpt.k and ckpt.keep_latest remain readable.
Checkpoint averaging
weights takes an average of one run's checkpoints:
Nitro(e; weights = :latest => Average(Last(10)))
Nitro(e; weights = :latest => Average(Best(5; among = Last(20))))
average_checkpoints("runs/my_run", :latest => Average(Last(10)); save = "runs/my_run/avg10.jld2")Parameters are averaged uniformly; layer state comes from the best picked checkpoint when the selector ranks and the latest otherwise. restore_best = Average(...) ends train! holding the run's own average. An averaged record carries averaged_from (the files, epochs and weights it averages), shown by checkpoint_info and recorded in export provenance. Best(1) and Last(1) are the same sources as :bestand :latest. Averaging across runs is refused for now.
The Kaimon gate tools take the same sources as strings, written as Julia (weights = "latest => Average(Last(10))", restore_best = "Average(Last(10))"), parsed against a whitelist and never evaluated.
Other fixes
EarlyStoppingcounters are per handle and restored on resume: a second handle no longer inherits the first run's counters, and a resume continues its patience- A logger without
run_idorrun_urlno longer fails at the first checkpoint, and a stop fromrequest_stop!reaches the logger as:requested - An export echo whose name matches a
forwardoutput is refused instead of silently replacing it
Docs
A Checkpoints page; SAM on the Optimization page, including how it interacts with gradient accumulation; grad and bind on the Schedules page; and a precision pass over the guides and docstrings.
Tests
SAM is checked against a hand-computed step, and bitwise against the default rule at rho = 0 with dropout, RAdam and accum = 2, on one and two devices. TopKCheckpointer and the equivalent Checkpointer write identical files. Averages are checked against hand-computed means, and every selector round-trips through the string grammar.
v0.1.10
ReactantNitro v0.1.10
Fix: program outputs on a multi-device mesh
With n_devs > 1, validation, predict and some training paths failed when a value leaving a compiled program was either an input returned unchanged or a lazy reshape/view wrapper:
MethodError: no method matching (ConcretePJRTArray{T, N})(::Tuple{AsyncBuffer, AsyncBuffer}, ...)for an unchanged inputTypeError: in new, expected ConcretePJRTArray{T, N, 1}, got a value of type ConcretePJRTArray{T, N, 2}for a wrapper
Both come from Reactant's PJRT output codegen (EnzymeAD/Reactant.jl#3488 for the unchanged input, EnzymeAD/Reactant.jl#2989 for wrappers). ReactantNitro now works around them at its program boundary:
program_outmaterializesreshape/viewwrappers and passes every returned array throughoptimization_barrier, so each output leaves a program as a fresh array with its sharding intact.forwardoutputs in the eval program (validate,evaluate,predict) now go throughprogram_out/program_in, like layer state and metrics already did.
A computed output costs nothing; an input returned unchanged costs one device copy. Single-device runs are unaffected. A user struct other than a Tuple, NamedTuple, Pair or array wrapper still cannot leave a program on a mesh until EnzymeAD/Reactant.jl#2989 is fixed upstream.
Tests
multi_device outputs (two forced host devices) covers an input returned unchanged, a reshape of an input, reshape and view of a computed array, and a layer state array that never changes, through training with :host metrics, validation and predict, compared against n_devs = 1.
v0.1.9
ReactantNitro v0.1.9
New features
- Metric modes. A validation metric is now written
value => mode. The mode can be a count (summed, then divided at the end),:sum,:max,:minor:concat.:maxand:minare elementwise across batches (#41) :concatjoins a value along its last axis across the split, sofinalize_metricsgets one entry per sample. This makes rank-based metrics exact, such as AUROC, average precision, Spearman and quantiles. The last axis must equal the batch's real sample count, and the joined values are aligned across keys, though not in dataset order. A:concatarray thatfinalize_metricsreturns without reducing it is refused, naming its path (#41)- Groups. A key can hold a
NamedTupleorTupleof related values. A mode on the group covers every value inside it, and values in a group can have different modes:(; score = p => :concat, n_pos = count(y) => :sum).finalize_metricsreceives the same shape with the modes removed. A value with no mode or with two modes is refused, and the error names its path (#41) - Modes and groups work under both
:hostand:devicemetric residency (#41)
Bug fixes
- Training with
n_devs > 1failed in three places, and none of them had a test (#42):fire_monitorscalledrankon the device mesh- A
ReactantRNGreplicated on a mesh failed after the first gradient step - Metric pairs returned from the eval program on a mesh failed in the same way. RNG state and metric pairs now leave compiled programs as untyped surrogates and are rebuilt after the call
Behavior changes
(value, count)and(value, nothing)still work at the top of a key.nothingnow means:sum. Any mode outside the list above is refused on the first batch (#41)- A
ReactantRNGplaced on a mesh uses the PHILOX algorithm, because XLA's GPU SPMD partitioner miscompiles the defaultrng_bit_generator. The RNG algorithm is now part of the compile-cache key (#42)
Documentation
- The Metrics page is rewritten around
value => mode. It traces one split end to end and covers exact R² from sums, exact AUROC from:concat, binned AUROC from:sum, and which mode each kind of metric needs. The tutorial, binding sheet,metricsdocstring and MNIST example use the new syntax (#41)
Infrastructure
- Multi-device regression tests train on 2 forced CPU devices in a subprocess as part of the default suite. They cover dropout masks, including across a resume, and BatchNorm global statistics compared with
n_devs = 1(#42)
v0.1.8
ReactantNitro v0.1.8
New features
- TensorFlow SavedModel export.
export_model(n, TFSavedModel(); ...)writes a SavedModel for TF Serving and the platforms built on it (SageMaker, Vertex AI, KServe). Each traced program is oneXlaCallModuleop wrapping the StableHLO, the form JAX's native serialization produces. The backend is aPythonCallextension:using PythonCallwithtensorflowimportable enables it (#40) - One signature per entry of
batch_sizes(serving_b{N}, withserving_defaultthe first), all sharing one copy of the weights. Tensors are batch-first: the Julia shape(W, H, C, N)is the TF shape[N, C, H, W](#40) TFSavedModel(; platform, stablehlo_version, call_module_version). The defaults ("CUDA",v"1.5.0",9) load on TF 2.17 and 2.20;stablehlo_version = v"0.14.0"withcall_module_version = 8loads on TF 2.15. All three releases were verified by running every signature against the eager program (#40)- A program containing a
custom_callis refused, and so is a model declaringexport_client_outputs, since a SavedModel cannot shipmodel.jl. Any other postprocess is dropped with a warning. Provenance and the working-tree patch are written toassets.extra/(#40) compile_optionsnow checks the fields ofxla_debug_options,xla_executable_build_optionsandxla_compile_optionsagainst XLA's protos whenNitrois constructed, rather than failing at the first compile (#39)
Behavior changes
site_provenancenow has a default for every backend,ReactantNitro.git_provenance(root), which collects the commit, tree hash, branch, dirty flag, remote and the working-tree patch. Previously a backend without its own method raised an error whenprovenance_rootwas given.ReactantServerBundlestill usesReactantServerExport.collect_provenance(#40)
Documentation
- A Compile options page, including autotuning (#39)
- A TensorFlow SavedModel section in the Export guide (#40)
Dependencies
- Added
PythonCall = "0.9.25"as a weak dependency andDatesas a dependency (#40)
Infrastructure
- A separate SavedModel CI job installs TensorFlow 2.20 through CondaPkg and runs the end-to-end test in
test/savedmodel. It runs only when the export code changes and is not cached, so the Python environment never evicts the other jobs' caches (#40)
v0.1.7
ReactantNitro v0.1.7
New features
- Precise invalidation, on by default. Each compile now records every method specialization its trace actually went through, and the compile cache poisons an entry when any of them is redefined, deleted, or shadowed by a more specific method (for example from a package extension loaded after compiling). This covers dependencies the inference-based closure could not see: methods that dispatch only on traced types (such as LuxLib's traced
mean_var), Reactant overlays, callees behind runtime dispatch, and helpers inlined several calls deep. Drift reports name the method that changed. Reactant itself is not patched and installing the hook invalidates nothing; tracing time is unchanged within measurement noise (#38) - Set the
precise_invalidationpreference tofalseto fall back to the inference closure. The fallback is also used automatically, with one warning, if Reactant's internals ever stop matching what the hook expects (#38)
Bug fixes
- A program compiled in an older world than the latest (for example after
@evalfollowed by a compile in the same function) is now recognized as compiled against stale definitions and is not reused (#38)
Documentation
- Removed the claim that a custom ChainRules
rruleis a cache coverage gap. Reactant differentiates at the MLIR level through Enzyme-JAX and never consults ChainRules, so the backward pass has no Julia methods to guard (#38)
Dependencies
- Added
Preferences = "1"(#38)
Closed issues:
- Address method revise invalidation blind spots (#1)