Metrics

Metrics

contilab computes metrics after training from raw data stored per run, so a metric can be added later with contilab metrics <exp> --add <name>. A metric that is undefined for a run returns no value with a reason (for example "needs at least 2 tasks") instead of a misleading number.

Notation: $T$ tasks; $R_{i,j}$ = test accuracy on task $j$ after training task $i$; $b_j$ = accuracy of the initial model on task $j$.

Raw data and tiers

Tier	Raw records collected
core	accuracy matrix, initial accuracy, predictions/confidences per task end, cost
extended	+ continual-evaluation stream (every ρ steps), learning curves (first β batches), probe features, neuron activations, LM perplexity
full	+ parameter snapshots, loss-interpolation curves, gradient-cosine matrices, few-shot recovery

Accuracy-matrix metrics (core)

Metric	Definition	Reference
acc	$\frac{1}{T}\sum_j R_{T-1,j}$	Lopez-Paz & Ranzato 2017
aia	$\frac{1}{T}\sum_i \frac{1}{i+1}\sum_{j\le i} R_{i,j}$ (average incremental accuracy)	Rebuffi et al. 2017
bwt	$\frac{1}{T-1}\sum_{j<T-1}(R_{T-1,j}-R_{j,j})$	Lopez-Paz & Ranzato 2017
fwt	$\frac{1}{T-1}\sum_{j\ge1}(R_{j-1,j}-b_j)$ — undefined in Class-IL, where a task's classes are masked out before it is learned	Lopez-Paz & Ranzato 2017
forgetting	mean of $f_j=\max_{l\in[j,T-2]}R_{l,j}-R_{T-1,j}$	Chaudhry et al. 2018
relative_forgetting	mean of $f_j / \max_l R_{l,j}$	—
learning_acc	$\frac{1}{T}\sum_j R_{j,j}$ (plasticity)	Riemer et al. 2019
intransigence	$\frac{1}{T}\sum_j(a^j-R{j,j})$ with $a^$ from the Joint run of the same preset and seed	Chaudhry et al. 2018
val_acc	final average accuracy on the validation splits (10 % of each task's training data): the score for selecting sweep variants, never a reported result	—

Continual evaluation (extended)

$A(k,n)$ = accuracy on task $k$ at step $n$, sampled every ρ steps on a fixed subset.

Metric	Definition	Reference
min_acc	mean over $k<T-1$ of the minimum $A(k,n)$ after task $k$ was learned	De Lange et al. 2023
wc_acc	$\frac{1}{T}A(T-1,\text{end}) + (1-\frac{1}{T}),\text{min-ACC}$	De Lange et al. 2023
stability_gap	accuracy of task $k$ at the end of task $k$ minus its minimum while task $k+1$ trains, averaged	De Lange et al. 2023
windowed_forgetting	largest drop within a window Δ = 10 % of a task's length; undefined when Δ < ρ	De Lange et al. 2023
aaa	average over evaluation points of the mean accuracy on seen tasks (anytime accuracy)	Caccia et al. 2022
lca	$\frac{1}{\beta+1}\sum_{b=0}^{\beta}Z_b$, accuracy on the current task during its first β batches	Chaudhry et al. 2019

Representation (extended)

Metric	Definition	Reference
probe_acc	matrix $P_{i,j}$ of linear-probe accuracy on frozen features	Davari et al. 2022
repr_forgetting / head_forgetting	probe drop $P_{j,j}-P_{T-1,j}$, and the rest of the accuracy drop attributed to the classifier head	Davari et al. 2022
cka_final, cka_matrix	linear CKA between features of the same samples at different times; cka_final averages $\mathrm{CKA}(F_{j,j}, F_{T-1,j})$ over $j<T-1$ (undefined for a single task, e.g. Joint)	Kornblith et al. 2019
feature_drift	mean normalised L2 movement of a task's features from when it was learned to the end	Ramasesh et al. 2021

Plasticity loss (extended, full)

Metric	Definition	Reference
dormant_ratio (+ curve)	share of neurons with normalised mean activation $\le\tau$ (τ = 0.025)	Sokar et al. 2023
effective_rank, srank (+ curve)	exp-entropy of normalised singular values; smallest k covering 99 % of singular mass	Roy & Vetterli 2007; Kumar et al. 2021
weight_norms	per-layer weight norm after each task	Dohare et al. 2024

Structural plasticity diagnostics are symptoms rather than guarantees; report them next to learning_acc and lca.

Bias and calibration (core)

Metric	Definition	Reference
recency_bias	Class-IL: share of old-task samples predicted as the newest task's classes	Wu et al. 2019; Masana et al. 2022
task_confusion	true-task × predicted-task matrix	Masana et al. 2022
ece, ece_per_task	expected calibration error, 15 bins	Guo et al. 2017

Recovery, landscape, LLMs (full, extended)

Metric	Definition	Reference
recovery	$(acc_{after}-acc_{before})/(R_{j,j}-acc_{before})$ after 20 SGD steps on 50 samples of an old task: high recovery suggests spurious forgetting	Zheng et al., ICLR 2025
loss_barrier	$\max_\alpha L((1-\alpha)\theta_k+\alpha\theta_{k+1}) - [(1-\alpha)L(\theta_k)+\alpha L(\theta_{k+1})]$ on task $k$	Mirzadeh et al. 2020
grad_interference	mean off-diagonal cosine between task gradients	—
general_ppl_delta (+ curve)	WikiText perplexity of the original LM head after training minus before (general-ability retention)	Wang et al. 2023 (TRACE)

Cost (core)

wall_time, samples_per_s, peak_mem (per run group), trainable_params, buffer_bytes, state_bytes, model_bytes, total_flops (measured with FlopCounterMode per member, including replay, reference gradients, experts and distillation).

• buffer_bytes is the memory a buffer of stored samples would take: items × (input + label + attention mask bytes, plus per-item extras such as DER++ logits). contilab stores only dataset indices internally, so index bytes are not counted. The transfer set of buffer / full distillation is counted too.
• state_bytes is the extra strategy state kept between tasks, e.g. the EWC Fisher diagonal and anchor parameters or the LwF teacher copy.

Statistics

• mean ± std over seeds, bootstrap 95 % confidence intervals (10,000 resamples)
• paired test against a baseline method: Wilcoxon signed-rank for n ≥ 6 seeds, otherwise an exact sign-flip permutation test (with 3 seeds the smallest possible p is 0.25, so the report warns about statistical power)
• across benchmarks: average ranks, Friedman test, Nemenyi critical difference (Demšar 2006)

Figures

Accuracy-matrix heatmaps, continual-evaluation curves with task boundaries, task-confusion matrices, CKA heatmaps, per-layer weight drift, gradient-interference heatmaps, PCA and t-SNE of features, loss-interpolation curves, critical-difference diagrams and throughput bars.