# Metrics

contilab computes metrics **after** training from raw data stored per run, so a metric can be added later with
`contilab metrics <exp> --add <name>`. A metric that is undefined for a run returns no value with a reason (for
example "needs at least 2 tasks") instead of a misleading number.

Notation: $T$ tasks; $R_{i,j}$ = test accuracy on task $j$ after training task $i$; $b_j$ = accuracy of the
initial model on task $j$.

## Raw data and tiers

| Tier | Raw records collected |
|---|---|
| `core` | accuracy matrix, initial accuracy, predictions/confidences per task end, cost |
| `extended` | + continual-evaluation stream (every ρ steps), learning curves (first β batches), probe features, neuron activations, LM perplexity |
| `full` | + parameter snapshots, loss-interpolation curves, gradient-cosine matrices, few-shot recovery |

## Accuracy-matrix metrics (`core`)

| Metric | Definition | Reference |
|---|---|---|
| `acc` | $\frac{1}{T}\sum_j R_{T-1,j}$ | Lopez-Paz & Ranzato 2017 |
| `aia` | $\frac{1}{T}\sum_i \frac{1}{i+1}\sum_{j\le i} R_{i,j}$ (average incremental accuracy) | Rebuffi et al. 2017 |
| `bwt` | $\frac{1}{T-1}\sum_{j<T-1}(R_{T-1,j}-R_{j,j})$ | Lopez-Paz & Ranzato 2017 |
| `fwt` | $\frac{1}{T-1}\sum_{j\ge1}(R_{j-1,j}-b_j)$ — undefined in Class-IL, where a task's classes are masked out before it is learned | Lopez-Paz & Ranzato 2017 |
| `forgetting` | mean of $f_j=\max_{l\in[j,T-2]}R_{l,j}-R_{T-1,j}$ | Chaudhry et al. 2018 |
| `relative_forgetting` | mean of $f_j / \max_l R_{l,j}$ | — |
| `learning_acc` | $\frac{1}{T}\sum_j R_{j,j}$ (plasticity) | Riemer et al. 2019 |
| `intransigence` | $\frac{1}{T}\sum_j(a^*_j-R_{j,j})$ with $a^*$ from the Joint run of the same preset and seed | Chaudhry et al. 2018 |
| `val_acc` | final average accuracy on the validation splits (10 % of each task's training data): the score for selecting sweep variants, never a reported result | — |

## Continual evaluation (`extended`)

$A(k,n)$ = accuracy on task $k$ at step $n$, sampled every ρ steps on a fixed subset.

| Metric | Definition | Reference |
|---|---|---|
| `min_acc` | mean over $k<T-1$ of the minimum $A(k,n)$ after task $k$ was learned | De Lange et al. 2023 |
| `wc_acc` | $\frac{1}{T}A(T-1,\text{end}) + (1-\frac{1}{T})\,\text{min-ACC}$ | De Lange et al. 2023 |
| `stability_gap` | accuracy of task $k$ at the end of task $k$ minus its minimum while task $k+1$ trains, averaged | De Lange et al. 2023 |
| `windowed_forgetting` | largest drop within a window Δ = 10 % of a task's length; undefined when Δ < ρ | De Lange et al. 2023 |
| `aaa` | average over evaluation points of the mean accuracy on seen tasks (anytime accuracy) | Caccia et al. 2022 |
| `lca` | $\frac{1}{\beta+1}\sum_{b=0}^{\beta}Z_b$, accuracy on the current task during its first β batches | Chaudhry et al. 2019 |

## Representation (`extended`)

| Metric | Definition | Reference |
|---|---|---|
| `probe_acc` | matrix $P_{i,j}$ of linear-probe accuracy on frozen features | Davari et al. 2022 |
| `repr_forgetting` / `head_forgetting` | probe drop $P_{j,j}-P_{T-1,j}$, and the rest of the accuracy drop attributed to the classifier head | Davari et al. 2022 |
| `cka_final`, `cka_matrix` | linear CKA between features of the same samples at different times; `cka_final` averages $\mathrm{CKA}(F_{j,j}, F_{T-1,j})$ over $j<T-1$ (undefined for a single task, e.g. Joint) | Kornblith et al. 2019 |
| `feature_drift` | mean normalised L2 movement of a task's features from when it was learned to the end | Ramasesh et al. 2021 |

## Plasticity loss (`extended`, `full`)

| Metric | Definition | Reference |
|---|---|---|
| `dormant_ratio` (+ curve) | share of neurons with normalised mean activation $\le\tau$ (τ = 0.025) | Sokar et al. 2023 |
| `effective_rank`, `srank` (+ curve) | exp-entropy of normalised singular values; smallest k covering 99 % of singular mass | Roy & Vetterli 2007; Kumar et al. 2021 |
| `weight_norms` | per-layer weight norm after each task | Dohare et al. 2024 |

Structural plasticity diagnostics are symptoms rather than guarantees; report them next to `learning_acc` and
`lca`.

## Bias and calibration (`core`)

| Metric | Definition | Reference |
|---|---|---|
| `recency_bias` | Class-IL: share of old-task samples predicted as the newest task's classes | Wu et al. 2019; Masana et al. 2022 |
| `task_confusion` | true-task × predicted-task matrix | Masana et al. 2022 |
| `ece`, `ece_per_task` | expected calibration error, 15 bins | Guo et al. 2017 |

## Recovery, landscape, LLMs (`full`, `extended`)

| Metric | Definition | Reference |
|---|---|---|
| `recovery` | $(acc_{after}-acc_{before})/(R_{j,j}-acc_{before})$ after 20 SGD steps on 50 samples of an old task: high recovery suggests *spurious* forgetting | Zheng et al., ICLR 2025 |
| `loss_barrier` | $\max_\alpha L((1-\alpha)\theta_k+\alpha\theta_{k+1}) - [(1-\alpha)L(\theta_k)+\alpha L(\theta_{k+1})]$ on task $k$ | Mirzadeh et al. 2020 |
| `grad_interference` | mean off-diagonal cosine between task gradients | — |
| `general_ppl_delta` (+ curve) | WikiText perplexity of the original LM head after training minus before (general-ability retention) | Wang et al. 2023 (TRACE) |

## Cost (`core`)

`wall_time`, `samples_per_s`, `peak_mem` (per run group), `trainable_params`, `buffer_bytes`, `state_bytes`,
`model_bytes`, `total_flops` (measured with `FlopCounterMode` per member, including replay, reference gradients,
experts and distillation).

- `buffer_bytes` is the memory a buffer of stored samples would take: items × (input + label + attention mask
  bytes, plus per-item extras such as DER++ logits). contilab stores only dataset indices internally, so index
  bytes are not counted. The transfer set of `buffer` / `full` distillation is counted too.
- `state_bytes` is the extra strategy state kept between tasks, e.g. the EWC Fisher diagonal and anchor
  parameters or the LwF teacher copy.

## Statistics

- mean ± std over seeds, bootstrap 95 % confidence intervals (10,000 resamples)
- paired test against a baseline method: Wilcoxon signed-rank for n ≥ 6 seeds, otherwise an exact sign-flip
  permutation test (with 3 seeds the smallest possible p is 0.25, so the report warns about statistical power)
- across benchmarks: average ranks, Friedman test, Nemenyi critical difference (Demšar 2006)

## Figures

Accuracy-matrix heatmaps, continual-evaluation curves with task boundaries, task-confusion matrices, CKA
heatmaps, per-layer weight drift, gradient-interference heatmaps, PCA and t-SNE of features, loss-interpolation
curves, critical-difference diagrams and throughput bars.