Benchmarks

Benchmarks

Name	Tasks	Scenarios	Source
split_mnist	5 × 2 classes	Task-IL, Class-IL	MNIST (Keras mirror on GCS)
permuted_mnist	10 pixel permutations, 10 shared classes	Domain-IL	MNIST
split_cifar10	5 × 2 classes	Task-IL, Class-IL	CIFAR-10 (fast.ai mirror)
split_cifar100	10 × 10 classes	Task-IL, Class-IL	CIFAR-100 (fast.ai mirror)
text5	AG News (4) → Yelp (5) → Amazon (5) → DBpedia (14) → Yahoo Answers (10)	Class-IL, Task-IL	Zhang et al. 2015 CSVs (fast.ai mirror)
synthetic, synthetic_text	Gaussian clusters / marker-token sequences	all	generated (smoke tests)

• text5 follows de Masson d'Autume et al. (2019): Yelp and Amazon share the five rating labels, giving 33 classes. Texts are tokenised once with the model's tokenizer and cached; language-model retention is measured on WikiText-2.
• 10 % of each task's training data is held out as a validation split (used for merge coefficients and sweeps).
• task_order_seed shuffles task order.

Profiles

Preset	cpu profile	gpu profile
Split MNIST	full data, MLP 2×400, 1 epoch	same
Permuted MNIST	10 tasks, 10k train / 2k test per task	full data
Split CIFAR-10	full data, reduced ResNet-18, 2 epochs	3 epochs
Split CIFAR-100	full data, reduced ResNet-18, 2 epochs	3 epochs
text5	Pythia-70M, 1,000 train / 200 test per task	Qwen2.5-1.5B, 115k train / 7.6k test per task

Models

• mlp — MLP (default 2 × 400 ReLU)
• resnet18r — reduced ResNet-18 (nf = 20, Lopez-Paz & Ranzato 2017) with GroupNorm (vmap-safe)
• hf — any Hugging Face causal LM; classification uses the last non-padding token; the original LM head is kept frozen for perplexity measurements