Repository navigation
contilab v0.2.0
contilab v0.2.0: metric fixes, a more robust runner, CI, validation-based hyperparameter selection, and teacher → student continual distillation. The CPU study is refreshed with all of these.
Highlights
- Teacher → student continual distillation (
consolidate:continual_distill). One teacher arrives per task and is dropped after use, so no expert is stored. With 1.8 MB of storage it beats Naive on every Class- and Domain-IL benchmark of the study. - Smaller students for both distillation methods (
method.params.student). A student with 5.3× fewer parameters matches the full-size student on Split MNIST Class-IL. - Validation-based sweep selection (spec §5.3) and per-method grids (
method_sweep). - New multi-teacher distillation defaults, chosen on validation accuracy: a TIES-merged student, routed targets and 10 epochs. Split MNIST Class-IL rises from 0.204 to 0.882, and distillation moves from last to second-best average rank. The v0.1 defaults under-trained the student.
- CI on Python 3.11 and 3.13.
Full results: Results
Added
consolidate:continual_distill: teacher → student continual distillation. Teachers arrive one per task and
are dropped after use; the student combines knowledge distillation with self-distillation on a replay buffer.- Smaller students for both distillation methods (
method.params.student). - Validation-based selection of sweep variants (
val_acc,selectedcolumn, ✓ in reports, ranking over
selected variants). method_sweep: one hyperparameter grid per method.state_bytesmetric.buffer_bytesnow counts stored samples rather than index bytes.- Experiment configs
cpu_tuned(validation-tuned comparison) andcpu_student(teacher → smaller student). - GitHub Actions test workflow.
Fixed
cka_finalno longer averages in the trivial comparison of the last task with itself.- Joint runs record continual-evaluation accuracy on every task, not only the first.
- Few-shot recovery fine-tunes with the run's optimizer instead of SGD.
- A resumed run keeps the divergence threshold of the uninterrupted run.
torch.compileerrors fall back to eager at any recompilation, not only at the first call.- Checkpoints, caches, raw logs and run files are written atomically. An expert cache without its metadata file is
rebuilt. - Expert training and distillation are checked for divergence, and affected seeds are marked as diverged.
contilab run exp.yaml --root runs key=valueworks on Python < 3.12. Before, argparse left the override
unparsed and the command exited.
Changed
- Multi-teacher distillation defaults (
consolidate.init: ties,routing: routed,epochs: 10), chosen on
validation accuracy incpu_tuned. The v0.1 defaults (base,entropy, 1 epoch) under-trained the student. - Run ids hash the full resolved configuration, so runs made with v0.1 defaults are not reused when defaults
change.
Install
pip install https://github.com/JonPark0/contilab/releases/download/v0.2.0/contilab-0.2.0-py3-none-any.whl
# or from source
git clone --branch v0.2.0 https://github.com/JonPark0/contilab && pip install -e ./contilab