Skip to content

contilab v0.2.0

Choose a tag to compare

@JonPark0 JonPark0 released this 31 Oct 13:36
· 61 commits to main since this release
e5eb510

contilab v0.2.0: metric fixes, a more robust runner, CI, validation-based hyperparameter selection, and teacher → student continual distillation. The CPU study is refreshed with all of these.

Highlights

  • Teacher → student continual distillation (consolidate:continual_distill). One teacher arrives per task and is dropped after use, so no expert is stored. With 1.8 MB of storage it beats Naive on every Class- and Domain-IL benchmark of the study.
  • Smaller students for both distillation methods (method.params.student). A student with 5.3× fewer parameters matches the full-size student on Split MNIST Class-IL.
  • Validation-based sweep selection (spec §5.3) and per-method grids (method_sweep).
  • New multi-teacher distillation defaults, chosen on validation accuracy: a TIES-merged student, routed targets and 10 epochs. Split MNIST Class-IL rises from 0.204 to 0.882, and distillation moves from last to second-best average rank. The v0.1 defaults under-trained the student.
  • CI on Python 3.11 and 3.13.

Full results: Results

Added

  • consolidate:continual_distill: teacher → student continual distillation. Teachers arrive one per task and
    are dropped after use; the student combines knowledge distillation with self-distillation on a replay buffer.
  • Smaller students for both distillation methods (method.params.student).
  • Validation-based selection of sweep variants (val_acc, selected column, ✓ in reports, ranking over
    selected variants).
  • method_sweep: one hyperparameter grid per method.
  • state_bytes metric. buffer_bytes now counts stored samples rather than index bytes.
  • Experiment configs cpu_tuned (validation-tuned comparison) and cpu_student (teacher → smaller student).
  • GitHub Actions test workflow.

Fixed

  • cka_final no longer averages in the trivial comparison of the last task with itself.
  • Joint runs record continual-evaluation accuracy on every task, not only the first.
  • Few-shot recovery fine-tunes with the run's optimizer instead of SGD.
  • A resumed run keeps the divergence threshold of the uninterrupted run.
  • torch.compile errors fall back to eager at any recompilation, not only at the first call.
  • Checkpoints, caches, raw logs and run files are written atomically. An expert cache without its metadata file is
    rebuilt.
  • Expert training and distillation are checked for divergence, and affected seeds are marked as diverged.
  • contilab run exp.yaml --root runs key=value works on Python < 3.12. Before, argparse left the override
    unparsed and the command exited.

Changed

  • Multi-teacher distillation defaults (consolidate.init: ties, routing: routed, epochs: 10), chosen on
    validation accuracy in cpu_tuned. The v0.1 defaults (base, entropy, 1 epoch) under-trained the student.
  • Run ids hash the full resolved configuration, so runs made with v0.1 defaults are not reused when defaults
    change.

Install

pip install https://github.com/JonPark0/contilab/releases/download/v0.2.0/contilab-0.2.0-py3-none-any.whl
# or from source
git clone --branch v0.2.0 https://github.com/JonPark0/contilab && pip install -e ./contilab