Skip to content

v0.3.1a0 (beta): cluster-locality default-on, adaptive expert paging

Pre-release
Pre-release

Choose a tag to compare

@FonaTech FonaTech released this 07 Nov 02:22

Beta (pre-release) of 0.3.1, core half. main stays at 0.3.0; this build is from dev. The rewritten Web UI follows in 0.3.1a1.

Install: pip install --pre project_chronos==0.3.1a0

Changed

  • Cluster-locality training on by default for new runs: train_chronos.py, the planner/presets and chronos bundle finetune set lambda_cluster_locality = 0.3; the window is the power of two nearest the median SFT record length (clamped to [16, seq_len]). Turn it off with --lambda_cluster_locality 0 / --no-cluster-locality. The ChronosConfig class default stays 0, so stored recipes resume unchanged and older checkpoints load as λ = 0.
  • Inference adapts to the checkpoint (paging=auto): weights trained with the constraint are paged one training cluster per SSD read, others one expert per read. cluster / expert force a layout; each layout has its own cache directory.

Added

  • chronos/lazy_experts.py: one lazy-expert implementation shared by the Web UI, bundle and chronos serve.
  • Next-use residency planner for MLX lazy experts; decay and prefetch waits are configurable (UI, bundle CLI, generate()).
  • chronos bundle translate|eval --lazy-experts --paging --residency --decay --ram-ratio --decode-wait --verify-wait; chronos bundle finetune --[no-]cluster-locality --lambda-cluster --cluster-window --clusters.
  • chronos serve --paging; /api/ps reports the resolved layout.

Evidence and limits

  • λ = 0.3 comes from a measured sweep on a tiny model with synthetic 4-topic streams (300 steps, from scratch): cluster reads per token 1.48 → 0.023, held-out CE 1.703 → 1.712. Its effect on real data and on fine-tuning previously unconstrained weights has not been measured.
  • Residency planner on the 50M s13996 checkpoint (16 FLORES+ dev prompts, greedy, CPU): −35 % synchronous SSD loads (plain), −33 % (speculative); outputs identical. That checkpoint was trained without the constraint, so it pages per expert.
  • Lazy paging is exact against full-memory greedy in the tests (MLX planned/window/speculative; Torch with sync_on_demand).
  • Weights trained on data with unknown licences are research-only and are not part of this release.

Full list: CHANGELOG.md.