Repository navigation
The first stable release. A base model can now be benchmarked the way published models are, with scores that match EleutherAI's lm-evaluation-harness, and a review of the training code fixed ten bugs that gave quietly wrong results. The repo also has a Docker image, and the test suite now passes with a GPU visible.
Compatibility notes, which are why this is 1.0.0:
- Mixture of Experts with
moe_top_k=1now weighs the chosen expert by its router probability instead of exactly 1, as the Switch Transformer does. A one-expert-per-token checkpoint from 0.2.0 gives different outputs; two or more experts per token are unchanged. - The dev loss of MoE models no longer includes the balancing term, so it reads lower than on 0.2.0 and now compares with dense models.
- SFT with
grad_accumabove 1 trains on the true full-batch gradient, so its loss curves differ from 0.2.0's. - GSM8K scores can move slightly, because the answer checker now reads
.5,10-15and5 + 3 = 8correctly. - The loader shuffles with its own seed, so a seeded run sees its data in a different order than on 0.2.0, and the DDP ranks of
pretrain_base.pynow split each batch.
New
- Base-model benchmarks.
scripts/eval_base.pyscores ARC-Easy, ARC-Challenge and LAMBADA on any checkpoint with lm-evaluation-harness's prompts, rules and metrics. GPT-2 124M on this code: ARC-Easy 43.81 / 39.48, ARC-Challenge 19.03 / 22.70 (acc / acc_norm), LAMBADA 32.56% at perplexity 40.06. lm-evaluation-harness 0.4.13 gives exactly these figures: each of the 19,341 answer log-likelihoods agrees within 0.0006, and no prediction differs. HellaSwag is left out on purpose: its source repository was taken down by a DMCA notice in September 2026. - A Docker image for training and for running the checks on a GPU (#43 by @LLiuJJ), with a
TORCH_INDEXbuild argument for drivers older than CUDA 13. E2E_DEVICE=cudaruns the end-to-end test of every training stage on a GPU.
Fixed
- A resumed run trained on its first batches again instead of continuing the data, in both trainers. The DDP ranks of
pretrain_base.pytrained on overlapping windows, and its final checkpoint recorded one step too many. - With one expert per token, the MoE router got no gradient from the language-modeling loss, and MoE dev loss included the balancing term.
- SFT's gradient accumulation weighed tokens unevenly: on a short and a long answer its gradient had a cosine of only 0.66 with the true one.
- The GSM8K checker, which also rewards the RL stages, marked some wrong answers correct:
.5read as 5,COVID-19as -19, and5 + 3 = 8as 5. - PPO's
ent_coefdid nothing; it now adds the entropy bonus. --max_tokenswas ignored below 8M tokens, anddata_download.pykept failed or interrupted downloads, which every later run skipped.- The test suite failed on any machine with a GPU (found in #43).
- The requirements files pointed at CUDA wheel indexes that stopped at torch 2.7.1 and 2.5.1, and installed
torchvisionandtorchaudio, which nothing used.
Verification
- 220 tests, ruff and mypy pass on Linux (Python 3.10 and 3.13), Windows and macOS in CI. Every fix to the code comes with a test that fails on 0.2.0.
- On an RTX A4000 (driver 570, torch 2.14.1 with CUDA 12.6): the full suite with the GPU visible, every training stage end to end on CUDA for both architectures,
torch.compilewith bf16, resuming and extending a run, the benchmarks, and the Docker image (build, GPU access, ruff, mypy and the full suite inside it).
Thanks to @LLiuJJ, whose Docker PR found the GPU test bug.
Full log in CHANGELOG.md. Diff: v0.2.0...v1.0.0.