Skip to content

v1.0.0

Latest

Choose a tag to compare

@FareedKhan-dev FareedKhan-dev released this 16 Nov 05:15

The first stable release. A base model can now be benchmarked the way published models are, with scores that match EleutherAI's lm-evaluation-harness, and a review of the training code fixed ten bugs that gave quietly wrong results. The repo also has a Docker image, and the test suite now passes with a GPU visible.

Compatibility notes, which are why this is 1.0.0:

  • Mixture of Experts with moe_top_k=1 now weighs the chosen expert by its router probability instead of exactly 1, as the Switch Transformer does. A one-expert-per-token checkpoint from 0.2.0 gives different outputs; two or more experts per token are unchanged.
  • The dev loss of MoE models no longer includes the balancing term, so it reads lower than on 0.2.0 and now compares with dense models.
  • SFT with grad_accum above 1 trains on the true full-batch gradient, so its loss curves differ from 0.2.0's.
  • GSM8K scores can move slightly, because the answer checker now reads .5, 10-15 and 5 + 3 = 8 correctly.
  • The loader shuffles with its own seed, so a seeded run sees its data in a different order than on 0.2.0, and the DDP ranks of pretrain_base.py now split each batch.

New

  • Base-model benchmarks. scripts/eval_base.py scores ARC-Easy, ARC-Challenge and LAMBADA on any checkpoint with lm-evaluation-harness's prompts, rules and metrics. GPT-2 124M on this code: ARC-Easy 43.81 / 39.48, ARC-Challenge 19.03 / 22.70 (acc / acc_norm), LAMBADA 32.56% at perplexity 40.06. lm-evaluation-harness 0.4.13 gives exactly these figures: each of the 19,341 answer log-likelihoods agrees within 0.0006, and no prediction differs. HellaSwag is left out on purpose: its source repository was taken down by a DMCA notice in September 2026.
  • A Docker image for training and for running the checks on a GPU (#43 by @LLiuJJ), with a TORCH_INDEX build argument for drivers older than CUDA 13.
  • E2E_DEVICE=cuda runs the end-to-end test of every training stage on a GPU.

Fixed

  • A resumed run trained on its first batches again instead of continuing the data, in both trainers. The DDP ranks of pretrain_base.py trained on overlapping windows, and its final checkpoint recorded one step too many.
  • With one expert per token, the MoE router got no gradient from the language-modeling loss, and MoE dev loss included the balancing term.
  • SFT's gradient accumulation weighed tokens unevenly: on a short and a long answer its gradient had a cosine of only 0.66 with the true one.
  • The GSM8K checker, which also rewards the RL stages, marked some wrong answers correct: .5 read as 5, COVID-19 as -19, and 5 + 3 = 8 as 5.
  • PPO's ent_coef did nothing; it now adds the entropy bonus.
  • --max_tokens was ignored below 8M tokens, and data_download.py kept failed or interrupted downloads, which every later run skipped.
  • The test suite failed on any machine with a GPU (found in #43).
  • The requirements files pointed at CUDA wheel indexes that stopped at torch 2.7.1 and 2.5.1, and installed torchvision and torchaudio, which nothing used.

Verification

  • 220 tests, ruff and mypy pass on Linux (Python 3.10 and 3.13), Windows and macOS in CI. Every fix to the code comes with a test that fails on 0.2.0.
  • On an RTX A4000 (driver 570, torch 2.14.1 with CUDA 12.6): the full suite with the GPU visible, every training stage end to end on CUDA for both architectures, torch.compile with bf16, resuming and extending a run, the benchmarks, and the Docker image (build, GPU access, ruff, mypy and the full suite inside it).

Thanks to @LLiuJJ, whose Docker PR found the GPU test bug.

Full log in CHANGELOG.md. Diff: v0.2.0...v1.0.0.