BitBop — latent-free ternary training

BitBop trains ternary language models — every weight is in {-1, 0, +1} — without keeping full-precision latent weights in the ternary body. Instead of a shadow float copy per weight (the STE recipe), the only per-weight training state is a small bf16 flip momentum (the tied token embedding and output head stay float with Adam — a small fraction of the parameters). That makes the optimizer state ~5× smaller, so larger models fit in the same GPU memory, and the shipped weights are int2-packed (2 bits/weight stored; ~1.3 bits of entropy at ~63% sparsity, below BitNet's 1.58).

This repository is the reference implementation and the reproduction kit for the technical report. It is a proof of concept, trained on a single RTX 3060 (6 GB) and an Apple M4 — not a production model.

A 325M-class model trains within a single 6 GB GPU only with BitBop; the STE and float baselines spill or do not complete (see §R2 of the report).

What we claim (and what we don't)

	Claim	Scope
Quality	BitBop beats STE — the fair, same-family paired comparison (CI-backed, up to 91M) — and is competitive with an untuned same-recipe float at a matched training budget. On BabyLM a 125M ternary model reaches parity with a same-data GPT-2‑124M (float): 68.0 vs 67.9 BLiMP, same harness — but single-seed, and the 0.16-pt margin is inside the measured seed spread (0.11), so a tie, not a win.	budget-matched, ≤125M; not asymptotic
Memory	~2.3 bytes/param of optimizer state (ternary body) vs 6 (Adam8bit) / 12 (AdamW). A 325M-class model trains in 6 GB — only BitBop completes a full step there (the 5× and the 6 GB fit are measured on the full model, float embed/head included).	training-state memory
Inference	int2-packed weights, 2 bpw stored (~1.3 bpw entropy at ~63% sparsity), ~16× smaller than fp32. Peak RAM/speed unchanged without a dedicated int2 kernel.	storage only
Speed	Not a claim. There is no ad-hoc kernel yet, so no wall-clock speedup. Ternary weights do remove the multiplies (adds/subtracts only) — the potential is there, the kernel is future work.	—

We are not claiming state of the art, a speedup, or anything above 325M-class.

On the baselines. All three variants (bop / ste / float) share a single recipe — thresholds, schedule and batch were tuned for BitBop; lr 1e-3, no warmup, no gradient clipping — and the float and STE baselines are not separately tuned. The comparison isolates the training algorithm at fixed hyperparameters; it is not evidence that BitBop beats a well-tuned float, which would likely be stronger. The robust, statistically-backed result is BitBop > STE (same ternary family, same recipe).

Results (all same-harness, our validated BLiMP harness)

BLiMP macro-accuracy (67 phenomena × 1000 minimal pairs):

Model	Params	Training data	BLiMP
BitBop (ternary)	125M	BabyLM strict-small (~10M words)	68.0
BitBop (ternary)	50M	BabyLM strict-small	66.8
BitBop (ternary)	25M	BabyLM strict-small	66.0
STE ternary (baseline)	25M	BabyLM strict-small	63.2
GPT-2‑124M (float)	124M	BabyLM 10M (same data)	67.9
BabyLlama‑58M (distilled)	58M	BabyLM 10M	68.9
GPT-2‑124M (float, data-rich)	124M	WebText (≫ data)	82.5

The data-rich GPT-2 (82.5) is the data ceiling we do not reach. The point is that a ternary model at the same data budget reaches parity within noise with a float GPT-2 of the same size (see the tie caveat above).

On TinyStories (internal, budget-matched), BitBop-91M reaches 1.49 validation nats vs STE 1.58 and float 1.59 — sample efficiency at a matched, under-trained budget (~1 token/param, far below compute-optimal), not an asymptotic result. The STE gap is the fair one (same recipe, same family); the float number uses BitBop's untuned recipe and should not be read as beating float. See the technical report.

Quickstart (~15 min on CPU, a few minutes on GPU)

Reproduce the headline BabyLM comparison from published weights:

pip install -r requirements.txt
bash scripts/prepare_data.sh          # downloads BLiMP
bash scripts/reproduce_babylm.sh      # downloads BitBop-125M + baselines, runs BLiMP

This downloads Feyd89/BitBop-125M-babylm from the Hugging Face Hub, evaluates it and the public baselines (GPT-2, BabyLlama) with the same harness, and prints the table above.

How it works

In five lines:

• Weights are ternary, stored 2-bit packed (4 trits/byte), unpacked just-in-time for the matmul.
• The only per-weight state is m, an EMA of the negative gradient (bf16).
• A trit flips when m crosses a threshold set from the per-row RMS of m — easy to fall into 0 (tau_in), harder to leave it (tau_out), which keeps the weights sparse.
• beta (the EMA rate) follows a one-decade decay schedule; m resets on flipped entries.
• No latent float weights are ever stored. Details and the full recipe in the technical report.

Train it yourself

# full reproduction of the 25M BabyLM point from scratch (~3h on a 6 GB GPU)
bash scripts/train_babylm_25m.sh
# or drive the trainer directly
python train.py --config configs/91m_bop.json --device cuda

Configs for every reported run are in configs/ (bop / ste / float, 91M TinyStories and the 25M/50M/125M BabyLM ladder).

Weights

On the Hugging Face Hub (MIT):

• Feyd89/BitBop-91M — TinyStories, the crossover headline
• Feyd89/BitBop-125M-babylm — BabyLM, BLiMP 68.0

STE and float baselines are reproducible from configs/ (weights not shipped).

Layout

bitbop/kernel.py    int2 packing + BopTernaryLinear/Int2Lin + ScheduledBop (the flip optimizer)
bitbop/model.py     LMComposite (SWA body + full-attention layers, NoPE, tied interface)
bitbop/data.py      BPE tokenizer + packed .bin loader
train.py           one driver for bop | ste | float
eval_blimp.py      BLiMP harness (+ HF branch for external baselines)
configs/           one JSON per reported run
scripts/           data prep + reproduction
docs/BitBop-technical-report.pdf   reference technical report (PDF)

Citing / data

BitBop is MIT-licensed. It builds on public datasets and benchmarks — please cite their authors: TinyStories, the BabyLM Challenge (2024), and BLiMP. BabyLM data is not redistributed here; scripts/prepare_data.sh points you to the source.

Limitations

Budget-scoped quality claims only; ≤325M-class; single corpus per experiment; one author with an AI-assisted workflow. No speed claim without a dedicated kernel. See the report for the full "tried and didn't work" table and the ongoing directions.