Phase 0.5b — bf16 Mixed Precision: 3.8× Speedup, Same Quality

[bitzic]

Date: May 27, 2026
Hardware: NVIDIA H100 PCIe 80GB (Shadeform / ShadeCloud)
Code: KarpathianBase/karpathian @ v0.5.1

---

What changed

One code change: use_bf16=True in recipe/train.py. PyTorch torch.amp.autocast with bfloat16 on the H100's tensor cores. Everything else identical — same model, same data, same seed, same config.

fp32 vs bf16 — head-to-head

Metric	fp32 (Phase 0.5)	bf16 (Phase 0.5b)	Change
Final loss	3.8173	3.8163	-0.001 (identical)
Throughput	16,882 tok/s	63,361 tok/s	3.8×
Wall-clock	258.8 min	69.0 min	3.8× faster
Tokens trained	262,144,000	262,144,000	same
Parameters	253,872,128	253,872,128	same
Seed	1337	1337	same
Data	FineWeb-Edu 1B tokens	FineWeb-Edu 1B tokens	same
Config	h100_default.json	h100_default.json	same

Key takeaway: bf16 is free performance. No quality degradation at all — loss within 0.001 of fp32. The H100's tensor cores are designed for bf16; running fp32 was leaving 73% of the hardware on the table.

What this means for miners

A proof-test that took 4.3 hours in fp32 now takes 69 minutes. At ~$3/hr for an H100, that's:

• fp32: ~$13 per Karpathian-1 proof test
• bf16: ~$3.50 per Karpathian-1 proof test

Cheaper proof tests → lower barrier to submit → more patches competing → faster recipe improvement.

New infrastructure in this release

Feature	What it does
wandb live monitoring	--wandb flag streams loss/lr/grad_norm/throughput to wandb.ai/karpathian-ai/karpathian. Public — anyone can view.
wandb metrics in proof bundles	Training traces auto-exported as wandb_metrics.json alongside checkpoints. Miners submit their complete training story, not just the final number.
Karpathian Live dashboard	Streamlit dashboard with auto-refreshing loss curves, king status, noise floor stats. streamlit run dashboard/app.py
HuggingFace Hub integration	miner/hub.py — upload/download proof bundles to HuggingFace. Checkpoints + logs + wandb traces + attestation, all content-addressed.

Live training run

Watch the bf16 Karpathian-1 training on wandb (public):

🔗 wandb: https://wandb.ai/karpathian-ai/karpathian

How to reproduce

git clone https://github.com/KarpathianBase/karpathian.git
cd karpathian && git checkout v0.5.1
bash scripts/run_h100.sh  # bf16 is now the default

Or run just the training step with wandb:

python -m recipe.train \
    --config configs/h100_default.json \
    --out-dir runs/karpathian1_bf16 \
    --seed 1337 \
    --wandb

What's next

• Phase 0.5c ✅: Real TDX + nvtrust attestation module — code-complete, auto-detects CC hardware, falls back to mock. Ready for when CC instances are available.
• Phase 0.5d: Bittensor testnet integration — replace JSON-file chain with real on-chain commitments.
• Phase 1.0: Subnet launch.

---

🔗 Repo: github.com/KarpathianBase/karpathian
🏷️ This milestone: v0.5.1
💬 Discussions: github.com/orgs/KarpathianBase/discussions
📊 wandb: wandb.ai/karpathian-ai/karpathian

[bitzic]

Wandb run logs

https://wandb.ai/karpathian-ai/karpathian/runs/ccdgm3ve

[bitzic]

Miner training monitor