TinyGPT v1.0 — public release

A GPT-2-shaped transformer, written from scratch and trained in your browser tab. Python reference, hand-written C++/WASM, hand-written WGSL — the same model at three levels, every gradient pinned down by a test.

Headline measured numbers (Apple M-series, same seed, same data)

Backend	Step time	Loss	Drift
WASM SIMD (baseline)	6.8 s	2.9385	—
WebGPU + blocked4 + FA2 fwd + FA2 bwd	0.7 s	2.8650	2.5%

~9.7× end-to-end speedup, 2.5% loss drift — float-reorder noise only, the model trains identically.

Major levers shipped

• Memory64 lifts the 4 GB tab heap → 473M-param model allocates
• Blocked4 matmul (workgroup-shared tiling + 4×4 register block) — default forward + backward
• vec4-aligned global loads on top of blocked4
• Subgroup-cooperative layernorm + cross-entropy reductions
• Flash Attention 2 (forward + backward) — backward recomputes attention from a saved log-sum-exp, which let the forward drop its O(B·H·T²) attention writeback entirely. ~67 MB of memory traffic per layer per step saved at Mega-class shapes.
• Watch-the-model-think view — click any generated token → see top-K next-byte probabilities and per-head attention thumbnails
• PostHog analytics, three events, DNT-respecting, no PII
• Astro frontend, five pages: /, /roadmap, /devlog, /speedup, /webgpu-test

Honest negative results (documented in the devlog)

• f16-packed storage on top of tiled matmul — wins standalone, doesn't compound on top of tiling because the kernel becomes compute-bound
• 8×8 register block — lost to 4×4 at every size, likely register spill
• vec4 first integration attempt — wins standalone but a silent WGSL access-mode mismatch broke non-square shapes; root-caused and fixed

End-to-end parity bar

tests/test_webgpu_train.mjs runs 50 WASM steps + 50 WebGPU steps on the same seed and asserts loss drift < 5%. Every kernel integration in this release went through that gate.

What's next

• Pre-trained model gallery — Cloudflare R2-hosted, manifest-driven
• Native macOS app — MLX-Swift + SwiftUI, same .tinygpt file format both ways

Docs

• browser/src/pages/devlog.astro — long-form session log with the negative results
• browser/src/pages/roadmap.astro — lever-by-lever
• docs/fa2_forward_notes.md + docs/fa2_backward_notes.md
• docs/online_softmax_in_attention.md
• docs/watch_the_model_think.md