Skip to content

v0.9.0 — Modern architecture (RoPE, RMSNorm, SwiGLU)

Choose a tag to compare

View all tags
@cattolatte cattolatte released this 06 Jul 16:54
v0.9.0
ba0584c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The decoder is now Llama-style from scratch — RoPE, RMSNorm, SwiGLU (configurable; GPT-2-style still available).

On tiny-shakespeare it reaches 2.08 bits/char (vs 2.11 GPT-2-style) at fewer params and ~2× faster convergence — see BENCHMARKS.md for the architecture ablation. RoPE is verified correct under the KV-cache; pre-v0.9 checkpoints still load.

Assets 2
Loading