Skip to content

v0.1.0

Choose a tag to compare

@adeoluo adeoluo released this 02 Nov 04:51
· 19 commits to main since this release
456921f

First release of a decoder-only transformer and reverse-mode autodiff engine in pure Python.

Highlights

  • Forward pass matches PyTorch to within 1e-9 (opt-in validation group)
  • Character-level training on public-domain prose: 3.64 bits per character versus a 4.30 unigram baseline
  • Checkpointing with exact save and restore
  • A pre-registered research study with 8 seeds: copying is carried by first-layer heads looking back a fixed distance, not an induction circuit; one of three hypotheses is not supported, two are mixed. See docs/RESEARCH.md

Install: pip install the attached wheel.