MicroGPT v1.0.0 — NumPy GPT from scratch

MicroGPT v1.0.0

A GPT language model implemented from scratch in pure NumPy — hand-written forward pass, hand-written backward pass, no autograd, no framework.

What is inside

• Decoder-only Transformer: causal multi-head self-attention, pre-LayerNorm, residual stream, GELU MLP (4x expansion), tied input/output embeddings.
• Hand-derived backpropagation: every gradient written explicitly and verified against central finite differences for the token embedding, positional embedding, attention projections, both MLP matrices, and both layer norms.
• Byte-level tokenizer: 256 symbols, no training required, exact UTF-8 round-trip.
• AdamW with bias correction, linear warmup and cosine decay.
• Sampling: temperature, top-k, plus a next-token distribution inspector and perplexity scoring.
• Checkpointing: compressed .npz weights with a JSON sidecar holding config, parameter count and eval history.
• CLI: train, sample, eval, info, next.
• 14 unit tests, including the gradient check (relative error bound 2e-4) and an end-to-end train/reload test.

Quick start

pip install -e .
python -m microgpt train --corpus examples/corpus.txt --steps 1200 --n-embd 128 --n-layer 3
python -m microgpt sample --prompt "A language model is a probability distribution over" --tokens 200

Layout

microgpt/model.py      forward + backward + sampling
microgpt/optim.py      AdamW and LR schedule
microgpt/train.py      batching, loop, checkpoints
microgpt/generate.py   completion, perplexity, next-token
microgpt/tokenizer.py  byte-level tokenizer
microgpt/__main__.py   CLI

Full Changelog: initial release.