v0.9.0 — Subword Tokenization (BPE)

A from-scratch Byte Pair Encoding tokenizer — the second real tokenizer — plus an honest re-run of the IMDB benchmark.

Added

• train_bpe learns subword merges from a corpus (word-level BPE, Sennrich et al.), and BPETokenizer applies them. It satisfies the v0.3 Tokenizer protocol (validating that contract) and splits rare/unseen words into known subwords instead of <unk>.
• A TOKENIZER switch in the example ("bpe" | "whitespace"); runs are recorded per model and tokenizer.

Benchmark (honest)

Re-running IMDB (25k/25k, seed 0) with BPE:

Model	Whitespace (20k)	BPE (10k)
Mean-pooling baseline	0.856	0.839
Transformer (from scratch)	0.855	0.838

BPE slightly hurt here. IMDB sentiment lives in common whole words, which BPE fragments (diluting the signal and losing more to truncation) — subwording pays off for out-of-vocabulary / morphology, not this task. This disproved the earlier hypothesis that tokenization was the ceiling: the real ceiling (~85–86%) is the model class — simple, from-scratch, no pretraining. The lever that breaks it (pretrained representations) is the next milestone.

BPE remains a correct, valuable tokenizer; the benchmark honestly shows when it helps.

See CHANGELOG.md for full details.