A byte level BPE tokenizer written in pure Mojo, with no Python interpreter at run time.
What it does
All seven tiktoken encodings: cl100k_base, o200k_base, o200k_harmony, p50k_base, p50k_edit, r50k_base and gpt2. Encode, decode, counting without allocating, windowing with overlap, truncation, batch encoding, and padded batches with attention masks. A knap command line tool with completions for bash, zsh and fish, and a Python extension you build locally.
What is verified
Every figure below was observed on the machine described in docs/BENCHMARKS.md, not inferred.
| Measure | Value |
|---|---|
Tokens compared against tiktoken over 110 MB |
191762320, byte identical |
| Piece boundaries compared over the same corpus | 83025959 |
| Token ids decoded and compared | 702463, every id in every encoding |
| Unicode code points verified against an independent reference | 1114112 |
Strings fuzzed against tiktoken |
20000000, zero divergences |
| Tests | 138 across 19 files |
How it performs
| Implementation | cl100k_base |
o200k_base |
gpt2 |
p50k_base |
|---|---|---|---|---|
rs-bpe |
8.75 MB/s | 8.83 MB/s | not shipped | not shipped |
| Knap | 5.98 MB/s | 6.32 MB/s | 4.90 MB/s | 5.62 MB/s |
tiktoken |
4.18 MB/s | 6.38 MB/s | 4.05 MB/s | 5.10 MB/s |
Faster than tiktoken on three of the four distinct encode behaviours and level with it on the fourth. Slower than rs-bpe everywhere rs-bpe runs, which the benchmarks document says with the same prominence. Decode runs at 140.36 MB/s against 52.14, and a loaded encoding costs between 4.4 and 11.7 times less memory than the reference implementation.
Requirements
Mojo 1.0.0, pinned exactly. The Mojo ABI is not stable, so a build against one toolchain is not interchangeable with a build against another. There is no wheel on PyPI and there should not be one until that changes.
Vocabularies are not bundled. They are third party data under their own terms, and scripts/fetch_vocabs.py downloads the four files that serve all seven encodings, recording a URL and a SHA-256 for each.
Where to start
- README.md, including when not to use Knap
- examples/, two working programs for the two problems callers of every tokenizer solve wrongly
- docs/CORRECTNESS.md, what parity means here and exactly what was checked
- docs/METHODOLOGY.md, how the numbers were measured, written to be reused elsewhere
- docs/TOOLCHAIN.md, thirty-five things about Mojo 1.0.0 that are not what you would assume, useful whether or not you use Knap
What is deliberately absent
No Hugging Face tokenizer.json loader: that format specifies its own pre-tokenizer, so it needs its own parity corpus and its own reference implementation, and shipping one without those would put an unverified path inside a library whose whole claim is verification. Batch encoding is single threaded, which is not a choice: Mojo 1.0.0 has no working task parallelism.
Full detail in CHANGELOG.md.