Highlights
Peak throughput: 618 MiB/s — up to 42x faster than other .NET tokenizers on multilingual/CJK text.
Performance improvements
- O(n log n) min-heap BPE merge — replaces O(n²) linear scan in
FindParts, improving cold-path CJK/multilingual by ~13%
- Restored ASCII fast path — fixes a 35% regression on cached CJK/multilingual text introduced in v2.3.0
- Cache speedup: 5-13x on repeated multilingual/CJK text (was 4-9x)
CountTokens — zero allocation
| Input |
Tiktoken |
Throughput |
vs competitors |
| Hello, World! (13 B) |
88 ns |
141 MiB/s |
1.9-3.6x |
| Multilingual (382 B) |
1.1 us |
339 MiB/s |
4.7-13.6x |
| CJK-heavy (1,676 B) |
2.6 us |
618 MiB/s |
14.3-42.3x |
| Python code (879 B) |
5.5 us |
153 MiB/s |
1.8-4.0x |
| Multilingual long (4,312 B) |
9.0 us |
458 MiB/s |
7.9-31.6x |
| Bitcoin whitepaper (19,884 B) |
105.1 us |
180 MiB/s |
2.4-3.8x |
New
- ColdPath benchmark project — dedicated benchmark for cache vs no-cache comparison
- SmokeTest project — local perf regression guard using machine-independent cache speedup ratios
- Aho-Corasick investigation — research document on alternative tokenization approaches
Cross-language context (Apple M4 Max, o200k_base)
| Implementation |
Encode Throughput |
CountTokens Throughput |
| Tiktoken .NET (cached) |
114-484 MiB/s |
141-618 MiB/s |
| Tiktoken .NET (no cache) |
44-145 MiB/s |
47-155 MiB/s |
| tiktoken Rust v3 |
34-88 MiB/s |
— |
| GitHub bpe Rust v0.3 |
33-64 MiB/s |
29-66 MiB/s |
| OpenAI tiktoken Python 0.12 |
7-20 MiB/s |
— |
