Skip to content

v3.8.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 08 Aug 18:37
· 53 commits to develop since this release
v3.8.0
8851d9e

The Unicode performance release, and an o200k pattern fix.

Performance — native encode is now 5–49× faster than tiktoken-rs 0.9 (Chinese/Japanese prose 15–17×, previously ~2×); the wasm build is 2–4× faster than gpt-tokenizer 3.4 in the browser on every corpus, CJK prose included. Three changes: a key-size-layered vocabulary (direct tables for 1–2-byte keys, inlined-key slots for 3–8), hand-written CJK fast-path scanners pinned char-by-char to the regex's Unicode tables, and thread-local whole-piece memoisation. Benchmarks and corpora ship in-repo (bench-compare/, web/bench/) — try it live at tiktoken.golia.jp.

Fixed — o200k's punctuation rule now admits / in its [\r\n/]* tail per upstream (".\n/" is a single token, 118550). Kimi, which shared the o200k scanner, got its own variant without the slash tail.

Verification — 44,518 differential fixture cases across 16 encodings, 66,546-case canonical parity vs Python tiktoken 0.12, 23 property tests × 20k random inputs per run: zero divergence.

Full details: CHANGELOG

Token ids change in this release for slash-after-newline shapes on o200k_base / o200k_harmony. The consolidated list of every id-affecting change since 3.5.1 is in the CHANGELOG's upgrade section.

Assets 2
Loading