feat(thicktoken): replace tiktoken with in-repo Rust→WASM tokenizer engine

[slvnperron]

Summary

Swaps thicktoken's engine from tiktoken to our own Rust→WASM build (thicktoken/wasm/) while keeping the TextTokenizer / getWasmTokenizer() public API unchanged — llmz and other consumers work as-is. The BPE core is the tokie crate (MIT © Chonkie, Inc. — attribution in NOTICE), wrapped with a thin wasm-bindgen layer and a typed TS API.

Benchmarks — before vs after

Measured on the built dist/, same machine (darwin/arm64, Node 22), TOKEN×10M input (60 MB, exactly 10M tokens):

op	before (tiktoken)	after	speedup
count (new default: approximate)	5,546 ms	163 ms (±0.2% error)	~34×
count (exact)	5,546 ms	1,119 ms	~5×
split	8,323 ms	3,256 ms	~2.6×
truncate to a small budget (10M → 1,000 tokens)	~7,800 ms	146 ms	~53×
cold-start init (median)	~54 ms	~62 ms	≈ par
bundle (single-file ESM)	2,483 KB	1,156 KB	−53%

Two of these are algorithmic, not just engine speed:

• count is approximate by default on large inputs ({ approximate: false } opts out): it tokenizes ~48 KB of stratified sample windows regardless of input size and extrapolates — cost is ~constant in input length, so the gap grows with input size (2–3 orders of magnitude on multi-MB texts).
• truncate is windowed and exact: it only tokenizes the prefix/suffix window it needs, so trimming a huge text to a small budget no longer tokenizes the whole thing. (It's also boundary-exact — the old chunked implementation dropped a token at chunk boundaries.)

How the wasm stays small & fast to init

• Vocab is shipped merges-only (469 KB gz): model.vocab is fully derivable for a topological BPE, so it's rebuilt at init (base-256 byte-level alphabet + merge concatenations + specials).
• The tokenizer is built directly from merge id-pairs (no tokenizer.json round-trip) using tokie's Simple encoder (no Aho-Corasick automaton) → ~44 ms engine init, on par with tiktoken.
• Verified token-id-identical to tiktoken on prose, source code, unicode, contractions (independent ground truth: gpt-tokenizer, kept as devDependency for tests). Known alpha edge case, documented in tests: glued mid-word contractions (e.g. na'transl) tokenize to different ids (same text round-trip); never observed on real prose/code.

Vocab variants (edge targets)

One code-only wasm engine (~127 KB gz); each entry bundles its own vocab asset (truncated cl100k merge lists — valid BPE by prefix-closure). Same TextTokenizer API everywhere:

import	vocab	total gz (ESM)	init	counts vs cl100k
@bpinternal/thicktoken	cl100k	~647 KB	~60 ms	exact
@bpinternal/thicktoken/lite	cl50k	~392 KB	~25 ms	overcounts ~+3-7%
@bpinternal/thicktoken/micro	cl25k	~275 KB	~13 ms	overcounts ~+8-20%

Truncated variants only ever overcount (safe for budget enforcement — never overflows a window); not for billing math. Inflation is content-dependent (worst on emoji/unicode-heavy text). Assets regenerate via wasm/scripts/gen-assets.mjs; all variants verified in Chromium (browser-test.html).

API changes (⚠️ minor break → v3.0.0)

• count(text, max?: number) → count(text, options?: { approximate?: boolean }). The old max ceiling is gone (obsolete — counting is fast now). llmz only calls count(text), unaffected.
• New exports: WasmTokenizer (direct engine access: encode/decode/split/slice/truncate(text, n, 'head'|'tail'|'middle')), TruncateMode, CountOptions.
• tiktoken dependency removed.

Rebuilding the wasm

wasm/pkg/ is committed, so the package builds and tests without a Rust toolchain. To rebuild from Rust source: see wasm/README.md (wasm-pack, --target web).

Test plan

• 63 tests pass: full legacy TextTokenizer suite + 39 primitive tests (parity vs independent cl100k reference, exact truncate head/tail/middle, slice incl. negative indices, approximate-count error bounds, documented edge case)
• tsc --noEmit clean
• CJS + ESM dist smoke-tested (init, count, split, truncate, splitAndSlice)

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

Replaces the tiktoken npm dependency with a self-contained Rust→WASM BPE engine (tokie crate, MIT) compiled to wasm/pkg/, keeping the TextTokenizer / getWasmTokenizer() public surface intact while adding head/tail/middle truncation modes and an approximate-count fast path. The version bump to 3.0.0 covers the count(text, max?) → count(text, options?) signature change.

• Engine swap — splitIntoChunks chunking and tiktoken's JS WASM are removed; the new Rust layer handles truncate/split/count natively, cutting bundle size ~53% and delivering 5–53× speedups on the main hot paths.
• Approximate count — count() now samples ~12 × 4 KB windows instead of full-encoding, making it near-constant-time for large inputs; exact mode is opt-in via { approximate: false }.
• Windowed truncate — take_head/take_tail use an adaptive character-window loop (up to 8 iterations) to avoid tokenizing the full input when only a small prefix/suffix is needed; a whitespace-boundary snap keeps cl100k pretokenizer alignment correct.

Confidence Score: 4/5

Safe to merge; the public API is backward-compatible for callers that only use count(text) and truncate, and the two findings are non-blocking quality notes.

The engine swap is well-tested (63 tests, independent gpt-tokenizer reference), the wasm bytes are pre-built and committed so CI requires no Rust toolchain, and the singleton initialization is correctly handled synchronously. Two issues are worth tracking: slice double-encodes the input text when negative or omitted indices are used, and CountOptions is defined independently in both tokenizer.ts and wasm/index.ts rather than shared from one source.

thicktoken/wasm/index.ts (slice double-encode) and thicktoken/src/tokenizer.ts (duplicate CountOptions type).

Important Files Changed

Filename	Overview
thicktoken/wasm/src/lib.rs	Core Rust→WASM BPE engine: build_tokenizer (merges-only cl100k), approx-count (stratified sampling), windowed head/tail/middle truncate, decode_lossy. Well-structured; snap_to_whitespace only covers ASCII whitespace (correct for cl100k pretokenizer).
thicktoken/wasm/index.ts	Typed JS wrapper over wasm-bindgen bindings; adds TruncateMode union, default approximate=true, and negative-index slice. The slice method calls count(text,false) then raw.slice(text,s,e), encoding the full text twice when indices are negative or end is omitted.
thicktoken/src/tokenizer.ts	Main public API: replaces Tiktoken with WasmTokenizer, drops splitIntoChunks chunking, adds head/tail/middle truncate modes. CountOptions is redefined locally instead of re-using the one from wasm/index.ts, creating a duplicate type in the public surface.
thicktoken/src/wasm-tokenizer.test.ts	New 39-test suite: exact parity against gpt-tokenizer for count/truncate/slice, approximate-count error bounds, known edge case (glued contractions) explicitly documented. Good coverage.
thicktoken/src/tokenizer.test.ts	Legacy TextTokenizer tests updated: old overflow test replaced by ±1% approximate-count bound (tight but safe for uniform SINGLE_TOKEN input); truncate boundary-exactness improved from 199_998 to 199_999; new head/tail/middle mode tests added.
thicktoken/vitest.wasm-plugin.ts	Vite plugin to inline WASM for vitest (Node.js only path via Buffer); mirrors the tsup esbuild plugin without the browser atob fallback, which is correct since vitest always runs in Node.
thicktoken/tsup.config.ts	Adds universal base64 decode for inlined WASM (Buffer on Node, atob+Uint8Array in browsers); removes leftover console.log. Clean.
thicktoken/wasm/Cargo.toml	Depends on tokie@0.0.10 (no default features → no Aho-Corasick, no network), miniz_oxide for gzip, serde_json for asset parsing. Release profile is correctly optimized (opt-level=z, LTO, strip).

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant C as Consumer (llmz etc.)
    participant TT as TextTokenizer
    participant WT as WasmTokenizer (wasm/index.ts)
    participant RW as RawTokenizer (wasm-bindgen)
    participant RS as Rust / WASM

    C->>TT: await getWasmTokenizer()
    activate TT
    TT->>WT: WasmTokenizer.create()
    WT->>WT: "ensureInit() -> initSync(wasmBytes)"
    WT->>RW: new RawTokenizer()
    RW->>RS: inflate_gzip + build_tokenizer
    RS-->>RW: tokie::Tokenizer
    RW-->>WT: RawTokenizer instance
    WT-->>TT: WasmTokenizer instance
    TT-->>C: TextTokenizer singleton

    C->>TT: count(text)
    TT->>WT: "count(text, {})"
    WT->>RW: "count(text, approximate=true)"
    RW->>RS: approx_count (stratified sampling)
    RS-->>C: approx tokens (fast, within few%)

    C->>TT: truncate(text, N, tail)
    TT->>WT: truncate(text, N, tail)
    WT->>RW: truncate(text, N, tail)
    RW->>RS: take_tail (optimistic suffix window)
    RS-->>C: exact last-N-token substring

    C->>TT: count(text, approximate false)
    TT->>WT: count(text, approximate false)
    WT->>RW: "count(text, approximate=false)"
    RW->>RS: count_tokens (full encode)
    RS-->>C: exact token count

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant C as Consumer (llmz etc.)
    participant TT as TextTokenizer
    participant WT as WasmTokenizer (wasm/index.ts)
    participant RW as RawTokenizer (wasm-bindgen)
    participant RS as Rust / WASM

    C->>TT: await getWasmTokenizer()
    activate TT
    TT->>WT: WasmTokenizer.create()
    WT->>WT: "ensureInit() -> initSync(wasmBytes)"
    WT->>RW: new RawTokenizer()
    RW->>RS: inflate_gzip + build_tokenizer
    RS-->>RW: tokie::Tokenizer
    RW-->>WT: RawTokenizer instance
    WT-->>TT: WasmTokenizer instance
    TT-->>C: TextTokenizer singleton

    C->>TT: count(text)
    TT->>WT: "count(text, {})"
    WT->>RW: "count(text, approximate=true)"
    RW->>RS: approx_count (stratified sampling)
    RS-->>C: approx tokens (fast, within few%)

    C->>TT: truncate(text, N, tail)
    TT->>WT: truncate(text, N, tail)
    WT->>RW: truncate(text, N, tail)
    RW->>RS: take_tail (optimistic suffix window)
    RS-->>C: exact last-N-token substring

    C->>TT: count(text, approximate false)
    TT->>WT: count(text, approximate false)
    WT->>RW: "count(text, approximate=false)"
    RW->>RS: count_tokens (full encode)
    RS-->>C: exact token count

Loading

_{Reviews (1): Last reviewed commit: "feat(thicktoken): expose truncate modes ..." | Re-trigger Greptile}