perf(pipeline): cut on metaspace runs when no pre-tokenizer is declared

[ArthurZucker]

Draft, on top of #2279 (poc/target-encode, 106327e1).

The problem

pre_tokenizer: null falls to PipelinePreTokenizer::None, which pushes one span over the whole document:

Self::None => { out.push(Span { start: 0, end: text.len() as u32 }); Ok(()) }

Two things go wrong at once: the merge runs over ~10 kB of symbols instead of a word, and the WordCache keys on that whole span — so it can only ever hit on a byte-identical repeat of the document. llama-2 ships exactly this shape (pre_tokenizer: null + a Prepend+Replace normalizer that writes the delimiters).

Measured, llama-2/english, 10 kB chunks:

	tokens / 10 kB	model stage	ns / token
gpt2	2293	1.62 ns/B	7.2
llama-2	2654	33.43 ns/B	129.0

Token counts within 16% of each other, so this is span length, not merge quality — the two-tier queue is the better algorithm handed a far worse problem shape.

The change

When the normalizer writes ▁ and the vocabulary proves a cut at ▁ cannot change the result, install a pre-tokenizer that makes those cuts.

The proof. BPE only ever emits vocabulary pieces, so a merge can span a ▁ boundary only if some piece holds a ▁ after a non-▁ character. None does in llama-2, llama-3, mistral-nemo or gpt2 — one scan at load. That is what makes this byte-exact rather than an approximation.

Runs, not single delimiters. MetaspaceRuns cuts before each run. Cutting before every ▁ breaks the 15 multi-delimiter pieces llama-2 carries (▁▁ … ▁▁▁▁▁▁▁▁▁▁▁): indentation then tokenizes as N separate ▁ where the reference emits one piece. On the code corpus, id 1678 (▁▁▁▁) came out as 29871,29871,29871. A Split on regex ▁+ says the right thing but needs the optional fancy-regex backend and is far slower than one vectorised memchr pass.

Probed, not pattern-matched. Whether the normalizer writes delimiters is decided by normalizing " a" and looking for the mark, so it cannot go stale when another mark-writing normalizer is added.

Measured

Disjoint-slice probe — warm on unseen text, time on unseen text, so nothing measures memoization of the benchmark's own input. 3 interleaved A/B rounds on current heads; every cell's ids verified against encode_fast (all EXACT).

cell	before	after	ratio	per-round after
llama-2/english	23.0	72.6	3.16×	71, 74, 73
llama-2/code	24.7	62.8	2.54×	63, 65, 61
llama-2/xnli	50.7	95.6	1.89×	96, 97, 96
llama-2/russian	51.6	86.4	1.67×	86, 91, 83
llama-2/chinese	192.6	179.4	0.93×	180, 179, 179

Per-round spreads do not overlap on any llama-2 cell, so these are signal rather than drift.

gpt2 is untouched — 0.99×–1.02× across english/code/chinese/russian/xnli. It declares a pre-tokenizer and never reaches this path.

Stage cost, llama-2/english — the work moves out of the model:

stage	before	after
normalize	2.90	2.83
split	0.00	1.11
model	40.51	10.33

Known regression, not hidden

Chinese loses ~7% (192.6 → 179.4). It contains almost no ▁, so the pass scans the chunk and returns essentially one span: cost with no benefit. A mark-density guard would fix it, but it makes the pre-tokenizer input-dependent, which wants more thought than a bolt-on. This is why the PR is a draft.

Second commit

convert_chars had no ASCII fast path where convert_bytes does — it decoded UTF-8 and went through the 2-D get_char lookup for every character, including the pure-ASCII ones that dominate Latin text. Worth ~6% (40.51 → 37.95 ns/B), byte-exact, and separable if you'd rather take one.

What this is not

Not a merge improvement, and not a char-atom fold — that was worth 6%, not the 18×. The remaining gap to gigatoken on llama-2/english (72.6 vs 91.7) is the normalize + split overhead it avoids by fusing the ▁ rewrite into its unit scan rather than materialising a normalized string: normalize 2.83 + split 1.11 = 3.94 ns/B of our 14.3, against its ~10.9. Fusing those two passes is the next step and crosses the normalizer/pre-tokenizer split that to_normalizer_and_split deliberately maintains, so it is a design question rather than a patch.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.