Embeddings tokenization

Tokenization — Lodestar.Embeddings

A transformer does not read text; it reads token ids. This namespace turns one into the other, for the three sub-word algorithms the models in use are built on, and it reproduces HuggingFace tokenizers and sentencepiece closely enough that the ids match theirs.

Getting them to match matters more than it sounds: a model fed ids from the wrong tokenizer returns vectors that are confidently wrong rather than an error.

Which tokenizer?

The answer is whichever the model was trained with — this is not a choice you get to make. The model's own files say which, and which loader reads them:

flowchart TD
    A["What did the model ship?"] --> B["vocab.txt"]
    A --> C["spiece.model"]
    A --> D["tokenizer.json"]
    A --> E["vocab.json + merges.txt"]

    B --> W["WordPieceTokenizer<br/>VocabTxtLoader"]
    C --> C1{"Trained with<br/>byte_fallback?"}
    C1 -->|no| S["SentencePieceTokenizer<br/>SentencePieceModelLoader"]
    C1 -->|yes| X["Refused at load, by design"]

    D --> D1{"What does<br/>model.type say?"}
    D1 -->|WordPiece| W2["WordPieceTokenizer<br/>TokenizerJsonLoader.LoadWordPiece"]
    D1 -->|Unigram| S2["SentencePieceTokenizer<br/>TokenizerJsonLoader.LoadUnigram"]
    D1 -->|BPE| D2{"Does the model declare<br/>byte_fallback?"}
    D2 -->|no| P["BpeTokenizer<br/>TokenizerJsonLoader.LoadBpe"]
    D2 -->|yes| X

    E --> P2["BpeTokenizer<br/>BpeFilesLoader"]

Loading

A tokenizer.json does not say which loader to call — its model.type does. The three Load… methods each assert it and refuse a file declaring another, so reaching for the wrong one fails with a message naming the mismatch rather than producing ids that look plausible.

byte_fallback is refused rather than ignored, on both paths that can carry it, because Python resolves an uncovered character into <0x..> byte pieces where these tokenizers emit the unknown piece — silently accepting it would return confidently wrong vectors. That is the SentencePiece-BPE lineage Llama-2 and Mistral v0.1 need, tracked at #175 and scoped by decision 0017 §3.

The same routing, as a table:

The model ships	Use	Loaded from
vocab.txt	WordPieceTokenizer	BERT and its descendants
spiece.model	SentencePieceTokenizer	T5, ALBERT, XLM-R, camemBERT
merges.txt or a tokenizer.json with merges	BpeTokenizer	GPT-2, Llama-3, Qwen2

All three implement ISubwordTokenizer, so code that only encodes can be written once against that.

The three families, and what actually differs

WordPiece splits a word into the longest pieces its vocabulary holds, marking every piece after the first with a continuation prefix — ##ize is "ize, continuing a word". A word it cannot cover at all becomes a single unknown token, not a sequence of partial ones.

SentencePiece treats the text as a stream and encodes the space itself, as ▁. That is why its tokens carry a leading ▁ and why it needs no pre-tokenizer: word boundaries are inside the vocabulary rather than assumed by a regex.

BPE starts from characters and applies a ranked list of merges in order. Byte-level BPE — what GPT-2 and Llama-3 use — maps bytes to printable characters first, which is what lets it round-trip any input exactly, emoji and broken UTF-8 included.

From text to a model's input

Encoding one string gives a TokenizationResult. Feeding a model wants more than that: a rectangular batch, padded, with an attention mask and the model's own special tokens. BatchEncoder does that, driven by EncodingOptions and a SpecialTokenTemplate, and hands back an EncodedBatch.

The template has to match the model and the vocabulary: asking for SpecialTokenTemplate.Bert against a vocabulary with no [CLS] is refused at construction rather than encoded into something the model will misread.

Types

Type	What it is
AddedToken	A token matched literally, before the model sees the text.
BatchEncoder	Strings in, a padded batch with an attention mask out.
BpePatterns	The four pre-tokenizer regexes real BPE models use.
BpeSplitStep	A Split step ahead of ByteLevel, as Llama-3 declares one.
BpeTokenizer	Byte-level and classic BPE, encoding and decoding.
BpeVocabulary	A BPE model: the vocabulary, the merges, and the flags.
EncodedBatch	The rectangular result: ids, mask, and true lengths.
EncodingOptions	Length, truncation, template, and batching.
ISubwordTokenizer	What the three tokenizers have in common.
MergePair	One BPE merge rule, left and right.
PrecompiledNormalizer	SentencePiece's charsmap normalization.
SentencePiece	One piece: its text, its score, its id.
SentencePieceTokenizer	Unigram encoding over a SentencePiece vocabulary.
SentencePieceType	What a piece is for — normal, control, unused.
SentencePieceVocabulary	The pieces, their types, and the four special ids.
SpecialTokenTemplate	Which tokens wrap a sequence, per model family.
SplitBehavior	What a Split step does with the text it matched.
TokenizationResult	Tokens and ids, from encoding one string.
TruncationStrategy	Which end is cut when a sequence is too long.
WordPieceTokenizer	Longest-match sub-word encoding with a continuation prefix.
WordPieceVocabulary	The vocabulary and the settings that read it.

See also

• Semantic search with embeddings — the guide, end to end.
• ONNX inference — what consumes the ids this namespace produces.
• Python → C# equivalence — every tokenizers call and its counterpart.