Embeddings tokenization

Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.3.1 — read its documentation.

Tokenization — Lodestar.Embeddings

A transformer does not read text; it reads token ids. This namespace turns one into the other, for the three sub-word algorithms the models in use are built on, and it reproduces HuggingFace tokenizers and sentencepiece closely enough that the ids match theirs.

Getting them to match matters more than it sounds: a model fed ids from the wrong tokenizer returns vectors that are confidently wrong rather than an error.

Which tokenizer?

The answer is whichever the model was trained with — this is not a choice you get to make. The model's own files say which:

The model ships	Use	Loaded from
vocab.txt	WordPieceTokenizer	BERT and its descendants
spiece.model	SentencePieceTokenizer	T5, ALBERT, XLM-R, camemBERT
merges.txt or a tokenizer.json with merges	BpeTokenizer	GPT-2, Llama-3, Qwen2

All three implement ISubwordTokenizer, so code that only encodes can be written once against that.

The three families, and what actually differs

WordPiece splits a word into the longest pieces its vocabulary holds, marking every piece after the first with a continuation prefix — ##ize is "ize, continuing a word". A word it cannot cover at all becomes a single unknown token, not a sequence of partial ones.

SentencePiece treats the text as a stream and encodes the space itself, as ▁. That is why its tokens carry a leading ▁ and why it needs no pre-tokenizer: word boundaries are inside the vocabulary rather than assumed by a regex.

BPE starts from characters and applies a ranked list of merges in order. Byte-level BPE — what GPT-2 and Llama-3 use — maps bytes to printable characters first, which is what lets it round-trip any input exactly, emoji and broken UTF-8 included.

From text to a model's input

Encoding one string gives a TokenizationResult. Feeding a model wants more than that: a rectangular batch, padded, with an attention mask and the model's own special tokens. BatchEncoder does that, driven by EncodingOptions and a SpecialTokenTemplate, and hands back an EncodedBatch.

The template has to match the model and the vocabulary: asking for SpecialTokenTemplate.Bert against a vocabulary with no [CLS] is refused at construction rather than encoded into something the model will misread.

Types

Type	What it is
AddedToken	A token matched literally, before the model sees the text.
BatchEncoder	Strings in, a padded batch with an attention mask out.
BpePatterns	The four pre-tokenizer regexes real BPE models use.
BpeSplitStep	A Split step ahead of ByteLevel, as Llama-3 declares one.
BpeTokenizer	Byte-level and classic BPE, encoding and decoding.
BpeVocabulary	A BPE model: the vocabulary, the merges, and the flags.
EncodedBatch	The rectangular result: ids, mask, and true lengths.
EncodingOptions	Length, truncation, template, and batching.
ISubwordTokenizer	What the three tokenizers have in common.
MergePair	One BPE merge rule, left and right.
PrecompiledNormalizer	SentencePiece's charsmap normalization.
SentencePiece	One piece: its text, its score, its id.
SentencePieceTokenizer	Unigram encoding over a SentencePiece vocabulary.
SentencePieceType	What a piece is for — normal, control, unused.
SentencePieceVocabulary	The pieces, their types, and the four special ids.
SpecialTokenTemplate	Which tokens wrap a sequence, per model family.
SplitBehavior	What a Split step does with the text it matched.
TokenizationResult	Tokens and ids, from encoding one string.
TruncationStrategy	Which end is cut when a sequence is too long.
WordPieceTokenizer	Longest-match sub-word encoding with a continuation prefix.
WordPieceVocabulary	The vocabulary and the settings that read it.

See also

• Semantic search with embeddings — the guide, end to end.
• ONNX inference — what consumes the ids this namespace produces.
• Python → C# equivalence — every tokenizers call and its counterpart.