Embeddings sentencepiecemodelloader

Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.4.0 — read its documentation.

SentencePieceModelLoader

Reads a SentencePiece spiece.model — the trained unigram vocabulary, its scores, its piece types and the model's special-token ids.

public static class SentencePieceModelLoader

Example — T5, ALBERT, camemBERT and XLM-R all ship this file.

using Lodestar.Embeddings.Persistence;
using Lodestar.Embeddings.Tokenization;

SentencePieceVocabulary vocab = SentencePieceModelLoader.Load("spiece.model");
var tokenizer = new SentencePieceTokenizer(vocab);

Remarks — spiece.model is a protobuf, and it carries far more than a word list. It records the type of every piece, so the tokenizer knows which entries are control markers instead of guessing from their ids, and it carries the scores unigram Viterbi segmentation needs.

It also carries the normalizer, as a compiled character map. That map is read from the file and never assumed to be identity — a stock model ships nmt_nfkc, and applying it is what makes tokenization here match Python on the same text.

Because all of that is in the file, Load takes only bounds. There is nothing left for a caller to get wrong.

Reference behaviour is sentencepiece.SentencePieceProcessor(model_file=…).

Applies to — net10.0, netstandard2.0.

See also — TokenizerJsonLoader, ArtifactLoadOptions, the persistence index.

Members

Member	What it does
SentencePieceModelLoader.Load	Reads a spiece.model.
SentencePieceModelLoader.LoadAsync	The same, asynchronously.