Embeddings 0.4.0 sentencepiecemodelloader

Lodestar.Embeddings 0.4.0. This page is frozen at that release. Read the current documentation for what main says now. A link to a decision or a migration page follows main, and leaves the archive.

SentencePieceModelLoader

Reads a SentencePiece spiece.model — the trained unigram vocabulary, its scores, its piece types and the model's special-token ids.

public static class SentencePieceModelLoader

Example — T5, ALBERT, camemBERT and XLM-R all ship this file.

using Lodestar.Embeddings.Persistence;
using Lodestar.Embeddings.Tokenization;

SentencePieceVocabulary vocab = SentencePieceModelLoader.Load("spiece.model");
var tokenizer = new SentencePieceTokenizer(vocab);

Remarks — spiece.model is a protobuf, and it carries far more than a word list. It records the type of every piece, so the tokenizer knows which entries are control markers instead of guessing from their ids, and it carries the scores unigram Viterbi segmentation needs.

It also carries the normalizer, as a compiled character map. That map is read from the file and never assumed to be identity — a stock model ships nmt_nfkc, and applying it is what makes tokenization here match Python on the same text.

Because all of that is in the file, Load takes only bounds. There is nothing left for a caller to get wrong.

Reference behaviour is sentencepiece.SentencePieceProcessor(model_file=…).

Applies to — net10.0, netstandard2.0.

See also — TokenizerJsonLoader, ArtifactLoadOptions, the persistence index.

Members

Member	What it does
SentencePieceModelLoader.Load	Reads a spiece.model.
SentencePieceModelLoader.LoadAsync	The same, asynchronously.