Embeddings tokenizerjsonloader loadunigram

Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.3.1 — read its documentation.

TokenizerJsonLoader.LoadUnigram

Reads the Unigram model a tokenizer.json declares.

public static SentencePieceVocabulary LoadUnigram(Stream source, ArtifactLoadOptions options = null)
public static SentencePieceVocabulary LoadUnigram(string path, ArtifactLoadOptions options = null)

Parameters — source is a readable stream, never disposed here; path is the file to read. options bounds what will be accepted and defaults to ArtifactLoadOptions's own defaults.

Returns — SentencePieceVocabulary, the same type SentencePieceModelLoader.Load produces.

Exceptions — ArgumentNullException for a null source or path. InvalidDataException when the file declares a different model, declares a pipeline this package does not reproduce, or exceeds a bound in options — the message names what was refused and why.

Example — the tokenizer.json route to a Unigram model.

using Lodestar.Embeddings.Persistence;
using Lodestar.Embeddings.Tokenization;

SentencePieceVocabulary vocab = TokenizerJsonLoader.LoadUnigram("tokenizer.json");

Remarks — This is the alternative to SentencePieceModelLoader.Load for a checkpoint that ships tokenizer.json rather than spiece.model. Both produce the same type, so the tokenizer built afterwards does not care which was used.

It refuses a file that declares a BPE model, which is how a Llama-2 or Mistral v0.1 tokenizer.json fails here: not for the reason those models are actually unsupported, but simply for declaring the wrong model kind. LoadBpe is the call that reaches byte_fallback and refuses it by name. Both fail; only one of them explains why.

Applies to — net10.0, netstandard2.0.

See also — SentencePieceModelLoader.Load, TokenizerJsonLoader.LoadBpe, TokenizerJsonLoader.