Embeddings isubwordtokenizer

Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.4.0 — read its documentation.

ISubwordTokenizer

What the three tokenizers have in common: encode a string, and look a token up.

public interface ISubwordTokenizer

Example — the same code against whichever tokenizer the model came with.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["[UNK]"] = 0, ["token"] = 1, ["##ize"] = 2, ["text"] = 3,
};

ISubwordTokenizer tokenizer = new WordPieceTokenizer(
    vocab, unkToken: "[UNK]", continuationPrefix: "##", maxCharsPerWord: 100, lowercase: true);

TokenizationResult encoded = tokenizer.Encode("tokenize text");
int count = encoded.Ids.Count;  // => 3

Remarks — deliberately narrow. Decoding is not here, because only BpeTokenizer can do it losslessly: byte-level BPE round-trips any input exactly, while WordPiece has already thrown away the information about where words were split. Putting Decode on the interface would promise something two of the three cannot keep.

BatchEncoder takes this interface rather than a concrete tokenizer, which is what lets one batching path serve all three families.

Applies to — net10.0, netstandard2.0.

See also — WordPieceTokenizer, SentencePieceTokenizer, BpeTokenizer.

Members

Member	What it does
ISubwordTokenizer.Encode	One string to tokens and ids.
ISubwordTokenizer.TryGetId	The id of a token, if the vocabulary holds it.