Embeddings 0.4.0 vocabtxtloader

Lodestar.Embeddings 0.4.0. This page is frozen at that release. Read the current documentation for what main says now. A link to a decision or a migration page follows main, and leaves the archive.

VocabTxtLoader

Reads a BERT-style vocab.txt: one token per line, the id being the line number.

public static class VocabTxtLoader

Example — the route for a stock BERT checkpoint.

using Lodestar.Embeddings.Persistence;
using Lodestar.Embeddings.Tokenization;

WordPieceVocabulary vocab = VocabTxtLoader.Load("bert-base-uncased/vocab.txt", lowercase: true);
var tokenizer = new WordPieceTokenizer(vocab);

Remarks — this is the route for stock BERT, not a fallback. A HuggingFace BERT tokenizer.json declares a BertPreTokenizer and a full BertNormalizer, which TokenizerJsonLoader.LoadWordPiece refuses because this package does not reproduce those steps. vocab.txt carries no pipeline to disagree about.

The format records nothing but the tokens, so everything else is a parameter — Load has the three that matter and why lowercase is the dangerous one.

Reference behaviour is transformers.BertTokenizer's vocabulary loading, including two quirks of the Python loop it reproduces deliberately; docs/equivalence.md's loader row names them.

Applies to — net10.0, netstandard2.0.

See also — TokenizerJsonLoader, ArtifactLoadOptions, the persistence index.

Members

Member	What it does
VocabTxtLoader.Load	Reads a vocab.txt into a WordPiece vocabulary.
VocabTxtLoader.LoadAsync	The same, asynchronously.