v1.3.1 — HuggingFace tokenizer.json Native Support

Load HuggingFace tokenizers directly from tokenizer.json — no conversion step required.

What's New

HuggingFace tokenizer.json Format Support

You can now load any HuggingFace tokenizer.json file without converting it to SentencePiece .model format first. This makes it straightforward to use tokenizers published on the HuggingFace Hub.

// Load from file
final tokenizer = await HuggingFaceTokenizerLoader.fromJsonFile('tokenizer.json');

// Load from a pre-parsed map
final tokenizer = HuggingFaceTokenizerLoader.fromMap(jsonMap);

// Auto-detection — works transparently with TokenizerJsonLoader
final tokenizer = await TokenizerJsonLoader.fromJsonFile('tokenizer.json');

Supported model types:

• Unigram — Llama, T5, ALBERT, XLNet, and other Unigram-based models
• BPE — Gemma, GPT-2, RoBERTa, and other BPE-based models

Automatic configuration inference:

• Special tokens (unk, bos, eos, pad) are detected from the added_tokens section
• Normalizer settings (addDummyPrefix, escapeWhitespaces) are inferred from the HuggingFace normalizer config
• Post-processor flags (addBosToken, addEosToken) are parsed from TemplateProcessing
• Byte fallback behavior is detected from the decoder configuration
• Tokens beyond the base vocabulary are handled automatically

Format detection:

TokenizerJsonLoader.isHuggingFaceFormat() lets you check whether a JSON map uses the HuggingFace format. When you call TokenizerJsonLoader.fromJsonFile(), HuggingFace format is detected and delegated automatically — no code changes needed if you already use TokenizerJsonLoader.

Install / Upgrade

dependencies:
  dart_sentencepiece_tokenizer: ^1.3.1

Full Changelog: https://github.com/brody-0125/dart_sentencepiece_tokenizer/blob/develop/CHANGELOG.md