v0.7.0 — Tokenizers
Added — Tokenizers (opt-in [tokenizer] extra)
Download the GWDG open-weight models' tokenizers and measure prompts offline:
chat-template token counts, special/structural-token overhead (absolute + relative),
subword fertility, system-prompt-from-file, and token_distribution for sizing a RAG
corpus (incl. images) against the embedding model qwen3-embedding-4b.
- New
saia_python.tokenizermodule andclient.tokenizersservice. download_tokenizer/download_all_tokenizers(tokenizer files only, never weights),
load_tokenizer,HF_TOKENsupport, and an expressiveGatedRepoAccessError.count_tiktoken_tokensfor the external OpenAI models.- Docs: Extensions guide + quickstart section; runnable
examples/tokenizer_features.ipynb.
Install: pip install saia-python[tokenizer]
Full changelog: https://github.com/fschwar4/saia_python/blob/main/docs/CHANGELOG.md