Skip to content

v0.7.0 — Tokenizers

Choose a tag to compare

@fschwar4 fschwar4 released this 22 Jun 16:46

Added — Tokenizers (opt-in [tokenizer] extra)

Download the GWDG open-weight models' tokenizers and measure prompts offline:
chat-template token counts, special/structural-token overhead (absolute + relative),
subword fertility, system-prompt-from-file, and token_distribution for sizing a RAG
corpus (incl. images) against the embedding model qwen3-embedding-4b.

  • New saia_python.tokenizer module and client.tokenizers service.
  • download_tokenizer / download_all_tokenizers (tokenizer files only, never weights),
    load_tokenizer, HF_TOKEN support, and an expressive GatedRepoAccessError.
  • count_tiktoken_tokens for the external OpenAI models.
  • Docs: Extensions guide + quickstart section; runnable examples/tokenizer_features.ipynb.

Install: pip install saia-python[tokenizer]

Full changelog: https://github.com/fschwar4/saia_python/blob/main/docs/CHANGELOG.md