Repository navigation
Releases: dmatth1/quicktok
Releases · dmatth1/quicktok
Release list
v0.4.0
v0.3.2
README: pre-launch review fixes 1-5 - document the importer's verification deps (tokenizers / mistral-common / huggingface_hub) — the one runtime surprise a new user would hit - quick-start import example is now the HF repo-id form, validated live (mistralai/Mistral-Nemo-Instruct-2407 -> verified vs mistral-common); the importer now prefers tekken.json over tokenizer.json when a repo has both - SentencePiece (Gemma/T5/Llama-2) explicitly out of scope in Importing - wheel platforms + Python floor stated in Install - llama4 guidance works from pip: import via repo id (verified, with the rank-vs-merges clause); the export tool stays as the rank-truth path