Repository navigation
Releases
Releases · vectorize-io/toktok
v0.5.0
pip install toktok-rs== · cargo add toktok-rs@
What's Changed
- Count lone surrogates instead of raising; add hs-decide-1-flash by @nicoloboschi in #9
Full Changelog: v0.4.0...v0.5.0
v0.4.0
pip install toktok-rs== · cargo add toktok-rs@
What's Changed
- Add embeddinggemma-2: exact token counts for Google's EmbeddingGemma 2 by @nicoloboschi in #7
Full Changelog: v0.3.0...v0.4.0
v0.3.0
pip install toktok-rs== · cargo add toktok-rs@
Full Changelog: v0.2.0...v0.3.0
v0.2.0
pip install toktok-rs== · cargo add toktok-rs@
What's Changed
- Add jev: an upper-bound token count for TypeSafe's Jev by @nicoloboschi in #5
Full Changelog: v0.1.4...v0.2.0
v0.1.4
pip install toktok-rs== · cargo add toktok-rs@
Full Changelog: v0.1.3...v0.1.4
v0.1.3
pip install toktok-rs== · cargo add toktok-rs@
What's Changed
- Test every Python version on every platform; run CI on PRs only by @nicoloboschi in #1
- README: lead with the Python numbers, fill the CPU column by @nicoloboschi in #2
- README: one benchmark table per runtime by @nicoloboschi in #3
- Native truncation: truncate() and batch_truncate() by @nicoloboschi in #4
New Contributors
- @nicoloboschi made their first contribution in #1
Full Changelog: v0.1.2...v0.1.3
v0.1.2
pip install toktok-rs== · cargo add toktok-rs@
Full Changelog: v0.1.1...v0.1.2
v0.1.1
pip install toktok-rs== · cargo add toktok-rs@
Full Changelog: https://github.com/vectorize-io/toktok/commits/v0.1.1