Skip to content

Releases

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 16 Nov 11:14

pip install toktok-rs== · cargo add toktok-rs@

What's Changed

  • Count lone surrogates instead of raising; add hs-decide-1-flash by @nicoloboschi in #9

Full Changelog: v0.4.0...v0.5.0

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 13 Nov 12:18

pip install toktok-rs== · cargo add toktok-rs@

What's Changed

  • Add embeddinggemma-2: exact token counts for Google's EmbeddingGemma 2 by @nicoloboschi in #7

Full Changelog: v0.3.0...v0.4.0

v0.3.0

Choose a tag to compare

@github-actions github-actions released this 21 Oct 09:08

pip install toktok-rs== · cargo add toktok-rs@

Full Changelog: v0.2.0...v0.3.0

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 19 Oct 18:34

pip install toktok-rs== · cargo add toktok-rs@

What's Changed

  • Add jev: an upper-bound token count for TypeSafe's Jev by @nicoloboschi in #5

Full Changelog: v0.1.4...v0.2.0

v0.1.4

Choose a tag to compare

@github-actions github-actions released this 02 Sep 21:52

pip install toktok-rs== · cargo add toktok-rs@

Full Changelog: v0.1.3...v0.1.4

v0.1.3

Choose a tag to compare

@github-actions github-actions released this 02 Sep 09:38

pip install toktok-rs== · cargo add toktok-rs@

What's Changed

  • Test every Python version on every platform; run CI on PRs only by @nicoloboschi in #1
  • README: lead with the Python numbers, fill the CPU column by @nicoloboschi in #2
  • README: one benchmark table per runtime by @nicoloboschi in #3
  • Native truncation: truncate() and batch_truncate() by @nicoloboschi in #4

New Contributors

Full Changelog: v0.1.2...v0.1.3

v0.1.2

Choose a tag to compare

@github-actions github-actions released this 02 Sep 07:11

pip install toktok-rs== · cargo add toktok-rs@

Full Changelog: v0.1.1...v0.1.2

v0.1.1

Choose a tag to compare

@github-actions github-actions released this 02 Sep 06:56

pip install toktok-rs== · cargo add toktok-rs@

Full Changelog: https://github.com/vectorize-io/toktok/commits/v0.1.1