v0.10.0

What's Changed

Features

• Use singleton BPE in num_tokens_from_messages and get_completion_max_tokens by @zurawiki in #140
• Count function_call tokens in num_tokens_from_messages by @zurawiki in #145
• Change decode() to accept &[Rank] and make decode_bytes() public by @zurawiki in #146
• Return &'static CoreBPE from get_bpe_from_model and get_bpe_from_tokenizer by @zurawiki in #147
• Rename misleading API functions, deprecate old names by @zurawiki in #154
• Add count-only token counting APIs by @zurawiki in #153

Bug Fixes

• Fix should-publish version check failing in CI by @zurawiki in #143
• Fix gpt-3.5 model version check in num_tokens_from_messages by @zurawiki in #144

Dependencies

• chore(deps): update extractions/setup-just action to v3 by @renovate[bot] in #127
• chore(deps): update actions/checkout action to v5 by @renovate[bot] in #126
• chore(deps): pin swatinem/rust-cache action to 98c8021 by @renovate[bot] in #125
• chore(deps): update actions/checkout v6, pin rust-toolchain, update create-gh-release v1.10.0 by @zurawiki in #142
• chore(deps): update extractions/setup-just action to v4 by @renovate[bot] in #150
• chore(deps): update swatinem/rust-cache action to v2.9.1 - autoclosed by @renovate[bot] in #149
• Sync regex patterns with upstream tiktoken, bump deps, port tests by @zurawiki in #151
• Bump async-openai to 0.34, add tool_calls/refusal token counting, update docs by @zurawiki in #152

Other Changes

• Fix release CI and harden workflow permissions by @zurawiki in #141
• Add release.yml for auto-generated release notes by @zurawiki in #148
• Release v0.10.0 by @zurawiki in #155

Full Changelog: v0.9.1...v0.10.0