Skip to content

Commit f902e25

Browse files
feat: Jamba tokenizer support (#86)
* feat: support get_tokenizer by tokenizer name * fix: Added deprecation warning to count_tokens in ai21 client * bump: ai21-tokenizer to 0.9.0 * test: Added tests for new tokenizer * fix: model name in tests * fix: test model name for chat * fix: Added kwargs * fix: type * fix: doc * docs: Count Tokens README * fix: tokenizer example --------- Co-authored-by: asafg <asafg@ai21.com>
1 parent 8be7ddb commit f902e25

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ packages = [
5757
[tool.poetry.dependencies]
5858
python = "^3.8"
5959
requests = "^2.31.0"
60-
ai21-tokenizer = "^0.3.9"
60+
ai21-tokenizer = "^0.9.0"
6161
boto3 = { version = "^1.28.82", optional = true }
6262
dataclasses-json = "^0.6.3"
6363
typing-extensions = "^4.9.0"

0 commit comments

Comments
 (0)