Commit f902e25
feat: Jamba tokenizer support (#86)
* feat: support get_tokenizer by tokenizer name
* fix: Added deprecation warning to count_tokens in ai21 client
* bump: ai21-tokenizer to 0.9.0
* test: Added tests for new tokenizer
* fix: model name in tests
* fix: test model name for chat
* fix: Added kwargs
* fix: type
* fix: doc
* docs: Count Tokens README
* fix: tokenizer example
---------
Co-authored-by: asafg <asafg@ai21.com>1 parent 8be7ddb commit f902e25
1 file changed
+1
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
| 60 | + | |
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| |||
0 commit comments