refactor: Removed ai21-tokenizer dependency

.pre-commit-config.yaml

@@ -8,7 +8,6 @@ repos:
     rev: v4.4.0
     hooks:
       - id: check-added-large-files
-        exclude: (ai21_tokenizer/resources|tests/resources)
       - id: check-case-conflict
       - id: check-executables-have-shebangs
       - id: check-shebang-scripts-are-executable

README.md

@@ -29,7 +29,6 @@
 - [Older Models Support Usage](#Older-Models-Support-Usage)
 - [More Models](#More-Models)
   - [Streaming](#Streaming)
-- [Token Counting](#Token-Counting)
 - [Environment Variables](#Environment-Variables)
 - [Error Handling](#Error-Handling)
 - [Cloud Providers](#Cloud-Providers) ☁️
@@ -323,39 +322,6 @@ file_id = client.library.files.create(
 uploaded_file = client.library.files.get(file_id)
 ```
 
-## Token Counting
-
----
-
-By using the `count_tokens` method, you can estimate the billing for a given request.
-
-```python
-from ai21.tokenizers import get_tokenizer
-
-tokenizer = get_tokenizer(name="jamba-tokenizer")
-total_tokens = tokenizer.count_tokens(text="some text")  # returns int
-print(total_tokens)
-```
-
-### Async Usage
-
-```python
-from ai21.tokenizers import get_async_tokenizer
-
-## Your async function code
-#...
-tokenizer = await get_async_tokenizer(name="jamba-tokenizer")
-total_tokens = await tokenizer.count_tokens(text="some text")  # returns int
-print(total_tokens)
-```
-
-Available tokenizers are:
-
-- `jamba-tokenizer`
-- `j2-tokenizer`
-
-For more information on AI21 Tokenizers, see the [documentation](https://github.com/AI21Labs/ai21-tokenizer).
-
 ## Environment Variables
 
 ---

ai21/clients/studio/ai21_client.py

@@ -7,16 +7,13 @@
 from ai21.clients.studio.resources.studio_chat import StudioChat
 from ai21.clients.studio.resources.studio_library import StudioLibrary
 from ai21.http_client.http_client import AI21HTTPClient
-from ai21.tokenizers.ai21_tokenizer import AI21Tokenizer
 
 
 class AI21Client(AI21HTTPClient):
     """
     This class would be sending requests to our REST API using http requests
     """
 
-    _tokenizer: Optional[AI21Tokenizer]
-
     def __init__(
         self,
         api_key: Optional[str] = None,