ValueError: tiktoken plugin tiktoken_ext.openai_public does not define ENCODING_CONSTRUCTORS

[renata-gotler]

Hi,

I was running the following code and it started to return error.

import tiktoken
tiktoken.get_encoding("o200k_base")

The error:
AttributeError: module 'tiktoken_ext.openai_public' has no attribute 'ENCODING_CONSTRUCTORS'

The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
File , line 2
1 import tiktoken
----> 2 tiktoken.get_encoding("o200k_base")
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-38b8f866-2ee0-46b2-9e25-6778b2c2c9d7/lib/python3.11/site-packages/tiktoken/registry.py:75, in get_encoding(encoding_name)
72 return ENCODINGS[encoding_name]
74 if ENCODING_CONSTRUCTORS is None:
---> 75 _find_constructors()
76 assert ENCODING_CONSTRUCTORS is not None
78 if encoding_name not in ENCODING_CONSTRUCTORS:
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-38b8f866-2ee0-46b2-9e25-6778b2c2c9d7/lib/python3.11/site-packages/tiktoken/registry.py:46, in _find_constructors()
44 constructors = mod.ENCODING_CONSTRUCTORS
45 except AttributeError as e:
---> 46 raise ValueError(
47 f"tiktoken plugin {mod_name} does not define ENCODING_CONSTRUCTORS"
48 ) from e
49 for enc_name, constructor in constructors.items():
50 if enc_name in ENCODING_CONSTRUCTORS:

It is weird because around 1h30min ago it worked and now it is throwing this error. Can some help me?

[prdai]

Hi,

Here are a few steps you can try:

1. Update tiktoken:
   Uninstall your current version and then install the latest one by running:

   pip uninstall tiktoken
   pip install --upgrade tiktoken

2. Alternative Option:
   If you still experience issues, consider switching to cl100k_base as it's known to be more reliable.

I also ran the sample code on my end, and it worked fine. Here’s a quick look at my environment:

• Python Version: 3.12.6
• Tiktoken Version: 0.8.0
• OS: Ubuntu 24.04.1 LTS

Hope this helps!

[renata-gotler]

I'm not sure what happened but it is working now, so I will close this issue.