Tiktoken package for openai script not working

[azmathr]

In my pthon script I have tiktoken enabled which essencially converts characters to openai tokens. Its a common module used in openai. Here is an associated code.

encoding = tiktoken.get_encoding("cl100k_base")

`def num_tokens_from_string(string: str, encoding_name: str) -> int:
"""Returns the number of tokens in a text string."""
encoding = tiktoken.get_encoding(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens

def get_text_until_limit(elements, limit):
text = ''
for element in elements:
new_text = text + ' ' + element.get_text()
if num_tokens_from_string(new_text, "cl100k_base") <= limit:
text = new_text
else:
break
return text
`
Unfortunately this is not working through zappa. I am getting error

 encoding = tiktoken.get_encoding("cl100k_base")   File "/var/task/tiktoken/registry.py", line 64, in get_encoding     _find_constructors()   File "/var/task/tiktoken/registry.py", line 44, in _find_constructors     raise ValueError( [1707923842317] [DEBUG] 2024-02-1

warmhandler error

[souravjamwal77]

@azmathr Can you please post the full error trace?

[azmathr]

It was solved by deleting the pycache file in tiktoken and site package folder Sent from Gmail Mobile

…

On Fri, Feb 23, 2024 at 3:40 PM Sourav Kumar ***@***.***> wrote: @azmathr <https://github.com/azmathr> Can you please post the full error trace? — Reply to this email directly, view it on GitHub <#1306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABEK25NEBOGBFM2GX5AHKCLYVBRQDAVCNFSM6AAAAABDKGZKAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRGAZDQMRSGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[monkut]

closing this issue as it was resolved by the op.