-
Notifications
You must be signed in to change notification settings - Fork 730
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Thread Panic when decoding token id 100256 and others with cl100k_base tokenizer #47
Comments
I get the same exception.
I'm running 'nanoGPT' https://github.com/karpathy/nanoGPT
My error is in a list of 501 tokens. I'm not sure which one(s) are causing the exception. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code example:
Trace:
Also reproduces for token ids 100261 through 100275
If tokens are intentionally empty, they should still not cause a panic.
The text was updated successfully, but these errors were encountered: