You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks for this. I've noticed a weird issue though both with this library and also the official code from open ai that I found a while back before gpt-4 came out.
What is happening is that the tokens calculated by this tool are much higher than the openai api is reporting in the completion. For example, a prompt I just submitted to gpt-4 was calculated as 7810 tokens by this library but when I got the completion from openai it told me my prompt had 5423 tokens. I'm not sure if you have also noticed something similar? In the prompt I'm submitting primarily Node JS code.
The text was updated successfully, but these errors were encountered:
Hi @Qarj! Thanks for flagging this problem.
As @ricardomatias noticed in #5, the tokenizer is using the r50k_base encoding, which isn't the one used by GPT-4. Hence the token offset. I'm working on v2 which will allow for choosing which encoding to use, which will correctly tokenize for GPT-4 specifically.
So it seems it is much closer now to the actual tokens, in a test I did the prompt was calculated as 998 tokens according to the library but 1003 tokens according to open ai. I suspect if we allow a 50 token margin then our completion token requests should always be within limit.