You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've tried running encode on each of the "content" values, and summing those, as well as running encode on the JSON.stringify(entireMessagesArray).
The first method (summation of messages alone) gave me about 200 tokens less than what the actual OpenAI returned for "prompt_tokens", and the second overshot by about 200. For reference, this was on a request with 2994 prompt_tokens.
I'm using the "gpt-3.5-turbo" model, and importing the default encode from "gpt-tokenizer" (which according to the docus should align with gpt-3.5-turbo).