Skip to content

Commit

Permalink
Increase token size buffer to account for underestimation
Browse files Browse the repository at this point in the history
In the `get_llm_model` function, the token size is now increased by 10% to account for the occasional underestimation by tiktoken. This change helps to ensure that the selected model can handle the required token size.
  • Loading branch information
TechNickAI committed Jun 29, 2023
1 parent 28e213c commit aa156b5
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions aicodebot/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,10 @@ def get_llm_model(token_size):
"gpt-3.5-turbo-16k": 16384,
}
gpt_4_supported = os.getenv("GPT_4_SUPPORTED") == "true"

# For some unknown reason, tiktoken often underestimates the token size by ~10%, so let's buffer
token_size = int(token_size * 1.1)

if gpt_4_supported:
if token_size <= model_options["gpt-4"]:
return "gpt-4"
Expand Down

0 comments on commit aa156b5

Please sign in to comment.