Skip to content
This repository has been archived by the owner on Jul 4, 2023. It is now read-only.

Handle chat history exceeding max tokens #2

Open
Tracked by #1
edoardob90 opened this issue Mar 9, 2023 · 2 comments · May be fixed by #17
Open
Tracked by #1

Handle chat history exceeding max tokens #2

edoardob90 opened this issue Mar 9, 2023 · 2 comments · May be fixed by #17
Labels
medium Requires some thinking

Comments

@edoardob90
Copy link
Owner

edoardob90 commented Mar 9, 2023 •
edited
Loading

Max number of tokens per request (prompt + completion) is 4096.

Two ways to get the number of tokens used:

  1. Use the num_tokens_from_messages before making the API call (Best way, we don't need any API call)
  2. Access response['usage']['total_tokens']

What happens if tokens > 4096?

  • Add a warning message to the logfile
  • Build a new messages list containing the prompt for the assistant (if any, should be the very first message) plus the last 3 messages in the chat history. The number of messages kept should be configurable by the user
@edoardob90 edoardob90 mentioned this issue Mar 9, 2023
Open
12 tasks
@edoardob90 edoardob90 changed the title Handle chat history exceeding max number of tokens per request (4096) Handle chat history exceeding max tokens Mar 9, 2023
@edoardob90 edoardob90 added the medium Requires some thinking label Mar 11, 2023
@edoardob90
Copy link
Owner Author

Another option to fit a long conversation into the max tokens limit: First, ask the model to summarize in Q&A the conversation up to know, using less than a given number of tokens. Then, the summarized version of the chat can be passed in as chat history.

@edoardob90
Copy link
Owner Author

edoardob90 commented Mar 27, 2023 •
edited
Loading

  • Create a history_max_length settings (default is None). If set, the messages sent for a chat completion are at most this many.
  • The finish_reason in the API response indicates why the model stopped generating (docs). If finish_reason == length, it means the output is incomplete due to the max tokens limit reached. In this case, a warning message should be sent to the user, asking them to start a new chat.

@edoardob90 edoardob90 linked a pull request Apr 11, 2023 that will close this issue
Handle model's tokens limit #17
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
medium Requires some thinking
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle model's tokens limit edoardob90/telegram-chatgpt
1 participant
@edoardob90