Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chat: fix at-mention token size #3526

Merged
merged 6 commits into from
Mar 26, 2024
Merged

Chat: fix at-mention token size #3526

merged 6 commits into from
Mar 26, 2024

Conversation

abeatrix
Copy link
Contributor

@abeatrix abeatrix commented Mar 22, 2024

RE: #3523 & #2636

This PR adds character limit tracking for chat models in the SimpleChatModel class. It introduces a new property maxChars that stores the maximum number of characters allowed for the selected chat model.
The charsLeft property is also added to calculate the remaining characters available based on the current chat messages without context.

These changes allow the chat interface to track the character limit for the selected chat model and ensure that the at-mention context items are showing the correct warning for large files based on the latest available token limit instead of a standardized one that doesn't represent the latest limit.

Test plan

Try @Simple and look for SimpleChatModel.ts in the cody repository following the examples below

Before

@ a file that is showing File too large warning, but when you select it, it will still be included in your message because the warning size was using the incorrect limit number.

image

You can see Cody is able to reply to you because the file was still included, meaning the warning was incorrectly displayed:

image

After

Type @Simple and look for SimpleChatModel.ts again, you can see SimpleChatModel.ts doesn't have the warning anymore, and selecting it still works:

image

However, keep asking Cody question so that the context window will become smaller, and when the messages is larger than the file, then the warning will show up correctly for the file:

image

Limitation

The large @-mention files can still be added by the user, which has been addressed in a separate PR.

Currently cannot dynamically calculate the token limit change for a resubmitted query, meaning users won't see the file size warning if they have selected an at-mention item that used all the limit, but they can still add more at-mention items. This will also be addressed in a future PR.

image

Follow-up

@abeatrix abeatrix requested review from toolmantim and a team March 25, 2024 18:15
Copy link
Contributor

@toolmantim toolmantim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to add an impressive amount of context:

CleanShot 2024-03-26 at 14 57 59@2x

Large files showed the correct thing:

CleanShot 2024-03-26 at 14 59 06@2x

@abeatrix abeatrix merged commit d735ff6 into main Mar 26, 2024
20 of 22 checks passed
@abeatrix abeatrix deleted the bee/at-size branch March 26, 2024 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants