fix(compass-generative-ai): Increase the character size restriction for query and aggregation genai COMPASS-8369 #6370
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently if folks use the generative ai feature in Compass with a collection with documents with a lot of fields, we'll show an error:
"Sorry, your collections have too many fields to process. Please try
using this feature on a collection with smaller documents."
PROMPT_TOO_LONG error from mms
This is because we don't want to exceed the token limit of open ai. We recently changed from gpt-3.5-turbo to gpt-4o. The new limit is 128,000 tokens and OpenAI approximates 1 token as ~= 4 english chars, so we updated this field to 4*128,000 = 512000 in the server. Since we want the final validation to be on the server lets do 10x what the server limit is at. The server does some further changes to schema and sample documents so we should let it be the decider. The frontend should only prevent uploading massive schemas and documents in the request. So, in compass the limit is set to 10x this (5,120,000).
Also updated the PROMPT_TOO_LONG message to be more transparent and clear that this is the limit causing the error.
Checklist
Motivation and Context
Open Questions
Dependents
Types of changes