Skip to content

Conversation

@ruchitharajaghatta
Copy link
Contributor

Description

Currently if folks use the generative ai feature in Compass with a collection with documents with a lot of fields, we'll show an error:
"Sorry, your collections have too many fields to process. Please try
using this feature on a collection with smaller documents."
PROMPT_TOO_LONG error from mms
This is because we don't want to exceed the token limit of open ai. We recently changed from gpt-3.5-turbo to gpt-4o. The new limit is 128,000 tokens and OpenAI approximates 1 token as ~= 4 english chars, so we updated this field to 4*128,000 = 512000 in the server. Since we want the final validation to be on the server lets do 10x what the server limit is at. The server does some further changes to schema and sample documents so we should let it be the decider. The frontend should only prevent uploading massive schemas and documents in the request. So, in compass the limit is set to 10x this (5,120,000).

Also updated the PROMPT_TOO_LONG message to be more transparent and clear that this is the limit causing the error.

Checklist

Motivation and Context

  • Bugfix
  • New feature
  • Dependency update
  • Misc

Open Questions

Dependents

Types of changes

  • Backport Needed
  • Patch (non-breaking change which fixes an issue)
  • Minor (non-breaking change which adds functionality)
  • Major (fix or feature that would cause existing functionality to change)

@github-actions github-actions bot added the fix label Oct 15, 2024
Copy link
Member

@Anemy Anemy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm ! Nice

@ruchitharajaghatta ruchitharajaghatta merged commit 5f45217 into main Oct 16, 2024
28 of 29 checks passed
@ruchitharajaghatta ruchitharajaghatta deleted the COMPASS-8369 branch October 16, 2024 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants