-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: truncate embedding tokens #918
Conversation
marcusschiesser
commented
Jun 10, 2024
•
edited
Loading
edited
- truncate text to embed before sending it to OpenAI
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
🦋 Changeset detectedLatest commit: a3694dd The changes in this PR will be included in the next version bump. This PR includes changesets to release 9 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
@@ -109,6 +123,16 @@ export class OpenAIEmbedding extends BaseEmbedding { | |||
* @param options | |||
*/ | |||
private async getOpenAIEmbedding(input: string[]): Promise<number[][]> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not a good solution.
This will break the return type of getTextEmbedding('large chunk')
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
doesn't getTextEmbedding('large chunk')
- it just ensures that getTextEmbedding('large chunk')
is not throwing an exception
21d27b0
to
5ada8db
Compare
8498f23
to
e6a26b9
Compare
let me ignore waku for now, it might be their issue |