perf(embedding): default embedding creation to base64 #1312
+92
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manekinekko
force-pushed
the
perf/wassim-chegham-issue-1310
branch
from
7702d54 to
270861b
Compare
1 task
manekinekko
force-pushed
the
perf/wassim-chegham-issue-1310
branch
from
83fbd84 to
185dbe5
Compare
manekinekko
force-pushed
the
perf/wassim-chegham-issue-1310
branch
2 times, most recently
from
e34c241 to
fd14cdf
Compare
|
This is a great idea! Who doesn't want 1/4 the network bandwidth? |
manekinekko
force-pushed
the
perf/wassim-chegham-issue-1310
branch
from
fd14cdf to
362f02f
Compare
RobertCraigie
requested changes
Mar 18, 2025
@RobertCraigie no worries about the delay. Thank you for reviews. I addressed your suggestions. |
manekinekko
force-pushed
the
perf/wassim-chegham-issue-1310
branch
2 times, most recently
from
c50fa5f to
84180db
Compare
Requesting base64 encoded embeddings returns smaller body sizes, on average ~60% smaller than float32 encoded. In other words, the size of the response body containing embeddings in float32 is ~2.3x bigger than base64 encoded embedding. We always request embedding creating encoded as base64, and then decoded them to float32 based on the user's provided encoding_format parameter. Closes openai#1310
Co-authored-by: Robert Craigie <robert@craigie.dev>
Co-authored-by: Robert Craigie <robert@craigie.dev>
Co-authored-by: Robert Craigie <robert@craigie.dev>
Co-authored-by: Robert Craigie <robert@craigie.dev>
Co-authored-by: Robert Craigie <robert@craigie.dev>
manekinekko
force-pushed
the
perf/wassim-chegham-issue-1310
branch
from
84180db to
d2bc20b
Compare
|
@RobertCraigie PR is now ready for review. Thank you. |
RobertCraigie
approved these changes
Mar 28, 2025
| export const toFloat32Array = (base64Str: string): Array<number> => { | ||
| if (typeof Buffer !== 'undefined') { | ||
| // for Node.js environment | ||
| return Array.from(new Float32Array(Buffer.from(base64Str, 'base64').buffer)); |
There was a problem hiding this comment.
curious if you've benchmarked how much of a difference just returning the Float32Array directly would have?
if it's a big difference we should probably have an opt-in flag to just do that. (doesn't block this PR)
RobertCraigie
changed the title
RobertCraigie
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requesting base64 encoded embeddings returns smaller body sizes, on average ~60% smaller than float32 encoded. In other words, the size of the response body containing embeddings in float32 is ~2.3x bigger than base64 encoded embedding.
Closes #1310
Changes being requested
We always request embedding creating encoded as base64, and then decoded them to float32 based on the user's provided encoding_format parameter.
Additional context & links
After running a few benchmarks, requesting base64 encoded embeddings returns smaller body sizes, on average ~60% smaller than float32 encoded. In other words, the size of the response body containing embeddings in float32 is ~2.3x bigger than base64 encoded embedding.
This performance improvement could translate to:
This is the result of a request that creates embedding from a 10kb chunk, run 10 times (the number are the size of response body in kb):
Read more #1310