Why does newline negatively impact embedding performance?

[ravwojdyla]

Describe the bug

While reading the code of the embeddings_utils I have stumbled upon this:

openai-python/openai/embeddings_utils.py

Lines 20 to 21 in db3f352

	# replace newlines, which can negatively affect performance.
	text = text.replace("\n", " ")

Could you please provide more context on:

replace newlines, which can negatively affect performance.

Are there any references/papers/numbers behind that negative impact?

To Reproduce

get_embedding("foo bar\nbaz")

Code snippets

No response

OS

macOS

Python version

3.10

Library version

0.27.4

[ravwojdyla]

Sorry, maybe this should not be marked with bug, tho the user may not expect that the text is actually preprocessed (new lines removed). This could potentially be confusing if they fingerprint text and compute embeddings using this lib and something else.

[ravwojdyla]

Git blame points me to @BorisPower, would appreciate your input please.

[BorisPower]

This used to be a problem due to how the v1 embeddings were trained. However this was addressed in v2, and v2 should not have a problem with newlines!

[ravwojdyla]

@BorisPower ah, great, thanks for a prompt response, will remove that in a PR then

[ravwojdyla]

@BorisPower submitted a PR #419, out of curiosity do you happen to have reference(s) that explain why V1 had issues with new lines?

Edit: I would like to include that in the code as reference for posterity ^

[EliahKagan]

Although this doesn't explain the reason, there's a more specific statement of which models perform better without newlines in the embeddings documentation:

With the -001 text embeddings (not -002, and not code embeddings), we suggest replacing newlines (\n) in your input with a single space, as we have seen worse results when newlines are present.

The first-generation code embedding models are those that end in -code-001; of the first-generation models, those are the ones whose input is intended to be code rather than natural-language text. The models that statement refers to as "-001 text embeddings" are the other first-generation models. (Those all end in -001, but they don't all end in -text-001.)

Unfortunately, the above statement hard to link to, so there might not be any good way to include a comment linking to it in the code. It appears at the bottom of the "First generation models (not recommended)" section, which itself can't be linked to, and which is collapsed by default. It appears immediately under the Code search embeddings subsection, but while the page offers that link, that link does not currently navigate to that section of the page.

That embeddings_utils.get_embedding and the three closely related functions currently replace newlines even when a non-first-generation model is specified, and that they currently replace newlines even when a first-generation code model is specified, are arguably two separate bugs. But they can be fixed together. I am also inclined to think this issue could cover both. (However, you think it's best to have a separate issue for -code-001 behavior, I can go ahead and open one.)