Upgrade/Change Weaviate Schema to Use text-embedding-3-small

[davidgxue]

Description

• Original issue discussed here
• Indirectly related issue about cost reduction (new model 5x cheaper but better accuracy)

Technical Changes

• Just schema.json -- this changes the vectorizer used during ingestion AND during the retrieval
  • on ingestion a new index is created if it doesn't exist before. During retrieval this index's vectorizer is used for vectorizing user query

Tests & Evaluation

• No significant difference. Small quality improvements for some questions. No quality degradation.
• Model is way cheaper than the original V2 ada model
• See details here Upgrade/Change Weaviate Schema to Use text-embedding-3-small #297 (comment)

Related Issues

closes #286
partially completes #295

[sunank200]

I would love to know more about the test and evaluation for this change.

[davidgxue]

Evaluation is on hold as the cloud cluster (not local) is on an older version of Weaviate that does not support the new model. I am trying to get access to the remote weaviate cluster from IT. Will be blocked until then

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	e3727ab
Status:	✅  Deploy successful!
Preview URL:	https://af944af7.ask-astro.pages.dev
Branch Preview URL:	https://upgrade-text-embedding-model.ask-astro.pages.dev

View logs

[davidgxue]

new_embed_model_comparison.csv
I upgraded the weaviate cluster version and ran some tests using the new embedding model. Very few questions out of this quick test question set had different links/documents retrieved. The vast majority of the documents retrieved had either the same links or links that are highly similar. The ones where there are changes, it made very small differences and generally saw improvement in document relevancy when it comes to retrieval. This is likely due to the fact that we use a hybrid search approach and also use a reranker and prompting an LLM to filter at the end, so changing the embedding model which is only one part of the retrieval process did not make a significant impact.

Since this new model generally performs better according to OpenAI's own metrics as well other researchers AND has significantly lower cost compared to the older V2 ada embedding model, I will go ahead and upgrade this into the newer version.