Custom Embedding Model

[walbercardoso]

Can i use a open Embedding Model instead openAI?

[fcakyon]

Yes for instance, you can use huggingface embedding models. Check the comment in this line for example usage: https://github.com/safevideo/autollm/blob/e3ce9884b36b037327146897fc2252eeb7d58c5c/examples/configs/config.example.yaml#L16

[walbercardoso]

Thanks for your quick reply. Only one more question, lancedb use 1536 dimensions, all HF embeddings models cant deal with it. Is there any docs how to use others vectorstore dbs?

[talhasarit]

Hello @walbercardoso,

Regarding your concern about embedding model dimensions and LanceDB: LanceDB now automatically adjusts to the vector size of your embedding model. This means you can use various Hugging Face models without worrying about dimensionality mismatches. For more details, check out LanceDB's embedding functions feature here: LanceDB makes multimodal AI easy.