Embedding model support

[jmorganca]

Add embedding models to use primarily with /api/embeddings

• instructor-xl
• bge-large
• all-MiniLM-L6-v2

See the full leaderboard

[brunnolou]

Yes, please! Any of these embedding models above text-embedding-ada-002 would be a great addition.

I've tried LLam2 and Mistral model with the /api/embeddings as is, and I'm getting poor-quality similarity scores.
Even with almost identical queries, It fails to retrieve results. Are there some prompting technics to improve the embedding quality?

Anyway, in comparison, I've tried Xenova/gte-small with transformers and it is much faster and yields better results.

[corani]

jinaai/jina-embeddings-v2-base-en (and other variants) also look promising.

[sandangel]

Hi, is there an update on this issue? I would love to contribute

[corani]

I've been playing with https://github.com/nlpodyssey/cybertron which is pure Go (but I guess CPU only?) and at least supports all-MiniLM-L6-v2, e5-*-v2, bge-*-en-v1.5 and ember-v1.

I did some testing with the STS-2016 dataset and got the below accuracies compared to llama2 and mistral:instruct (Pearson correlation with the gold answers):

• Ollama
  • llama2: 0.23431
  • mistral:instruct: 0.5656
• Cybertron
  • all-MiniLM-L6-v2: 0.80344
  • e5-small-v2: 0.82318
  • e5-base-v2: 0.83845
  • bge-small-en-v1.5: 0.84514
  • bge-base-en-v1.5: 0.85297

So I agree with the previous comment that the embeddings generated by the completion models are pretty bad!

[sandangel]

That is interesting. For GPU support, I guess we will need to use: https://github.com/skeskinen/bert.cpp ?
I think the implementation would be something similar to llama.cpp?

[sandangel]

I found this issue: ggerganov/llama.cpp#2872
I think they plan to implement it in llama.cpp. So maybe we will just need to update the llama.cpp when it's done?

[sandangel]

I also found this https://github.com/ml-explore/mlx-examples/blob/main/bert/README.md, which we can use to run inference on M1 mac. Is it possible to support mlx with Ollama?

[CodeWithKyrian]

Any update on this or plan to allow Bert Models?

[tjohnson4]

Any update on this issue?

[ymohamed08]

Do you have any updates so far? very interested to contribute

[ill-yes]

Any updates here?

[easp]

Plans to support BERT models in llama.cpp stalled out when the dev who had assumed the task ended up focusing on something else. In the last few days it looks like the project management artifacts were updated to acknowledge this, so maybe there will be some action soon. Actually, it looks like there has been some activity. Maybe there will be working code soon:
ggerganov/llama.cpp#2872

[AndreBerzun]

BERT support was merged 3 days ago into llama.cpp

[easp]

Looks like there are still kinks being worked out.

[s-kostyaev]

Looks like there are still kinks being worked out.

Link to check the progress ggerganov/llama.cpp#5500

[s-kostyaev]

Looks like there are still kinks being worked out.

Link to check the progress ggerganov/llama.cpp#5500

It is merged now

[AndreBerzun]

@jmorganca just wanted to follow up and see if this topic is on your roadmap. Since llama.cpp added support for BERT models, this seems like a great low-hanging fruit, no?

Initial support for BERT models has been merged with ggerganov/llama.cpp#5423 and released with b2127. Some kinks related to embedding pooling were fixed with ggerganov/llama.cpp#5500. Batch embedding is supported as well.

There has been a new bug related to the tokenizer implementation but that's it as far as I can tell.

[jmorganca]

@AndreBerzun it absolutely is – working on it!