llama-cpp-python (vicuna 13b) producing extremely poor embeddings with chromadb/langchain

[khimaros]

i'm working on a question answering chatbot over my personal document store using langchain's LlamaCppEmbeddings, the LlamaCpp LLM, and the Chroma vectorstore.

when i use LlamaCppEmbeddings as the embedding_function (with vicuna v1 13b 4bit quantized weights), the performance of similarity_search is extremely poor. often, the best result is the very last or second to last result. this happens even when the text query is an exact substring of the input text, when there is no other input text with the same string.

switching to the default embedding function SentenceTransformerEmbeddingFunction with no other changes vastly improves the behavior of my chatbot due to returning a much saner ranking of input documents. this workaround is okay for me, but it seems like LlamaCppEmbeddings should be able to far outperform this much smaller transformer model and i suspect there is not-subtle bug lurking.

the consistency of these poor results makes me wonder if there is a sign or lt/gt mixed up somewhere. i looked at the implementation of LlamaCppEmbeddings and it is so simple that i don't see how the error could be coming from there instead of llama-cpp-python (or llama.cpp).

i'd be happy to help troubleshoot this, and i have a self contained, simple, and easily redistributable example which demonstrates this issue consistently.

[khimaros]

here's the demonstration case: https://gist.github.com/khimaros/58db0c79a7ab69547e058422a848712a

if you comment out line 58 and uncomment lines 51-57 (switching the embedding function to LlamaCppEmbeddings) you will see the degradation of similarity search quality as well as a few orders of magnitude slower vectorstore creation.

[jakvb]

same problem, switched from vicuna 13B to llama and same poor results

[MakkiNeutron]

Same happening for me llama embedding for GPT4All, using FAISS or chromadb , annoy is faster then all for similar search.
All are not fetching more relevant chunk of the text.
totally poor results after embedding, is this matter of FAISS or llama embedding

[khimaros]

@MakkiNeutron in the meantime, i recommend using the workaround provided above. sentence_transformer support has now been merged into langchain and the performance is both several orders of magnitude faster (30s vs. 6 hours) and more accurate.

[gjmulder]

Any update?

[bergos]

See this issue in the llama.cpp repo for more details. Let's hope they switch to pooling soon. In case you want an OpenAI compatible API for Hugging Face sentence transformer models, you can try: https://github.com/bergos/embedding-server