cant store embeddings from llama_cpp_python

[HolzerDavid]

System Info

Python==3.11.5
langchain==0.0.300
llama_cpp_python==0.2.6
chromadb==0.4.12

Running on Windows and on CPU

Who can help?

@agola11
@hwchase17

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

Embeddings can't be stored in a chroma/FAISS vectorstore using llama-cpp-python.
I found that embedding the contents of very very simple websites, such as "http://example.org" work fine.

Here is the code I am executing in my notebook:

from langchain.document_loaders import WebBaseLoader
from langchain.embeddings import LlamaCppEmbeddings
from langchain.vectorstores import Chroma

llama = LlamaCppEmbeddings(model_path="../models/openorca_stx.gguf")

loader = WebBaseLoader("https://www.bbc.com/")
pages = loader.load()

vectordb = Chroma.from_documents(
    documents=pages,
    embedding=llama,
    persist_directory='../data/vectorstores/'
)

Here is the Traceback:

ValueError                                Traceback (most recent call last)
[p:\git_repos\langchain-test\src_langchain\04_vectorstore.ipynb](file:///P:/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb) Cell 8 line 1
      [7](vscode-notebook-cell:/p%3A/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb#X12sZmlsZQ%3D%3D?line=6) loader = WebBaseLoader("https://www.bbc.com/")
      [8](vscode-notebook-cell:/p%3A/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb#X12sZmlsZQ%3D%3D?line=7) pages = loader.load()
---> [10](vscode-notebook-cell:/p%3A/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb#X12sZmlsZQ%3D%3D?line=9) vectordb = Chroma.from_documents(
     [11](vscode-notebook-cell:/p%3A/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb#X12sZmlsZQ%3D%3D?line=10)     documents=pages,
     [12](vscode-notebook-cell:/p%3A/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb#X12sZmlsZQ%3D%3D?line=11)     embedding=llama,
     [13](vscode-notebook-cell:/p%3A/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb#X12sZmlsZQ%3D%3D?line=12)     persist_directory='../data/vectorstores/'
     [14](vscode-notebook-cell:/p%3A/git_repos/langchain-test/src_langchain/04_vectorstore.ipynb#X12sZmlsZQ%3D%3D?line=13) )

File [d:\miniconda3\envs\langchain\Lib\site-packages\langchain\vectorstores\chroma.py:646](file:///D:/miniconda3/envs/langchain/Lib/site-packages/langchain/vectorstores/chroma.py:646), in Chroma.from_documents(cls, documents, embedding, ids, collection_name, persist_directory, client_settings, client, collection_metadata, **kwargs)
    644 texts = [doc.page_content for doc in documents]
    645 metadatas = [doc.metadata for doc in documents]
--> 646 return cls.from_texts(
    647     texts=texts,
    648     embedding=embedding,
    649     metadatas=metadatas,
    650     ids=ids,
    651     collection_name=collection_name,
    652     persist_directory=persist_directory,
    653     client_settings=client_settings,
    654     client=client,
    655     collection_metadata=collection_metadata,
    656     **kwargs,
...
--> 510 self.input_ids[self.n_tokens : self.n_tokens + n_tokens] = batch
    511 # Save logits
    512 rows = n_tokens if self.params.logits_all else 1

ValueError: could not broadcast input array from shape (8,) into shape (0,)

Expected behavior

I expect this function Chroma.from_documents( documents=pages, embedding=llama, persist_directory='../data/vectorstores/' ) to create document embeddings and store them in the persistent chromadb

[dosubot]

🤖

Hello,

Thank you for providing detailed information about your issue. Based on the traceback you provided, it seems like the error is occurring when the Chroma.from_documents() method is trying to store the embeddings generated by the LlamaCppEmbeddings model. The error message suggests that there's a shape mismatch when trying to store the embeddings.

This issue might be related to the maximum token limit of the LlamaCppEmbeddings model. In LangChain, the maximum token limit for the LlamaCppEmbeddings is defined by the n_ctx parameter, which is set to 512 by default. If your documents contain more than 512 tokens, this could be causing the error.

Here are a few suggestions that might help you resolve this issue:

1. Check the length of your documents: Ensure that none of your documents exceed the maximum token limit of the LlamaCppEmbeddings. You can do this by tokenizing your documents and checking their length before passing them to the Chroma.from_documents() method.

2. Use a different embedding model: You could try using the GPT4AllEmbeddings instead of the LlamaCppEmbeddings. Here's how you can modify your code to do this:

from langchain.embeddings import GPT4AllEmbeddings

# Replace LlamaCppEmbeddings with GPT4AllEmbeddings
llama = GPT4AllEmbeddings()

# Rest of your code remains the same

Please note that you'll need to install the gpt4all package before you can use GPT4AllEmbeddings. You can do this by running pip install gpt4all.

3. Update your llama-cpp-python package: Updating the llama-cpp-python package might resolve the issue. You can do this by running pip install --upgrade llama-cpp-python.

If none of these suggestions work, please provide more information about the documents you're trying to embed. Specifically, it would be helpful to know the average length (in tokens) of your documents.

I hope this helps! Let me know if you have any other questions.

Best,
Dosu

Sources

• Chroma integration .from_documents() isn't working
• Embedding Seems Unsuccessful for Chroma + OpenAI
• LlamaCppEmbeddings not working with faiss
• libs/langchain/langchain/embeddings/llamacpp.py
• libs/langchain/langchain/vectorstores/chroma.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[groovybits]

I got this to work with llamacpp, which seems better to do it all with llama to me.

I also found the gpt4all embeddings conflict with the model used for the llama package, so they don't play well together.

Here is what I did: https://github.com/groovybits/consciousChat/blob/main/characterChat.py#L112

I also did a recursive webpage scrape too.

It's all slow on my macbook vs. the gpt4all or the normal llama run with the same model. I am looking at that, but the results are so correct while the gpt4all embeddings were bringing out useless information and led to inaccurate analysis from what I could tell. It may be from how it is conflicting on the model load so it could be corrupted or something like it warns...

Found model file at  /Users/christi/.cache/gpt4all/ggml-all-MiniLM-L6-v2-f16.bin
objc[75636]: Class GGMLMetalClass is implemented in both /Users/christi/src/consciousChat/test/lib/python3.11/site-packages/llama_cpp/libllama.dylib (0x12c5e71f8) and /Users/christi/src/consciousChat/test/lib/python3.11/site-packages/gpt4all/llmodel_DO_NOT_MODIFY/build/libreplit-mainline-metal.dylib (0x131430228). One of the two will be used. Which one is undefined.
objc[75636]: Class GGMLMetalClass is implemented in both /Users/christi/src/consciousChat/test/lib/python3.11/site-packages/llama_cpp/libllama.dylib (0x12c5e71f8) and /Users/christi/src/consciousChat/test/lib/python3.11/site-packages/gpt4all/llmodel_DO_NOT_MODIFY/build/libllamamodel-mainline-metal.dylib (0x1316dd228). One of the two will be used. Which one is undefined.

[dosubot]

Hi, @HolzerDavid,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. It seems that you were encountering an issue with storing embeddings from llama_cpp_python into a chroma/FAISS vectorstore, and there were detailed discussions in the comments about potential causes of the error and suggested solutions. Additionally, another user, groovybits, shared their experience with getting llamacpp to work and mentioned conflicts with gpt4all embeddings, providing a link to their solution and discussing performance differences between the two models.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!