Retrievers don't seem to work properly with ollama

[codebanesr]

System Info

I have retriever implementation like this

def get_vector_store(options: StoreOptions) -> VectorStore:
    """Gets the vector store for the given options."""
    vector_store: VectorStore
    embedding = get_embeddings()

    store_type = os.environ.get("STORE")

    if store_type == StoreType.QDRANT.value:
        client = qdrant_client.QdrantClient(
            url=os.environ["QDRANT_URL"],
            prefer_grpc=True,
            api_key=os.getenv("QDRANT_API_KEY", None),
        )

        vector_store = Qdrant(
            client, collection_name=options.namespace, embeddings=embedding
        )

        # vector_store = Qdrant.from_documents([], embedding, url='http://localhost:6333', collection=options.namespace)

    else:
        raise ValueError("Invalid STORE environment variable value")

    return vector_store

get-embeddings.py

         return OllamaEmbeddings(base_url=f"host.docker.internal:11434", model="mistral")

knowledgebase: VectorStore = get_vector_store(StoreOptions("knowledgebase"))
async def get_relevant_docs(text: str, bot_id: str) -> Optional[str]:
    try:
        kb_retriever = knowledgebase.as_retriever(
            search_kwargs={
                "k": 3,
                "score_threshold": vs_thresholds.get("kb_score_threshold"),
                "filter": {"bot_id": bot_id},
            },
        )

        result = kb_retriever.get_relevant_documents(text)

        if result and len(result) > 0:
            # Assuming result is a list of objects and each object has a page_content attribute
            all_page_content = "\n\n".join([item.page_content for item in result])

            return all_page_content

        return None

    except Exception as e:
        logger.error(
            "Error occurred while getting relevant docs",
            incident="get_relevant_docs",
            payload=text,
            error=str(e),
        )
        return None

As long as i use chatgpt embeddings and chat models, i get the correct outputs. Once i switch to ollama, none of my retrievers are working.

I see the documents being ingested to qdrant, which means embeddings are working, but retrievers fail to retrieve any document

Who can help?

No response

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

ss

Expected behavior

retrievers should be able to fetch the documents from qdrant irrespective of embedding models being used

[dosubot]

🤖

Hey @codebanesr! Long time no code. Good to see you again in our corner of the code-verse! 🚀

Based on the information you've provided and the context from the LangChain repository, it seems like the issue might be related to how the get_vector_store function is handling the Ollama embeddings.

The get_vector_store function you've shared is designed to work with a specific type of embeddings and it's not clear if it's compatible with Ollama embeddings. The function retrieves the vector store based on the STORE environment variable and uses the Qdrant client if the STORE is set to QDRANT.value.

However, it's not clear how the function handles the embeddings. The embeddings are passed to the Qdrant class, but it's not clear how the Qdrant class uses these embeddings.

In the case of Ollama embeddings, you need to create an instance of the OllamaEmbeddings class and use the embed_documents or embed_query methods to get the embeddings. Here's an example:

from langchain.embeddings import OllamaEmbeddings
ollama_emb = OllamaEmbeddings(model="llama:7b")
r1 = ollama_emb.embed_documents(["Alpha is the first letter of Greek alphabet", "Beta is the second letter of Greek alphabet"])
r2 = ollama_emb.embed_query("What is the second letter of Greek alphabet")

Each of these methods returns a list of embeddings, one for each document or query.

It's possible that the get_vector_store function or the Qdrant class is not handling these embeddings correctly, which could be why the retrievers are not working when you switch to Ollama.

To troubleshoot this issue, I would recommend checking how the Qdrant class uses the embeddings and if it's compatible with Ollama embeddings. You might also want to check if the get_vector_store function is correctly handling the Ollama embeddings.

If you could provide more information about how the Qdrant class uses the embeddings and how the get_vector_store function handles the Ollama embeddings, it would be easier to identify the cause of the issue.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• libs/langchain/langchain/embeddings/ollama.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[codebanesr]

@dosu-bot

This is the implementation for get_embeddings

from functools import lru_cache
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.embeddings.ollama import OllamaEmbeddings
from .embedding_type import EmbeddingProvider
import os
from langchain.embeddings.base import Embeddings
from utils.get_logger import CustomLogger
import os, warnings


logger = CustomLogger(module_name=__name__)

LOCAL_IP = os.getenv("LOCAL_IP", "host.docker.internal")

def get_embedding_provider():
    """Gets the chosen embedding provider from environment variables."""
    return os.environ.get("EMBEDDING_PROVIDER")

def get_azure_embedding():
    """Gets embeddings using the Azure embedding provider."""
    deployment = os.environ.get("AZURE_OPENAI_EMBEDDING_MODEL_NAME")
    openai_api_key = os.environ.get("AZURE_OPENAI_API_KEY")
    client = os.environ.get("AZURE_OPENAI_API_TYPE")
    openai_api_base = os.environ['AZURE_OPENAI_API_BASE']
    openai_api_version = os.environ['AZURE_OPENAI_API_VERSION']

    return OpenAIEmbeddings(
        openai_api_key=openai_api_key,
        deployment=deployment,
        client=client,
        chunk_size=8,
        openai_api_base=openai_api_base,
        openai_api_version=openai_api_version
    )

def get_openai_embedding():
    """Gets embeddings using the OpenAI embedding provider."""
    openai_api_key = os.environ.get("OPENAI_API_KEY")

    return OpenAIEmbeddings(openai_api_key=openai_api_key)

def choose_embedding_provider():
    """Chooses and returns the appropriate embedding provider instance."""
    embedding_provider = get_embedding_provider()

    if embedding_provider == EmbeddingProvider.azure.value:
        return get_azure_embedding()
    
    elif embedding_provider == EmbeddingProvider.openchat.value:
        logger.info("Got ollama embedding provider", provider=embedding_provider)
        return OllamaEmbeddings(base_url=f"{LOCAL_IP}:11434", model="openchat")
    
    elif embedding_provider == EmbeddingProvider.OPENAI.value or embedding_provider is None:
        if embedding_provider is None:
            warnings.warn("No embedding provider specified. Defaulting to OpenAI.")
        return get_openai_embedding()

    else:
        available_providers = ", ".join([service.value for service in EmbeddingProvider])
        raise ValueError(
            f"Embedding service '{embedding_provider}' is not currently available. "
            f"Available services: {available_providers}"
        )

# Main function to get embeddings
@lru_cache(maxsize=1)
def get_embeddings() -> Embeddings:
    """Gets embeddings using the chosen embedding provider."""
    return choose_embedding_provider()

It uses an environment variable to decide which embedding to pick up

[codebanesr]

In the following
https://python.langchain.com/docs/integrations/vectorstores/qdrant

can i not just replace OpenAiEmbeddings with Ollama embeddings ?

[codebanesr]

Seems to me like the similarity score is much lower when using embeddings other than openai embeddings. Had to reduce the threshold parameter to a much lower number and the retrievers started working.