Support for LLama cpp

[GaneshMystic]

🚀 The feature

llama cpp is one of the widely used LLM inference tool that has native support for more than 95% GGUF models.

Although llama server supports open ai style API, its hard to run Main LLM and embedding model.

Python Library llama-cpp-python allows to run both models simultaneously. It will be really useful if there is native support for llama cpp using the llama-cpp-python library.

Motivation, pitch

llama cpp also supports Embedding models which makes it suitable for local and on device (EDGE) AI. Extending support to llama cpp will encourage local and edge ai use case to adapt mem0 capabilities.

[mimi3421]

Using mem0 with llama.cpp through OpenAI API may be something like this if someone just want to have a try and come across this issue.

Start the llama-server like this:

# --jinja -fa: Enable tool call for something like MCP
#  --pooling last: Important, tell the server which pooling type for embeddings is used to export
# If needed, just host another model specilized for embedding on another port, or start the llama-server with multi-model configuration.

llama-server --jinja -fa  \
    --pooling last \
    --model phi-4.gguf 

And then start the mem0 with the configuration like this:

from mem0 import Memory
config = {
    "vector_store": {
        "provider": "qdrant",
        "config": {
            "collection_name": "test",
            "host": "localhost",
            "port": 6333,
            "embedding_model_dims": 5120 # Set as what the embedding model request. 
        }
    },
    "embedder": { 
        "provider": "openai",
        "config": {
            "model": "phi-4.gguf", # Choose a specialized model for embedding as 5120 is too large for searching purpose.
            "api_key":"asdkfjkl", # No api-key is needed for local server, just add some random text.
            "openai_base_url":"http://localhost:8080/v1", 
            "embedding_dims": 5120 # Set as what the embedding model request. 
        }
    },
    "llm": {
        "provider": "openai",
        "config": {
            "model": "phi-4.gguf",
            "api_key":"asdkfjkl", 
            "openai_base_url":"http://localhost:8080/v1",
            "temperature": 0.8,
            "max_tokens": 8000,
            "top_p": 0.95,
            "top_k": 40
        }
    }
}
m = Memory.from_config(config)
m.add("Likes to play cricket on weekends", user_id="alice", metadata={"category": "hobbies"}) # Test the availability