Feature Request: Proposing User-Customizable RAG Integration in llama.cpp: A Path to Enhanced Contextual Retrieval

[gnusupport]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

After three months of research and experimentation, I've reached a point where implementing embeddings stored in a pgvector column in a PostgreSQL database has proven to be both effective and rewarding. The ability to search by embeddings alone is a game-changer for me. I find it indispensable, even without the processing power of an LLM.

From my understanding, RAG involves the LLM retrieving results from a database—be it files, database column entries, or links—and then generating answers in the context of those retrieved documents. This process is what constitutes a RAG, and I believe I have nearly achieved this in my setup.

What excites me now is the possibility of integrating this RAG functionality directly into the llama.cpp server. I envision it as an external function:

• Users would have the option to enable the RAG feature on command line or in the web UI.
• Users could customize the RAG template through UI options or command line inputs.
• The external function could be defined via a command line parameter, such as:

  llama-server --rag-function fetch-rag.sh
  

• The external function fetch-rag.sh would receive the prompt via its standard input.
• It would then return a list of RAG documents through standard output.
• The exact implementation details of the fetching from external source is not the point; the concept is what's important. There are other ways of doing it, important is that user can decide how.
• With the RAG option enabled, the LLM running on the llama-server would receive a new context based on the prompt. The new context is related to list of documents.
• Subsequently, the llama-server would proceed with the final inference.
• Users would receive a list of relevant documents and RAG results.

In my specific use case, I rely on database entries and customizable URIs, such as <a href="hyperscope:123">My document</a>, which open documents in an external program. Therefore, the RAG input cannot be limited to just files—it must be entirely customizable by the user.

I am eager to contribute US $100 to develop this option, enabling llama.cpp to support user-customizable RAG input. This would ensure that the final output is enriched with context derived from the documents and information gathered before the actual inference.

Motivation

I am motivated to enhance llama.cpp with user-customizable RAG functionality to leverage the full potential of context-aware retrieval and generation, thereby significantly improving the relevance and accuracy of its outputs.

Possible Implementation

See above

[arnfaldur]

This seems fairly out of scope for this project. A RAG can be built using the /embeddings and /completions endpoints of the server for example. But either integrating a database or adding database interfaces for vector lookups seem like feature creep.

There are many other databases that do this better, and there are other tools that integrate LLM inference backends (like llama.cpp) with them to implement RAG.

If you need help getting this done I'd be willing to help but I don't think this is the right place.

[gnusupport]

/embeddings I understand, that is how I create embeddings. Embeddings doesn't make RAG, we can think of it as one part of it to find similar documents.

No, I did not propose for llama.cpp to create or generate RAG, though it would be good.

I have proposed to include a command line option which would call external program to ask for document list that correspond to user's prompt, provided user turned on RAG feature.

[arnfaldur]

I see, I did not read your proposal properly initially. I'm sorry about that.

I still think this functionality makes little sense. Why would the API consumer (program talking to the llama.cpp server) not just pass the prompt to the RAG program and then pass the prompt + RAG context into the model being served with llama.cpp ?

It seems unnecessary and less flexible to integrate that directly into llama-server. To give the user proper control over how the RAG documents are inserted into the prompt you would also need to add some more templating functionality which adds a lot of complexity.

The way I would achieve this is to use the /embeddings endpoint and external software to create the RAG. I would then use the /apply-template endpoint to construct a prompt and finally the /completions endpoint to generate a response with the RAG enhanced context.

[gnusupport]

I was talking about web UI, not API. On API people can do what they want for sure.

Your procedure with the apply-template sounds surely feasible

Though in the web UI we have not get that feature. My concept was to use external program:

• web UI option for RAG would be enabled;
• web UI prompt would be first sent to external program specified on command line --rag-function do-some-search.sh
• whatever is returned from do-some-search.sh would be used as the new prompt with the context;
• web UI would generate new prompt with context answers and references

I understand how things don't make sense to you, I am heavy user of thousands of documents for teaching purposes, and I really dislike searching for third party solutions, while llama.cpp has basically very stable web interface.

[arnfaldur]

AH! I see. It sounded like you wanted something much more integrated than just to be able to have a program augment the prompt. Most of the features you speak of sound like they belong in the WebUI. I suppose it could be extended.

To be clear I think RAG is a perfectly valid thing to do.

[gnusupport]

I am glad that you got the idea.

In relation to RAG, I see that many other applications ask user to upload files. But imagine doing that multiple times for multiple applications, for thousands of files, it sounds impossible.

I do not see a problem that external script runs the path list and creates embeddings in any kind of a database. I am doing that daily with simple functions.

Uploading files requires so much effort.

Then we can develop such external scripts in whatever language, as all what they do, they capture the prompt, convert to embeddings (where model must be running for this reason) and compare that embeddings to the existing one that pinpoint to various documents.

Then they would return back the context to the running LLM for the actual output.

[gnusupport]

I am ready to donate, just tell me.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.