Best Practice: Knowledge Base, RAG and Custom Modell

[nboehlke]

Hi,

As many have asked in this discussion section on GitHub and Reddit...
What is the best practice for chatting with documents in OpenWebUI?
I'd really like to set up something like a knowledge base wizard/chat with PDF agent.

Having experienced some problems using RAG in OpenWebUI myself, I'd like to ask everyone who has found a working solution for this to share their settings here.
Rag problems are known and f.e. mentioned here #3868

To make this easier, here is a template to fill in (or just paste screenshots):

0. Used Environment Variable:

1. Chosen VectorDB:
   elasticsearch, chroma, milvus, qdrant, opensearch, or pgvector

2. Admin Panel -> Settings -> Documents
   a) Content Extraction Engine
   b) Text Splitter
   c) Chunk Size
   d) Chunk Overlap
   e) Embedding Model Engine
   f) Embedding Model
   f.1) Embedding Batch Size
   g) Full Context Mode (on/off)
   h) Hybrid Search (on/off)
   h.1) Reranking Model
   i) Top K
   j) Minimum Score
   k) RAG Template

3. Knowledge Base
   a) data format (.pdf, .md, .txt, etc)

4. Custom model
   a) System Prompt
   b) Advanced Parameters
   -> name any which aren't "default"

Thank y'all for sharing - let’s (deep)seek the best solution!

@tjbck
If we find a mentionable solution, it could be added to the Docs as best practice

[mahenning]

Currently I'm neck-deep in the RAG topic and Open-WebUI, so I can share my config and some topics:

0: Used Environment Variable: None. Only used the configuration inthe UI.

1: Chosen VectorDB: Chromadb. There are problems that I wrote a fix for in #11876

2: Admin Panel -> Settings -> Documents
a) Content Extraction Engine: Default
b) Text Splitter: Token (Tiktoken)
c) Chunk Size: 512 (beware the maximum for some models, e.g. intfloat/multilingual-e5-base had at the bottom of its info text "Long texts will be truncated to at most 512 tokens.")
d) Chunk Overlap: 64
e) Embedding Model Engine: default (sentence-transformers)
f) Embedding Model: ferrisS/german-english-snowflake-arctic-embed-l-v2.0
f.1) Embedding Batch Size -
g) Full Context Mode (on/off): off
h) Hybrid Search (on/off): on
h.1) Reranking Model: jinaai/jina-reranker-v2-base-multilingual
i) Top K: Here it's a bit tricky, I use a modified version with 2 different k's (one for the embedding model, one for the reranker), details see #11497. k=150, k_reranker=20. With only one k i'd go to 20, depending on the size of the context window (mine is 30k token, ollama default is 2k). Less context window ->lower k.
j) Minimum Score: 0.0
k) RAG Template: Default, but I experiment with a modified query template: #11913

Knowledge Base
a) data format (.pdf, .md, .txt, etc): pdf mostly

Custom model: Chat model is Llama 3.3 70b AWQ on vLLM

[danielrosehill]

I've found that moving to Qdrant has made a significant difference.

Happy to share my little use case too:

My wife and I are expecting our first child in a few months. We have a few parenting books that we really trust.

I've created a knowledge store where I upload TXTs and then connected a model ("Our Parenting Books") to that store.

What I've noticed is that, as is always the case, the more you can chunk your material the better.

The first time I did this I uploaded the entire books in one go and on this approach I separated them by chapters

It seems to make it a bit easier for the model to pull in only what's necessary.

The technical parameter stuff is very complicated and I've tweaked that a bit but I think these fundamentals matter too.

[rgaricano]

Majal Tov Daniel, L’Chaim!

Another way to improve RAG is embedd the text and also a shorter sumarize, this way models have easier to put the info in context and choose/ranking the most apropiate.

For the system to be agile, the key is to find the balance between lexicon and semantics that each RAG needs.

[danielrosehill]

Majal Tov Daniel, L’Chaim!

Another way to improve RAG is embedd the text and also a shorter sumarize, this way models have easier to put the info in context and choose/ranking the most apropiate.

Ha. Thank you!

I was wondering actually if there's any way to script this kind of process.

As in a tool that would take a book, let's say, and chunk it up into chapters, ideally identifying the chapter markers by itself.

Any chance you've come across such a thing?

[rgaricano]

Precisely in this way, indicate as precisely as possible what they need to do, something like:

"I want you to read the book, make chapter summaries with the most relevant aspects of each, and structure it so that the searches I can perform, both in long and short contexts, reflect the most relevant aspects of the text... Rate them according to this criterion:... Don't take them into account:... If you find inconsistencies in the process, show them for verification... At the end, show a list of the processed information, how many tokens the text has been stored in, the weight of each one..."

That's another thing that, at least I, want to compile to apply to different scenarios. There are very good and well-trained agents in these areas.

I don't think that some many tools or chunkes is essential, a big embed text could be rechunked and smalls chunkes could be remixed by a well instructed model.

[mahenning]

As in a tool that would take a book, let's say, and chunk it up into chapters, ideally identifying the chapter markers by itself.

https://python.langchain.com/docs/how_to/markdown_header_metadata_splitter/ could go in that direction if the book is converted into markdown. But I doubt that it will be used here, as it requires a very specific text structure that other users and their documents don't have. Maybe if someday open-webui allows to connect a custom configured RAG pipeline instead of handling it on its own, but I don't think there are plant for that.

[apunkt]

Qdrant as a vectorstore is truely awesome. However I wouldn't recommend to use it with openwebui (yet).
The way openwebui handels files and knowledges is completely against qdrant best practices [qdrant: single collection, many tenants || openwebui: collection for everything] as discussed in #11976

[danielrosehill]

Hi,

As many have asked in this discussion section on GitHub and Reddit... What is the best practice for chatting with documents in OpenWebUI? I'd really like to set up something like a knowledge base wizard/chat with PDF agent.

Having experienced some problems using RAG in OpenWebUI myself, I'd like to ask everyone who has found a working solution for this to share their settings here. Rag problems are known and f.e. mentioned here #3868

To make this easier, here is a template to fill in (or just paste screenshots):

0. Used Environment Variable:

1. Chosen VectorDB:
   elasticsearch, chroma, milvus, qdrant, opensearch, or pgvector

2. Admin Panel -> Settings -> Documents
   a) Content Extraction Engine
   b) Text Splitter
   c) Chunk Size
   d) Chunk Overlap
   e) Embedding Model Engine
   f) Embedding Model
   f.1) Embedding Batch Size
   g) Full Context Mode (on/off)
   h) Hybrid Search (on/off)
   h.1) Reranking Model
   i) Top K
   j) Minimum Score
   k) RAG Template

3. Knowledge Base
   a) data format (.pdf, .md, .txt, etc)

4. Custom model
   a) System Prompt
   b) Advanced Parameters
   -> name any which aren't "default"

Thank y'all for sharing - let’s (deep)seek the best solution!

@tjbck If we find a mentionable solution, it could be added to the Docs as best practice

Great idea re: documenting ideas for implementation too.

The only question that I have regarding RAG that I haven't. thought of an answer to is this:

How does OpenWebUI handle the actual embedding and retrieval with the vector stores?

Let's say that I want to upload a mixture of CSV data and books as mentioned above.

The same set of embedding settings is not going to be optimal for both of these data stores.

With the ability to only set one set of parameters, what's the recommendation or thinking as to how to handle that?