Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to configure models specifically for RAG (localdocs)? #2226

Closed
Seedmanc opened this issue Apr 16, 2024 · 0 comments
Closed

How to configure models specifically for RAG (localdocs)? #2226

Seedmanc opened this issue Apr 16, 2024 · 0 comments
Labels
documentation Improvements or additions to documentation

Comments

@Seedmanc
Copy link

Meaning the Generation settings area for the model, such as the context size and so on. Can retrieval proficiency be improved by tuning those?

Here are some assumptions, I would expect that the larger the document split size is, the better would be the understanding of the excerpts by the model, but the size X chunks has to fit the context window, therefore setting it as high as the model allows should benefit the retrieval goals, right?

On the other hand, in my experience the chat memory of previous messages often does more harm than good for RAG, and that also depends on the context size. E.g. I've made a list of questions to the models about the topic of my document to quicky access their proficiency, but asking them all in succession confuses the bot, since they're about different parts of the original document and seem random and unrelated to each other. Some models even directly ask me whether I want to continue dicussing previous question or switch topic.
Is there a way to reduce or disable chat memory for the purposes of RAG?

Thirdly, I think temperature should be kept to 0 to make sure the LLM only uses the provided context to answer questions instead of its imagination. However it's hard to notice the effect of this setting as some model continue to hallucinate even at 0 while others fail to come up with anything at high setting.

Any other settings that would affect RAGging I should know about?

@Seedmanc Seedmanc added the documentation Improvements or additions to documentation label Apr 16, 2024
@nomic-ai nomic-ai locked and limited conversation to collaborators Apr 17, 2024
@cebtenzzre cebtenzzre converted this issue into discussion #2229 Apr 17, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

1 participant