Is it possible to inject RAG retrieved documents as SystemMessage instead of UserMessage

[Nicht1]

Hi, @dliubarskyi
I have a question about the RAG content injection mechanism in LangChain4j.
Current Behavior:
As I understand, the retrieved documents from RAG are currently injected into the UserMessage by default through the ContentInjector.
What I Want to Achieve:
I would like to convert the RAG retrieved documents into a SystemMessage instead of appending them to the UserMessage. My goal is to:

Inject retrieved documents as SystemMessage
Persist these messages to a database through my custom ChatMemoryStore implementation
Maintain the RAG context in chat history for future reference

My Question:

Is this approach feasible with the current architecture?
Is there a recommended way to achieve this in LangChain4j?
Would this be considered a best practice, or are there potential issues I should be aware of?

Use Case:
I'm building a system where I need to preserve the RAG context as part of the conversation history, and storing it as SystemMessage seems more semantically appropriate than mixing it with user input.
Any guidance would be greatly appreciated!
Thanks!

[dliubarskyi]

Hi, this is not possible to do OOTB with the current implementation, but you can do some tricks to make it work.
Please keep in mind that there is usually a single SystemMessage for the entire conversation, so if you are going to put RAG content (that is retrieved before each LLM request) inside the system message, this meand that RAG content from previous iterations will be overriden. If this is intended, you can try using https://docs.langchain4j.dev/tutorials/ai-services#programmatic-chatrequest-rewriting and manipulate messages that are sent to the LLM. You could cut RAG content from UserMessage s and insert it into the SystemMessage. This will not be reflected in ChatMemory though, but at least LLM will see RAG content inside the system message, if it is your intention.

[Nicht1]

thanks

[dliubarskyi]

BTW, the downside of this strategy is that if you have a long conversation history and change system message on every LLM request, you lose the benefit of prompt caching that many LLMs support. They usually cache data in this order: system message -> tools -> other messages.

So if you change system message, the whole request will not be cached, only the part of system message that goes before the changed RAG content can be taken from the cache