Foundation for advanced RAG

[langchain4j]

So far, LangChain4j had only a simple (a.k.a., naive) RAG implementation: a single Retriever was invoked on each interaction with the LLM, and all retrieved TextSegments were appended to the end of the UserMessage. This approach was very limiting.

This PR introduces support for much more advanced RAG use cases. The design and mental model are inspired by this article and this paper, making it advisable to read the article.

This PR introduces a RetrievalAugmentor interface responsible for augmenting a UserMessage with relevant content before sending it to the LLM. The RetrievalAugmentor can be used with both AiServices and ConversationalRetrievalChain, as well as stand-alone.

A default implementation of RetrievalAugmentor (DefaultRetrievalAugmentor) is provided with the library and is suggested as a good starting point. However, users are not limited to it and can have more freedom with their own custom implementations.

DefaultRetrievalAugmentor decomposes the entire RAG flow into more granular steps and base components:

• QueryTransformer
• QueryRouter
• ContentRetriever (the old Retriever is now deprecated)
• ContentAggregator
• ContentInjector

This modular design aims to separate concerns and simplify development, testing, and evaluation. Most (if not all) currently known and proven RAG techniques can be represented as one or multiple base components listed above.

Here is how the decomposed RAG flow can be visualized:

This mental and software model aims to simplify the thinking, reasoning, and implementation of advanced RAG flows.

Each base component listed above has a sensible and simple default implementation configured in DefaultRetrievalAugmentor by default but can be overridden by more sophisticated implementations (provided by the library out-of-the-box) as well as custom ones. The list of implementations is expected to grow over time as we discover new techniques and implement existing proven ones.

This PR also introduces out-of-the-box support for the following proven RAG techniques:

• Query expansion
• Query compression
• Query routing using LLM
• Reciprocal Rank Fusion
• Re-ranking (Cohere Rerank integration is coming in a separate PR).

[geoand]

I am just reading up on the article, but so far I like what I see!

cc @cescoffier and @maxandersen who I am sure will be interested in this as well