[FEATURE] Option to pass extra parameters to RAG/Tool/XyzProvider via AI service

[langchain4j]

TODO
Currently only memory id can be passed around.
It would be great to be able to pass custom parameters.
For example, to be able to set Filter expression.
Consider passing ChatMemory or List<ChatMessage> as well, so that i cna be accessed from tool method.

[Yellow--]

how about this?

• add annotationUserDatawhich will be used by AiService Interface
• add annotationToolUserDatawhich indicated that relatived parameter in tools is userData
• the userData passed in the AiService method will by bypass to tools and MetaData of RAG

Example

    // the definition of tool
    public static class SendMailTools {
        public void sendEmail(String content, @ToolUserData Map<String, Object> userData) {
            String subject = (String) userData.getOrDefault("subject", "default subject");
            log.info("send email with content: {} userData: {}", content, subject);
        }
    }
  
    //the definition of AiService
    public interface ChatService {
        @UserMessage("summary text ```{{text}}``` and send summary by email")
        String summaryAndSendEmail(@V("text") String text, @UserData Map<String, Object> userData);
    }

    //example
    public static void main(String[] args) {
        ChatService chatService = AiServices.builder(ChatService.class)
                .tools(new SendMailTools())
                .build();
        Map<String, Object> userData = Collections.singletonMap("subject", "customizeSubject");
        chatService.summaryAndSendEmail("a very long text", userData);
    }

Things may need to discuss

• does the annotation name understandable?may be need another name?
• does strict the UserData to Map<String, Object> is good?

@langchain4j

[langchain4j]

@Yellow-- Sounds good!

does the annotation name understandable?may be need another name?

We should consider other names, because this data might not be really a user data, but something related to the tenant or the whole app

does strict the UserData to Map<String, Object> is good?

Ideally we shouldn't restrict the type (as it is done for memory id)

[Yellow--]

@langchain4j
with you advices
the final look may be like this?

    // the definition of tool
    public static class SendMailTools {
        public void sendEmail(String content, @ToolTransparentData Map<String, Object> emailContext) {
            String subject = (String) userData.getOrDefault("subject", "default subject");
            log.info("send email with content: {} userData: {}", content, subject);
        }
    }
  
    //the definition of AiService
    //the ```@TransparentData``` parameter type is not restricted,you could pass any object as you like
    public interface ChatService {
        @UserMessage("summary text ```{{text}}``` and send summary by email")
        String summaryAndSendEmail(@V("text") String text, @TransparentData Map<String, Object> emailContext);
    }

    //example
    public static void main(String[] args) {
        ChatService chatService = AiServices.builder(ChatService.class)
                .tools(new SendMailTools())
                .build();
        Map<String, Object> userData = Collections.singletonMap("subject", "customizeSubject");
        chatService.summaryAndSendEmail("a very long text", userData);
    }

[langchain4j]

A few comments:

1. I would use the same annotation for AI Services method and tool method
2. We need to think of other names for this annotation
3. We should think how to propagate this object into RAG flow (RetrievalAugmentor). One option is to include it in query metadata, but let's consider other options as well

[jmartisk]

We would want to allow the different components (tools and RAG components) to update the contents of these parameters over time, right? So that for example if a query transformer adds something into it, and later the LLM decides to execute a tool, the tool should be able to see it.

In that case, my suggestion would be to introduce a class like PromptContext (or a different name) that is scoped to the processing of a single user prompt.

public static class SendMailTools {
    public void sendEmail(String content, PromptContext ctx) {
        String subject = (String) userData.getOrDefault("subject", "default subject");
        log.info("send email with content: {} userData: {}", content, subject);
    }
}

//the definition of AiService
// it either declares a PromptContext argument or not - if not, then an empty one will be assumed
public interface ChatService {
    @UserMessage("summary text ```{{text}}``` and send summary by email")
    String summaryAndSendEmail(@V("text") String text, PromptContext context);
}

// example for accessing the context in a ContentRetriever
public class ExampleContentRetriever implements ContentRetriever {

    @Override
    public List<Content> retrieve(Query query, PromptContext ctx) {
        String subject = ctx.get("subject");
        // ....
    }
}

//example
public static void main(String[] args) {
    ChatService chatService = AiServices.builder(ChatService.class)
            .tools(new SendMailTools())
            .build();
    PromptContext ctx = PromptContext.builder()
            .put("subject", "a subject")
            .build();
    chatService.summaryAndSendEmail("a very long text", ctx);
}

This would be quite robust I think, but a potential problem with this is that we would have to change the interfaces for RAG components (ContentInjector etc.) to allow passing the PromptContext inside

[langchain4j]

Hi @jmartisk, PromptContext can be a good option, we can include it (or inline it's fields) as a new filed of a Query.Metadata. This will avoid breaking changes.

I am not sure if we should restrict the type though. With the annotation approach, user will be able to pass any object between tools and RAG.

[jmartisk]

Hi @jmartisk, PromptContext can be a good option, we can include it (or inline it's fields) as a new filed of a Query.Metadata. This will avoid breaking changes.

Query.Metadata is only applicable to RAG, no? How would a tool then access it?
Also, for one prompt you potentially get multiple Query objects, so the metadata from them would somehow have to be merged, which complicates things...

I am not sure if we should restrict the type though. With the annotation approach, user will be able to pass any object between tools and RAG.

I'm not restricting the type, just wrapping it into a class that will internally contain Map<String, Object> to achieve better type-safeness and future extensibility (and we won't need to add any specific annotations, because the engine, when it sees a PromptContext argument, will immediately know what to do with it)

[langchain4j]

Related: #1566