feature/introducting-conversational-retrieval-tool-agent

[niztal]

The ConversationalQARetriverChain is a very helpful chain in order to have a RAG based on VectorStore Retriever.

One of his major capabilities are querying the vectorstore by the user's input and calling the LLM using the query's results as a placeholder of {context} and combining this {context} as part of the LLM's prompt.

But since it's just a chain it doesn't have the capability to combine Tools.

The only options are (as described over Bug & Question)

1. Combine a flow based on some Agent (e.g. ToolAgent) with a Chain Tool that connects as a base chain to the ConversationalQARetriverChain - This is a bug which currently not working. I was trying to fix it but it was too cumbersome.
2. Having an LLM chain with a VectorStoreRetreiverMemory (based on a solution provided by langchain) - It didn't provided the needed solution, it's just using the VectorStore (e.g. pinecone) as a memory for the LLM Chain. In general I think that's a good feature to have on flowise I can have another dedicated PR for that (FYI @HenryHengZJ)
3. Basing on an existing Agent (e.g. ToolAgent) but enhancing the agent with context, Vector Store Retriever and {context} placeholder as part of the system message

The 3rd option works the best 🚀

@HenryHengZJ please review I would love to get any feedback from you or any other 🙏

[niztal]

@HenryHengZJ can you please review it? I'd really love to get your feedback about it.

Thanks

[HenryHengZJ]

@niztal thanks for the suggestion!

Question: why do we need the context in system message when user can use retriever tool? From my testing, including the context in system message doesn't work well. For example, I was asking how to install flowise using the following:

You can see the correct installation context was placed in the system message, but still it replies I dont know.

Trace link here - https://smith.langchain.com/public/6e42424b-29b9-4df2-b14c-073bf2329ccd/r

[HenryHengZJ]

As comparison to using Retriever Tool + Agent, you have much better response:

https://smith.langchain.com/public/c9dec40b-f36b-4cfb-898a-8141c09714d4/r

[niztal]

Hi @HenryHengZJ,

Thanks for your feedback I really appreciate that and I was anticipating that 👍

I was investigating your analysis, before answering your great analysis, I think there was some mis-understanding, please let me know if I miss anything, but based on the langsmith reports you provided seems like you manually set the context within the system message.

If that's correct, that was not what I mentioned by initiating this new agent. You need to leave the {context} as a placeholder (same as in the Conversational Retrieval QA Chain). The agent would know to fulfill that once needed by querying the Vector Store and prompting the LLM model.

In general, I would say that, this agent's main purpose is to have a bot (RAG) based on Specific knowledge-base that the model (in your case GPT 3.5) was not trained on. In your case GPT3.5 is already familiar with flowise since I guess it was trained on parsing the web.
You can see it by using ChatGPT w/o any needed additional knowledge-base

If you would like to have a bot with some specific instructions + specific knowledge-base (a.k.a "context") + executing some tools, at least per my knowledge it is not doable w/o my agent.

Thanks,

[HenryHengZJ]

The {context} is being used as placeholder in system message. Upon asking question, it will look for vector store and retrieve documents, then fill it in, thats what you saw from the trace.

To create a bot that has access to specific knowledge, typically you will use Retriever Tool for that - https://docs.flowiseai.com/use-cases/multiple-documents-qna#agent. And specify instruction in the System Message like:

You are an expert financial analyst that always answers questions with the most relevant information using the tools at your disposal.
These tools have information regarding companies that the user has expressed interest in.
Here are some guidelines that you must follow:
* For financial questions, you must use the tools to find the answer and then write a response.
* Even if it seems like your tools won't be able to answer the question, you must still use them to find the most relevant information and insights. Not using them will appear as if you are not doing your job.
* You may assume that the users financial questions are related to the documents they've selected.
* For any user message that isn't related to financial analysis, respectfully decline to respond and suggest that the user ask a relevant question.
* If your tools are unable to find an answer, you should say that you haven't found an answer but still relay any useful information the tools found.
* Dont ask clarifying questions, just return answer.

The tools at your disposal have access to the following SEC documents that the user has selected to discuss with you:
- Apple Inc (APPL) FORM 10K 2022
- Tesla Inc (TSLA) FORM 10K 2022

The current date is: 2024-01-28

Including the context in the system message usually dont work well because LLM tends to lose focus as the token increase

[niztal]

The {context} is being used as placeholder in system message. Upon asking question, it will look for vector store and retrieve documents, then fill it in, thats what you saw from the trace.

To create a bot that has access to specific knowledge, typically you will use Retriever Tool for that - https://docs.flowiseai.com/use-cases/multiple-documents-qna#agent. And specify instruction in the System Message like:

You are an expert financial analyst that always answers questions with the most relevant information using the tools at your disposal.
These tools have information regarding companies that the user has expressed interest in.
Here are some guidelines that you must follow:
* For financial questions, you must use the tools to find the answer and then write a response.
* Even if it seems like your tools won't be able to answer the question, you must still use them to find the most relevant information and insights. Not using them will appear as if you are not doing your job.
* You may assume that the users financial questions are related to the documents they've selected.
* For any user message that isn't related to financial analysis, respectfully decline to respond and suggest that the user ask a relevant question.
* If your tools are unable to find an answer, you should say that you haven't found an answer but still relay any useful information the tools found.
* Dont ask clarifying questions, just return answer.

The tools at your disposal have access to the following SEC documents that the user has selected to discuss with you:
- Apple Inc (APPL) FORM 10K 2022
- Tesla Inc (TSLA) FORM 10K 2022

The current date is: 2024-01-28

Including the context in the system message usually dont work well because LLM tends to lose focus as the token increase

ok I got your point, one small question:

Does the system message is part of the LLM's prompt and if so does Flowise/LC send it over and over again each iteration with the model? If so, it's seems pretty expensive from tokens' perspective, isn't it?

Thank you.

[HenryHengZJ]

The whole conversation including system message is always sent to the OpenAI API as messages array - https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages

Its the same as if you were to use OpenAI API directly

[toi500]

@HenryHengZJ just a quick clarification. The main reason of this PR is because, for some reason, the Chain Tool is not working properly when is attached to a Conversational Retrieval QA Chain as I reported here #2400

So, is the Chain Tool bugged or the fact that it does not work with Conversational Retrieval QA Chain is a technical limitation?

I have some people asking me about this on discord too.

[HenryHengZJ]

@toi500 we introduced a new tool - Chatflow Tool:

Can you try create a seperate chatflow that has Conversational Retrieval QA Chain, then link it via this new tool?

[toi500]

@HenryHengZJ that is an amazing tool, did not know that you added it.

Unfortunately, it doesn't work properly according to my testing. The Tool Agent is able to ask the correct question to the Chatflow Tool, but it receives no response.

Of course, I tested if the other flow works and there was not any problem.

[niztal]

@HenryHengZJ that is an amazing tool, did know that you added it.

Unfortunately, it doesn't work properly according to my testing. The Tool Agent is able to ask the correct question to the Chatflow Tool, but it receives no response.

Of course, I tested if the other flow works and there was not any problem.

@toi500 please take my commit and test it as well using my agent, I wonder if it'll work for your scenario. I admit I'm not an expert in langchainso I don't know precisely to explain g why but all the tests I've done including the ones offered by @HenryHengZJ , ,unfortunately wasn't accurate enough in comparison to the flow I've done via my agent.

[soumyabhoi]

It works for me but slow to get response from other chain when using chatflow tool. I wish a agent with a capability of QA and tool calling would be really beneficial.

[toi500]

It works for me but slow to get response from other chain when using chatflow tool. I wish a agent with a capability of QA and tool calling would be really beneficial.

Can you share a capture of the toolOutput?

I am asking cos OpenAI will made it up to fullfill a response, even if it gets nothing from the Chatflow Tool -> Conversational Retrieval QA Chain

[HenryHengZJ]

It definitely works for me though:

1.)

2.)

3.) the chatflow being called:

4.)

Check if your chatflow has API key protected, if yes you need to create credential

[niztal]

It works for me but slow to get response from other chain when using chatflow tool. I wish a agent with a capability of QA and tool calling would be really beneficial.

@soumyabhoi this is exactly what my agent here is doing, you're welcome to try it and let us know. It works for me perfect

[toi500]

@HenryHengZJ just inform that after a lot of testing, the new Chatflow Tool, which is amazing BTW, works. I tried in local and Render and worked as expected.

However It does not work in my production deploment on Railway. I will try to contact them as see what they say. I am getting the following error:

FetchError: invalid json response body at https://flowise-production-8ec4.up.railway.app/api/v1/prediction/9b7b16cf-e649-4e32-9f0d-41a157457402 reason: Unexpected token '<', "<!DOCTYPE "... is not valid JSON

[HenryHengZJ]

yeah I noticed that on Railway too, debugging. But is using Chatflow Tool + Tool Agent fits your purpose? Does it achieves what you wanted to do in first place?

Edit: fix for not working on Railway - #2482

[niztal]

@HenryHengZJ @toi500

I was "playing" with this new tool for almost two days, I must say it may produce high value in some cases and sounds like this tool can be very effective for many people.
Having said that, for my specific use case, my agent still did a better job, and I'll explain:

1. From performance/latency perspective it took ~7-8 seconds slower to answer, for me and for my users' experience it's crucial.
2. From tokens perspective I feel like having a single system message rather than two would be more cost effective.
3. From maintenance perspective, at least for me it was much easier to use one single flow rather than "jumping" from one chatflow to another.
4. From quality perspective, my agent answered better answers and used my tools much more effective.
5. In my case, I need to "hand over" one crucial parameter for one of my tools (function) from the QA chatflow to the main chatflow and unfortunately it didn't work properly so my main agent had to hallucinate this param 😢

Again, it may be individually for my own use-case but I rather leave my agent and use it, if you feel like it's redundant for most of the people we can close this PR and I'll use it only on my forked repo.

Thanks anyway, highly appreciated

[toi500]

@niztal, I think your new agent is a great addition to Flowise. I haven't had a chance to use it yet, but I definitely want to.

I never quite understood why Chains can't use tools. Maybe it's a limitation of the LangChain framework, but I'm sure there's a good reason for that.

In any case, the Chatflow tool, along with LangGraph opens up a whole new paradigm here on Flowise. However, I still think it would be incredibly useful to have either your agent or the Chain tool fixed. (I find the latency of the Chatflow tool a bit too much)

Personally, I'm very grateful for all your hard work here to help the comunity and this project.

[HenryHengZJ]

@niztal yep totally get it, and really appreciate you putting the effort on creating this new agent! We're working on a new plugin system that allow community nodes, and you will also get the attribution tag on the node. I think thats a better place to put it, as we also want to encourage more community nodes from you guys! Will leave this PR open for now, until we have that plugin feature in place, then we can migrate it over!

[niztal]

@niztal yep totally get it, and really appreciate you putting the effort on creating this new agent! We're working on a new plugin system that allow community nodes, and you will also get the attribution tag on the node. I think thats a better place to put it, as we also want to encourage more community nodes from you guys! Will leave this PR open for now, until we have that plugin feature in place, then we can migrate it over!

Thanks @HenryHengZJ highly appreciated, that sounds perfect, LMK when this feature is alive.

[soumyabhoi]

I have been desperately waiting for it from last 1 month and i am unable to lunch into production or beta without it. Any alternative will be immense help. Seems like race against time to have an agent that both can chat and call tools.

[niztal]

I have been desperately waiting for it from last 1 month and i am unable to lunch into production or beta without it. Any alternative will be immense help. Seems like race against time to have an agent that both can chat and call tools.

@soumyabhoi
You're welcome to try my agent for me it works amazing.

I guess once we'll have the community nodes that @HenryHengZJ mentioned above it will globally available. For now I'm using it on my own version, I guess if it fits your use case you can do it as well. LMK if you need any help please.

Thanks

[toi500]

@soumyabhoi, until the new plugin is released where the community can add custom nodes (like this one), you can use the new Chatflow Tool that allows the Agent Tool to manage a whole chatflow (where you host your RAG).

https://docs.flowiseai.com/integrations/langchain/tools/chatflow-tool

[soumyabhoi]

thanks for the suggestion @niztal and @toi500 .. appreciate your effort. I am going with chatflow tool for our beta phase and will switch to new agent when community node will be lunch.