Multi input tools

[haesleinhuepf]

Hey @kevinyamauchi ,

thanks again for working on this ! This PR closes #7 and closes #1

To prevent markdown-images (![]()), I'm using a regexp now. It was hard to prevent this output otherwise.

I reran all notebooks and modified tiny bits. I have the impression that the new agent is smarter than the old. Two examples:

• I've seen multiple cases where I asked "segment the image" and it also showed the image.
• In the segmentation_algorithms notebook, it was hard to fool Bob. It always used the right algorithm for the membranes.tif image, presumably because the name of the file (membranes) appears in the description of an algorithm. I had to rename the image to demonstrate how to make it choose the right algorithm.

Fantastic work! Do you see any other reason for not merging this?

Thanks again!

Best,
Robert

[haesleinhuepf]

I'm playing mroe with it and see that the agent tends to execute tasks multiple times (such as showing images) and sometimes even runs in [endless] loops.

Edit: I think this is fixed in #9

[haesleinhuepf]

I love this

[kevinyamauchi]

Edit: I think this is fixed in #9

Nice! I'll have a look tomorrow. In case it's not, I found this comment that might be useful:

langchain-ai/langchainjs#1290 (comment)

[kevinyamauchi]

I love this

Dude, yeah, this is super exciting!