Implement EvalQueryEngineTool

[d-mariano]

Description

Notice

I would like input on this PR from the llama-index team. If the team agrees with the need and approach, I will provide unit tests, documentation updates, and Google Colab notebooks as required.

Summary

The reason for this change is to provide a plug-and-play method of using on-the-fly evaluation for tools within an agent. Some areas I would like feedback on:

1. Handling this on the tool-level makes sense for the use-case
2. This is a good approach for addressing the use-case, and aligns with the llama-index vision
3. Anything else you can think of! <3

If the team decides this is a good approach, I will provide unit tests, documentation updates, and Google Colab notebooks as required.

Implements the EvalQueryEngineTool:

• Inherits from QueryEngineTool
• Uses a given evaluator to evaluate query engine responses
• If the response fails evaluation:
  • The tool output is manipulated to deter the LLM from using it
  • The evaluator feedback is used as a reason for failure

Type of Change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Added new unit/integration tests
• Added new notebook (that tests end-to-end)
• I stared at the code and made sure it makes sense

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[logan-markewich]

@d-mariano Yea I think I like this approach. If this was combined with an agent that had the chance to "retry" tool calls, it would be pretty powerful (and also helps avoid tracebacks when calling a tool)

[d-mariano]

@d-mariano Yea I think I like this approach. If this was combined with an agent that had the chance to "retry" tool calls, it would be pretty powerful (and also helps avoid tracebacks when calling a tool)

@logan-markewich great! And totally agree, that's the idea down the road. I would like to test this with the simple ReAct implementation. Then, after that, test it with more complex use-cases like within a query pipeline.

Also curious what implications this has on a QueryPlan, but that doesn't need to be addressed today.

Okay, great. So with this, I can implement some UTs, of course...should I introduce this in the docs with a notebook example as well? <3

[logan-markewich]

@d-mariano yea may as well make this PR a little more complete 💪

[d-mariano]

@logan-markewich I've made a few updates so far:

1. I corrected the change to update ToolOutput.content rather than ToolOutput.raw_output:
   1. content is injected into agent prompts and the entire point of this change is to inform agents on evaluation failures
   2. Preserving raw_output is probably desirable for debug purposes? I.e "This tool failed eval, but this was its original output."
2. I've added a test suite covering evaluation pass and fail modes

I would like to update the llama-index documentation with a section for EvalQueryEngine. Here I can write up a blurb on the purpose of it and even include a notebook example of it being used with an agent. Are these docs included in this repo? (I haven't looked around, I can also do that at some point.)

I need to get at some other stuff so I'll have this open a bit longer.

[logan-markewich]

@d-mariano yea, there's a docs folder in this repo 💪 it's all markdown.

There should be some existing evaluation sections you can maybe hook into? Or maybe the section on tools would be better

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB