[Bug]: Agent sometimes tries to use tools that do not exist and then crashes

[buckmaxwell]

Bug Description

I have a data agent with a collection of tools attached to it. It usually functions fine. Sometimes it tries to use a tool that never existed (makes up a name for it matching some pattern) and then it crashes. My agent is OpenAI

Some example output,

Bot:  Hi there! I noticed that you haven't scheduled your FBI/BCI background check or connected your OCCRRA account yet. Since you have worked in child care before, let's start by connecting your OCCRRA account. You can do this by clicking on the profile tab at the bottom of the screen and then clicking on the "Complete Basic Compliance" button in the "Basic Compliance" section. Let me know if you need any assistance with that!
You: whats an occrra account
=== Calling Function ===
Calling function: occcra_information with args: {}
Traceback (most recent call last):
  File "/Users/maxbuck/Documents/join-tandem/api/sitter/vendor/ai/llamaindex/agenttest.py", line 55, in <module>
    response = agent.chat(query)
  File "/Users/maxbuck/Library/Caches/pypoetry/virtualenvs/ai-IvBGZgFJ-py3.10/lib/python3.10/site-packages/llama_index/agent/openai_agent.py", line 145, in chat
    function_message, tool_output = call_function(
  File "/Users/maxbuck/Library/Caches/pypoetry/virtualenvs/ai-IvBGZgFJ-py3.10/lib/python3.10/site-packages/llama_index/agent/openai_agent.py", line 42, in call_function
    tool = get_function_by_name(tools, name)
  File "/Users/maxbuck/Library/Caches/pypoetry/virtualenvs/ai-IvBGZgFJ-py3.10/lib/python3.10/site-packages/llama_index/agent/openai_agent.py", line 29, in get_function_by_name
    raise ValueError(f"Tool with name {name} not found")
ValueError: Tool with name occcra_information not found

Please not there is a tool called occrra_information (note the difference in spelling). I see no misspelled version of the tool in my project.

Version

0.7.17

Steps to Reproduce

Provoke invocation of a custom tool or non existent tool; intermittently the agent tries to call a tool that does not exist and there is an error.

Relevant Logs/Tracbacks

Calling function: occcra_information with args: {}
Traceback (most recent call last):
  File "/Users/maxbuck/Documents/join-tandem/api/sitter/vendor/ai/llamaindex/agenttest.py", line 55, in <module>
    response = agent.chat(query)
  File "/Users/maxbuck/Library/Caches/pypoetry/virtualenvs/ai-IvBGZgFJ-py3.10/lib/python3.10/site-packages/llama_index/agent/openai_agent.py", line 145, in chat
    function_message, tool_output = call_function(
  File "/Users/maxbuck/Library/Caches/pypoetry/virtualenvs/ai-IvBGZgFJ-py3.10/lib/python3.10/site-packages/llama_index/agent/openai_agent.py", line 42, in call_function
    tool = get_function_by_name(tools, name)
  File "/Users/maxbuck/Library/Caches/pypoetry/virtualenvs/ai-IvBGZgFJ-py3.10/lib/python3.10/site-packages/llama_index/agent/openai_agent.py", line 29, in get_function_by_name
    raise ValueError(f"Tool with name {name} not found")
ValueError: Tool with name occcra_information not found

[dosubot]

Answer generated by a 🤖

Answer

Hi there,

Thank you for bringing this issue to our attention. It seems like the agent is trying to call a tool that doesn't exist, which is causing the crash. This could be due to a typo or a misconfiguration in the tool's name.

From the traceback you provided, it appears that the agent is trying to call a tool named occcra_information, but the actual tool in your project is named occrra_information. This discrepancy in the tool's name could be the cause of the issue.

As per the current design of the LlamaIndex, there is no mechanism that checks for the existence of a tool before attempting to call it. This is something we could consider adding in future versions to prevent such issues.

In the meantime, you could add a check in your code to ensure that the tool exists before trying to call it. Here's an example of how you could modify your code:

def call_function(tools, name, args):
    try:
        tool = get_function_by_name(tools, name)
    except ValueError:
        print(f"Tool with name {name} not found. Skipping.")
        return None, None
    else:
        return tool(args)

In this example, if the tool does not exist, the error will be caught and the tool will be skipped, preventing a crash.

Please note that the behavior of the agent can be unpredictable as it's based on a probabilistic model. The agent might sometimes make mistakes or behave in unexpected ways. We're constantly working on improving the model and your feedback is very valuable in this process.

I hope this helps! Let me know if you have any other questions or need further clarification.

Best,
Dosu

Sources

• llama_index/langchain_helpers/agents/toolkits.py
• llama_index/tools/ondemand_loader_tool.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[sigren]

I've encountered sth similar. In my case, the OpenAI agent prepended "functions." to the actual tool name, which resulted in a failure in the name lookup. This happened with both gpt-3.5-turbo and gpt-4.

I simply told it to stop doing that in the system prompt and it stopped happening.

[dosubot]

Hi, @buckmaxwell,

I'm helping the LlamaIndex team manage their backlog and am marking this issue as stale. From what I understand, the issue was reported by you and describes a bug in version 0.7.17 of the OpenAI data agent, causing intermittent attempts to use a non-existent tool, leading to crashes. In the comments, I provided a potential explanation for the issue, suggesting a discrepancy in the tool's name as the cause. Additionally, sigren shared a similar experience and mentioned resolving the issue by modifying the system prompt.

Could you please confirm if this issue is still relevant to the latest version of the LlamaIndex repository? If it is, please let the LlamaIndex team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation. If you have any further questions or need assistance, feel free to reach out.

I