Designing a Tool to interface any Python custom function #832
Comments
|
i would reccomend doing something like this: eg being very explicit about the input format |
|
lmk if this works. if it does, im going to keep this issue open and add a decorator to make this more easy |
|
Oh, got it! The description says the LLM should format JSON structured data. If it works, this may be a way to interface with any Python function. I'll give it a try and get back to you. Thanks! |
|
Thanks Harrison! It seems working! I updated the 2 modules following your suggestions: #
# weather_data
# is an example of a custom python function
# that takes a list of custom arguments and returns a text (or in general any data structure)
#
def weather_data(where: str = None, when: str = None) -> str:
'''
given a location and a time period, this custom function
returns weather forecast description in natural language.
This is a mockup function, returning a fixed text tempalte.
The function could wrap an external API returning realtime weather forecast.
parameters:
where: location as text, e.g. 'Genova, Italy'
when: time period, e.g. 'today, now'
returns:
weather foreast description as flat text.
'''
if where and when:
# return a fake/hardcoded weather forecast sentence
return f'in {where}, {when} is sunny! Temperature is 20 degrees Celsius.'
elif not where:
return 'where?'
elif not when:
return 'when?'
else:
return 'I don\'t know'
def weather(json_request: str) -> str:
'''
Takes a JSON dictionary as input in the form:
{ "when":"<time>", "where":"<location>" }
Example:
{ "when":"today", "where":"Genova, Italy" }
Args:
request (str): The JSON dictionary input string.
Returns:
The weather data for the specified location and time.
'''
arguments_dictionary = json.loads(json_request)
where = arguments_dictionary["where"]
when = arguments_dictionary["when"]
return weather_data(where=where, when=when)
#
# instantiate the langchain tool.
# The tool description instructs the LLM to pass data using a JSON.
# Note the "{{" and "}}":
# this double quotation is needed to avoid a runt-time error triggered by the agent instatiation.
#
name = "weather"
request_format = '{{"when":"<time>","where":"<location>"}}'
description = f'helps to retrieve weather forecast. Input should be JSON in the following format: {request_format}'
# create an instance of the custom langchain tool
Weather = Tool(name=name, func=weather, description=description)
if __name__ == '__main__':
print(weather_data(where='Genova, Italy', when='today'))
# => in Genova, Italy, today is sunny! Temperature is 20 degrees Celsius.
print(weather('{ "when":"today", "where":"Genova, Italy" }'))
# => in Genova, Italy, today is sunny! Temperature is 20 degrees Celsius.
# print the Weather tool
print(Weather)#
# weather_agent
#
import argparse
from langchain.agents import initialize_agent
from langchain.llms import OpenAI
from langchain import LLMChain
from langchain.prompts import PromptTemplate
# import custom tools
from weather_tool import Weather
llm = OpenAI(temperature=0)
prompt = PromptTemplate(
input_variables=[],
template="Answer the following questions as best you can."
)
# Load the tool configs that are needed.
llm_weather_chain = LLMChain(
llm=llm,
prompt=prompt,
verbose=True
)
tools = [
Weather
]
# Construct the react agent type.
agent = initialize_agent(
tools,
llm,
agent="zero-shot-react-description",
verbose=True
)
#
# get the question asa a command line argument.
# $ py weather_agent.py --question "What about the weather today in Genova, Italy"
#
parser = argparse.ArgumentParser(description='agent using a weather forecast custom tool')
parser.add_argument('-q', '--question', dest='question', type=str, help='question to submit to the agent. Enclose the question sentence in quotes.', required=True)
args = parser.parse_args()
# run the agent
agent.run(args.question)TestsRuntime Test 1The question contains all info (where and when) tool requires to retrieve data => PASSED (SUCCESSFUL)!
Runtime Test 2The question do not contains the "when" argument => PASSED (WARNING). The LLM guess the "when"="now", that in this case makes sense, but in general this is a possible confabulation trend.
Runtime Test 3The question contains all info (where and when) tool requires to retrieve data => PASSED (SUCCESSFUL)! Interesting, the LLM correct the grammar of the sentence returned by the tool; that's perfect!
Temporary conclusionsAs you suggested in your previous comment, maybe you could optionally add a decorator to make the function-interface custom tool more easy to build. Maybe adding an example on the documentation page: https://langchain.readthedocs.io/en/latest/modules/agents/examples/custom_tools.html It worth nothing that in general a tool could get
Thanks |
UPDATE 2I explored the last thought of my previous comment; A generic tool (that wrap any python function) could:
Summary
A proof-of-concept trivial demoI extended my previous example, now including 2 tools:
So the updated agent now implement a react pattern using these 2 tools.
weather_tool.py#
# weather_tool.py
# This module contains all ingredients to build a langchain tool
# that incapsule any custom function.
#
import json
from langchain.agents import Tool
#
# weather_data
# is an example of a custom python function
# that takes a list of custom arguments and returns a text (or in general any data structure)
#
def weather_data(where: str = None, when: str = None) -> str:
'''
given a location and a time period, this custom function
returns weather forecast description in natural language.
This is a mockup function, returning a fixed text tempalte.
The function could wrap an external API returning realtime weather forecast.
parameters:
where: location as text, e.g. 'Genova, Italy'
when: time period, e.g. 'today, now'
returns:
weather foreast description as flat text.
'''
if where and when:
# return a fake/hardcoded weather forecast sentence
return f'in {where}, {when} is sunny! Temperature is 20 degrees Celsius.'
elif not where:
return 'where?'
elif not when:
return 'when?'
else:
return 'I don\'t know'
def weather(json_request: str) -> str:
'''
Takes a JSON dictionary as input in the form:
{ "when":"<time>", "where":"<location>" }
Example:
{ "when":"today", "where":"Genova, Italy" }
Args:
request (str): The JSON dictionary input string.
Returns:
The weather data for the specified location and time.
'''
arguments_dictionary = json.loads(json_request)
where = arguments_dictionary["where"]
when = arguments_dictionary["when"]
result = weather_data(where=where, when=when)
return result
#
# instantiate the langchain tool.
# The tool description instructs the LLM to pass data using a JSON.
# Note the "{{" and "}}": this double quotation is needed to avoid a runt-time error triggered by the agent instatiation.
#
name = "weather"
request_format = '{{"when":"<time>","where":"<location>"}}'
description = f'helps to retrieve weather forecast. Input should be JSON in the following format: {request_format}'
# create an instance of the custom langchain tool
Weather = Tool(name=name, func=weather, description=description)
if __name__ == '__main__':
print(weather_data(where='Genova, Italy', when='today'))
# => in Genova, Italy, today is sunny! Temperature is 20 degrees Celsius.
print(weather('{ "when":"today", "where":"Genova, Italy" }'))
# => in Genova, Italy, today is sunny! Temperature is 20 degrees Celsius.
# print the Weather tool
print(Weather)datetime_tool.py#
# datetime_tool.py
# This module contains all ingredients to build a langchain tool
# that incapsule any custom function.
#
import datetime
import json
from langchain.agents import Tool
def time():
# Get the current time
current_time = datetime.datetime.now()
# Format the time as a string in a local format
local_time = current_time.strftime("%I:%M %p")
return local_time
def date():
# Get the current time
current_time = datetime.datetime.now()
# Format the time as a string in a local format
local_time = current_time.strftime("%A, %B %d, %Y")
return local_time
def fulldate():
# Get the current time
current_time = datetime.datetime.now()
# Format the time as a string in a local format
local_time = current_time.strftime("%A, %B %d, %Y %I:%M %p %Z")
return local_time
def datetime_tool(request: str = 'now') -> str:
'''
Example:
{ "when":"today", "where":"Genova, Italy" }
Args:
request (str): optional/not used.
Returns:
date and time as a JSON data structure, in the format:
'{{"fulldate":"<fulldate>","date":"<date>","time":"<time>"}}'
'''
data = {
'fulldate': fulldate(),
'date': date(),
'time': time()
}
response_as_json = json.dumps(data)
return response_as_json
#
# instantiate the langchain tool.
# The tool description instructs the LLM to pass data using a JSON.
# Note the "{{" and "}}": this double quotation is needed to avoid a runt-time error triggered by the agent instatiation.
#
name = "datetime"
response_format = '{{"fulldate":"<fulldate>","date":"<date>","time":"<time>"}}'
description = f'helps to retrieve date and time. Output is a JSON in the following format: {response_format}'
# create an instance of the custom langchain tool
Datetime = Tool(name=name, func=datetime_tool, description=description)
if __name__ == '__main__':
print(fulldate())
# => Tuesday, January 24, 2023 11:19 AM
print(date())
# => Tuesday, January 24
print(time())
# => 11:19 AM
print(datetime_tool())
# =>
print(Datetime)tools_agent.py#
# tools_agent.py
#
# zero-shot react agent that reply questions using available tools
# - Weater
# - Datetime
#
# get the question as a command line argument (a quoted sentence).
# $ py tools_agent.py What about the weather today in Genova, Italy
#
import sys
from langchain.agents import initialize_agent
from langchain.llms import OpenAI
from langchain import LLMChain
from langchain.prompts import PromptTemplate
# import custom tools
from weather_tool import Weather
from datetime_tool import Datetime
llm = OpenAI(temperature=0)
template='''\
Please answer the following questions with precision. \
If you are unable to find the required information after seeking assistance, \
please indicate that you do not know.
'''
prompt = PromptTemplate(input_variables=[], template=template)
# debug
# print(prompt.format())
# sys.exit()
# Load the tool configs that are needed.
llm_weather_chain = LLMChain(
llm=llm,
prompt=prompt,
verbose=True
)
tools = [
Weather,
Datetime
]
# Construct the react agent type.
agent = initialize_agent(
tools,
llm,
agent="zero-shot-react-description",
verbose=True
)
if __name__ == '__main__':
if len(sys.argv) > 1:
question = ' '.join(sys.argv[1:])
print('question: ' + question)
# run the agent
agent.run(question)
else:
print('agent that answers questions using Weather and Datetime custom tools')
print('usage: py tools_agent.py <question sentence>')
print('example: py tools_agent.py what time is it?')Usage testsTest 1:
|
solyarisoftware
changed the title
UPDATE 3i modified the tool behavior to mitigate agent's inventions:
#
# weather_tool.py
# builds a langchain tool that incapsules a custom function
# that retrieve weather forecasts data
#
import json
from typing import List
from langchain.agents import Tool
#
# weather_data_retriever
# is an example of a custom python function
# that takes a list of custom arguments and returns a text (or in general any data structure)
#
def weather_data_retriever(where: str = None, when: str = None, required_data: List[str] = []) -> str:
'''
given a location and a time period, this custom function
returns weather forecast as a data structure (in JSON format).
This is a mockup function, returning a fixed text tempalte.
The function could wrap an external API returning realtime weather forecast.
parameters:
where: location as text, e.g. 'Genova, Italy'
when: time period, e.g. 'today, now'
returns:
weather foreast description as a JSON. E.g.
{"forecast": "sunny all the day", "temperature": "20 degrees Celsius"}
'''
if where and when:
# this function is a mockup, returns fake/hardcoded weather forecast data
data = {
'forecast': 'sunny',
'temperature': '20 degrees Celsius'
}
if not where:
data['where'] = 'location is not specified'
if not when:
data['when'] = 'date is not specified'
# if required variable names are not included in the data section,
# the attribute is added to the dictionary with value I don't know.
for variable_name in required_data:
if variable_name not in data.keys():
data[variable_name] = 'unknown'
return json.dumps(data)
def weather(json_request: str) -> str:
'''
Takes a JSON dictionary as input in the form:
{ "when":"<time>", "where":"<location>" }
Example:
{ "when":"today", "where":"Genova, Italy" }
Args:
request (str): The JSON dictionary input string.
Returns:
The weather data for the specified location and time.
'''
arguments = json.loads(json_request)
where = arguments["where"]
when = arguments["when"]
required_data = arguments["required_data"]
return weather_data_retriever(where=where, when=when, required_data=required_data)
#
# instantiate the langchain tool.
# The tool description instructs the LLM to pass data using a JSON.
# Note the "{{" and "}}": this double quotation is needed
# to avoid a runt-time error triggered by the agent instatiation.
#
name = "weather"
request_format = '{{"when":"<time>","where":"<location>","required_data":["variable_name"]}}'
description = f'''
Helps to retrieve weather forecast.
Input should be JSON in the following format: {request_format}
'''
# create an instance of the custom langchain tool
Weather = Tool(name=name, func=weather, description=description)
if __name__ == '__main__':
print(weather_data_retriever(where='Genova, Italy', when='today'))
# => in Genova, Italy, today is sunny! Temperature is 20 degrees Celsius.
print(weather('{ "when":"today", "where":"Genova, Italy" }'))
# => in Genova, Italy, today is sunny! Temperature is 20 degrees Celsius.
# print the Weather tool
print(Weather)Some testsNotes
What do you think? |
|
This is a great thread and very helpful. I've been doing something similar, but nowhere near as elegantly as this or rigorously tested. I've been working on my own AGI system and I have a voice interface as well as a Discord interface. I have the best luck when I use the This is a fun little tool that generates an image link from the DallE endpoint. |
|
I have a return time method which isn't returned directly. This one is less reliable, but because its not needing any data I generally don't run into issues trying to get the llm to interact with it |
|
I like the idea of being more explicit with the typing and wasn't aware the python was a typed language as you are using it. This is very helpful for me. The main reason I am here is because I have been running into this issue with the "Could not parse LLM output:" in the Search tool, using the Google wrapper For instance, after asking to generate an example of some technical architecture I got the error: |
|
Thanks Josh for your notes and examples!
Good remind of this flag, also documented here: https://langchain.readthedocs.io/en/latest/modules/agents/examples/custom_tools.html#using-tools-to-return-directly Some tests: Test 1 (return_direct=False => PASSED)I have my dumb weather forecasts tool, set with Test 2 (return_direct=False => FAILED)I have my dumb weather forecasts tool, set with Test 3 (return_direct=True => PASSED)I have my dumb weather forecasts tool, set with ConsiderationsAs far as I understand setting Pros Cons BTW the "Could not parse LLM output:" seems to me a true bug. I'd suggest to open a new different issue to explain in detail the problem. Thanks again |
UPDATE 4I did more experiments and I updated the code I share here. 1. Lesson learned; initial issue maybe solvedThe initial problem I raised is how to build a langchain tool wrapping any predefined python application function with a variable number of arguments. The @hwchase17 suggestion to use JSON to manage structured data was great! Recent LLMs (openai GPT 3.5) seem to well "understand" the JSON format, so
Some successful tests@hwchase17 2. Tool exception handlingNevertheless, I'm perplexed by results of inserting tools (using the described JSON I/O) in a React langchain agent. The problem is that I miss a way to exit form the agent when a tool return an exception, or an error. A specific issue arise by example when the tool fails because it lack of a REQUIRED argument, by example inhibiting a successful data retrieval. See this FAILING/controverse case: Here the problem is that the weather tool states it can't return the humidity (the data is not available): The expected behavior is an exit, with a message like:
Ok, this is maybe a problem related to the React agent. My general question is how to allow a tool to BREAK the agent elaborating (in case of any tool exception/error? BTW, setting
All in all I miss a mechanism to BREAK the agent reasoning, when a tool "fails" for some reasons, inhibiting further agent elaboration. In these case I want the tool exit (JSON) object be processed (interpreted) by the LLM before the final return to the agent caller. Proof of concept code used in tests#
# tools_agent.py
#
# zero-shot react agent that reply questions using available tools
# - Weater
# - Datetime
# - Location
#
# The agent gets the question as a command line argument (a quoted sentence).
# $ py tools_agent.py What about the weather today in Genova, Italy
#
import sys
from langchain.agents import initialize_agent
from langchain.llms import OpenAI
from langchain import LLMChain
from langchain.prompts import PromptTemplate
# import custom tools
from weather_tool import Weather
from datetime_tool import Datetime
from location_tool import Location
llm = OpenAI(temperature=0, verbose=True)
template = '''\
Please respond to the questions accurately and succinctly. \
If you are unable to obtain the necessary data after seeking help, \
indicate that you do not know.
'''
prompt = PromptTemplate(input_variables=[], template=template)
# debug
# print(prompt.format())
# Load the tool configs that are needed.
llm_weather_chain = LLMChain(
llm=llm,
prompt=prompt,
verbose=True
)
tools = [
Weather,
Datetime,
Location
]
# Construct the react agent type.
agent = initialize_agent(
tools,
llm,
agent="zero-shot-react-description",
verbose=True
)
# DEBUG
# https://github.com/hwchase17/langchain/issues/912#issuecomment-1426646112
# agent.agent.llm_chain.verbose=True
if __name__ == '__main__':
if len(sys.argv) > 1:
question = ' '.join(sys.argv[1:])
print('question: ' + question)
# run the agent
answer = agent.run(question)
print(answer)
else:
print('Agent answers questions using Weater and Datetime custom tools')
print('usage: py tools_agent.py <question sentence>')
print('example: py tools_agent.py what time is it?')#
# weather_tool.py
# A langchain tool that retrieves (fake) weather forecasts data
#
import json
from typing import List
from langchain.agents import Tool
def weather_data_retriever(
location: str = None,
period: str = None,
specific_variables: List[str] = []
) -> str:
'''
The function is an example of a custom python function
that takes a list of custom arguments and returns a text (or in general any data structure)
Given a location and a time period, this custom function
returns weather forecast as a data structure (in JSON format).
This is a mockup function, returning a fixed text tempalte.
The function could wrap an external API returning realtime weather forecast.
parameters:
location: location as text, e.g. 'Genova, Italy'
period: time period, e.g. 'today'
specific_variables: list of specific/required variable names, e.g ["temperature", "humidity"]
returns:
weather foreast description as a JSON. E.g.
{"forecast": "sunny all the day", "temperature": "20 degrees Celsius"}
'''
data = {}
# this function is a mockup, returns fake/hardcoded weather forecast data
data['forecast'] = 'sunny'
data['temperature'] = '20 degrees Celsius'
# ERROR/EXCEPTION: DISAMBIGUATION REQUIRED
# the tool can't elaborate because it doesn't has the mandatory variable 'location',
# so the returned content is an hardcoded error sentence (not a JSON), requiring a user disambiguation.
if not location or ('location' in location):
return 'The location is not specified. Where are you?'
# warning: the variable period is not defined so a default value is assigned
if not period or period == 'period':
data['period'] = 'now'
# if required variable names are not included in the data section,
# the attribute is added to the dictionary with value I don't know.
for variable_name in specific_variables:
if variable_name not in data.keys():
data[variable_name] = 'data not available'
return json.dumps(data)
def weather(json_request: str) -> str:
'''
wraps the weather_data_retriever function,
converting the input JSON in separated arguments.
Args:
request (str): The JSON dictionary input string.
Takes a JSON dictionary as input in the form:
{ "period":"<period>", "location":"<location>", "specific_variables":["variable_name", ... ]}
Example:
{ "period":"today", "location":"Genova, Italy", "specific_variables":["humidity"]}
Returns:
The weather data for the specified location and time.
'''
arguments = json.loads(json_request)
location = arguments.get('location', None)
period = arguments.get('period', None)
specific_variables = arguments.get('specific_variables', [])
return weather_data_retriever(location=location, period=period, specific_variables=specific_variables)
#
# instantiate the langchain tool.
# The tool description instructs the LLM to pass data using a JSON.
# Note the "{{" and "}}": this double quotation is needed
# to avoid a runt-time error triggered by the agent instatiation.
#
name = 'weather'
request_format = '{{"period":"period","location":"location","specific_variables":["variable_name"]}}'
description = f'''
Helps to retrieve weather forecast.
Input should be JSON in the following format: {request_format}
Supply "specific_variables" list just if you really need them.
If don't know the value to be assigned to a key, omit the key.
'''
# create an instance of the custom langchain tool
Weather = Tool(
name=name,
func=weather,
description=description,
return_direct=False
)
if __name__ == '__main__':
# print(weather_data_retriever(location='Genova, Italy', period='today'))
# => in Genova, Italy, today is sunny! Temperature is 20 degrees Celsius.
print(weather('{ "period":"today", "location":"Genova, Italy" }'))
# => {"forecast": "sunny", "temperature": "20 degrees Celsius"}
print(weather('{ "period":"today" }'))
# => The location is not specified. Where are you?
# print the Weather tool
print(Weather)# datetime_tool.py
# A langchain tool that returns current local date and time
#
import datetime
import json
from langchain.agents import Tool
def time():
# Get the current time
current_time = datetime.datetime.now()
# Format the time as a string in a local format
local_time = current_time.strftime("%I:%M %p")
return local_time
def date():
# Get the current time
current_time = datetime.datetime.now()
# Format the time as a string in a local format
local_time = current_time.strftime("%A, %B %d, %Y")
return local_time
def datetime_tool(request: str = None) -> str:
'''
returns currend date and time
Args:
request (str): optional.
If specified contains a list of specific variable needed, e.g.
{"specific_variables":["time"]}
Returns:
date and time as a JSON data structure, in the format:
'{{"fulldate":"<fulldate>","date":"<date>","time":"<time>"}}'
'''
data = {
'date': date(),
'time': time()
}
response_as_json = json.dumps(data)
return response_as_json
#
# instantiate the langchain tool.
# The tool description instructs the LLM to pass data using a JSON.
# Note the "{{" and "}}": this double quotation is needed to avoid a runt-time error triggered by the agent instatiation.
#
name = "date_time"
# response_format = '{{"fulldate":"<fulldate>","date":"<date>","time":"<time>"}}'
request_format = '{{"specific_variables":["variable_name"]}}'
response_format = '{{"date":"<date>","time":"<time>"}}'
description = f'''
helps to retrieve date and time.
Input should be an optional JSON in the following format: {request_format}
Output is a JSON in the following format: {response_format}'
'''
# create an instance of the custom langchain tool
Datetime = Tool(
name=name,
func=datetime_tool,
description=description,
return_direct=False
)
if __name__ == '__main__':
print(datetime_tool('{"specific_variables":["date"]}'))
# => {"date": "Tuesday, February 14, 2023", "time": "07:22 PM"}
print(Datetime)#
# location_tool.py
# A langchain tool that retrieves current location data
#
import json
from langchain.agents import Tool
def location(json_request: str = None) -> str:
'''
Returns:
The current location data in JSON format.
'''
data = {}
# this function is a mockup, returns fake/hardcoded location forecast data
data['city'] = 'Genova'
data['country'] = 'Italy'
# data['latitude'] = 44.411111
# data['longitude'] = 8.932778
# data['timezone'] = 'CET'
return json.dumps(data)
#
# instantiate the langchain tool.
# The tool description instructs the LLM to pass data using a JSON.
# Note the "{{" and "}}": this double quotation is needed
# to avoid a runt-time error triggered by the agent instantiation.
#
name = 'current_location'
description = 'Helps to retrieve current location data (where I\'m now). Returns a JSON with relevant variables'
# create an instance of the custom langchain tool
Location = Tool(
name=name,
func=location,
description=description,
return_direct=False
)
if __name__ == '__main__':
print(location())
# => {"city": "Genova", "country": "Italy", "latitude": 44.411111, "longitude": 8.932778, "timezone": "CET"}
# print the Location tool
print(Location) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is not necessarily an issue, but more of a 'how-to' question related to discussion topic #632.
This the general topic: You would like to create a language chain tool that functions as a custom function (wrapping any custom API). For example, let's say you have a Python function that retrieves real-time weather forecasts given a location (
where) and date/time (when) as input arguments, and returns a text with weather forecasts, as in the following mockup signature:weather_datain a langchain custom toolWeather, following the notebook here: https://langchain.readthedocs.io/en/latest/modules/agents/examples/custom_tools.html:weather_agent.py:An when I run the agent I have this output:
The custom weather tool is currently returning "when?" because the date/time argument is not being passed to the function. The agent tries to guess the date/time, which is not ideal but acceptable, and also invents the temperatures, leading to incorrect information:
Final Answer: The weather in Genova, Italy today is currently sunny with a high of 24°C and a low of 16°C.This occurs because the tool requires a single input string argument.
What would be your suggestion for mapping the information contained in the input string to the multiple arguments that the inner function/API (
weather_data(), in this case) expects?May you help to review the above tool behavior to process multiple arguments?
Thank you for your help,
Giorgio
The text was updated successfully, but these errors were encountered: