diff --git a/docs/source/en/custom_tools.mdx b/docs/source/en/custom_tools.mdx index f69a2cde90d75a..29a54b61cbcd92 100644 --- a/docs/source/en/custom_tools.mdx +++ b/docs/source/en/custom_tools.mdx @@ -21,7 +21,7 @@ If you are not aware of what tools and agents are in the context of transformers -Transformers Agent is an experimental API which is subject to change at any time. Results returned by the agents +Transformers Agent is an experimental API that is subject to change at any time. Results returned by the agents can vary as the APIs or underlying models are prone to change. @@ -36,13 +36,15 @@ In this guide we'll take a look at: ## Customizing the prompt As explained in [Transformers Agents](transformers_agents) agents can run in [`~Agent.run`] and [`~Agent.chat`] mode. -Both the run and chat mode underlie the same logic. The language model powering the agent is conditioned on a long prompt -and simply asked to complete the prompt by generating next tokens until the stop token is reached. -The only difference between the `run` and `chat` mode is that during the `chat` mode the prompt is extended with -previous user inputs and model generations, which seemingly gives the agent a memory and allows it to refer to -past interactions. +Both the `run` and `chat` modes underlie the same logic. The language model powering the agent is conditioned on a long +prompt and completes the prompt by generating the next tokens until the stop token is reached. +The only difference between the two modes is that during the `chat` mode the prompt is extended with +previous user inputs and model generations. This allows the agent to have access to past interactions, +seemingly giving the agent some kind of memory. -Let's take a closer look into how the prompt is structured to understand how it can be best customized. +### Structure of the prompt + +Let's take a closer look at how the prompt is structured to understand how it can be best customized. The prompt is structured broadly into four parts. - 1. Introduction: how the agent should behave, explanation of the concept of tools. @@ -50,16 +52,16 @@ The prompt is structured broadly into four parts. - 3. A set of examples of tasks and their solution - 4. Current example, and request for solution. -To better understand each part, let's look at a shortened version of how such a prompt can look like in practice. +To better understand each part, let's look at a shortened version of how the `run` prompt can look like: -``` +```` I will ask you to perform a task, your job is to come up with a series of simple commands in Python that will perform the task. [...] You can print intermediate results if it makes sense to do so. Tools: -- document_qa: This is a tool that answers a question about an document (pdf). It takes an input named `document` which should be the document containing the information, as well as a `question` that is the question about the document. It returns a text that contains the answer to the question. -- image_captioner: This is a tool that generates a description of an image. It takes an input named `image` which should be the image to caption, and returns a text that contains the description in English. +- document_qa: This is a tool that answers a question about a document (pdf). It takes an input named `document` which should be the document containing the information, as well as a `question` that is the question about the document. It returns a text that contains the answer to the question. +- image_captioner: This is a tool that generates a description of an image. It takes an input named `image` which should be the image to the caption and returns a text that contains the description in English. [...] Task: "Answer the question in the variable `question` about the image stored in the variable `image`. The question is in French." @@ -90,21 +92,286 @@ image = image_generator("A banner showing " + answer) Task: "Draw me a picture of rivers and lakes" I will use the following +```` + +The introduction (the text before *"Tools:"*) explains precisely how the model shall behave and what it should do. +This part most likely does not need to be customized as the agent shall always behave the same way. + +The second part (the bullet points below *"Tools"*) is dynamically added upon calling `run` or `chat`. There are +exactly as many bullet points as there are tools in `agent.toolbox` and each bullet point consists of the name +and description of the tool: + +``` +- : +``` + +Let's verify this quickly by loading the document_qa tool and printing out the name and description. + +```py +from transformers import load_tool + +document_qa = load_tool("document-question-answering") +print(f"- {document_qa.name}: {document_qa.description}") +``` + +which gives: +``` +- document_qa: This is a tool that answers a question about a document (pdf). It takes an input named `document` which should be the document containing the information, as well as a `question` that is the question about the document. It returns a text that contains the answer to the question. +``` + +We can see that the tool name is short and precise. The description includes two parts, the first explaining +what the tool does and the second states what input arguments and return values are expected. + +A good tool name and tool description are very important for the agent to correctly use it. Note that the only +information the agent has about the tool is its name and description, so one should make sure that both +are precisely written and match the style of the existing tools in the toolbox. + + + +Check the naming and description of the curated Transformers tools to better understand what name and +description a tool is expected to have. You can see all tools with the [`Agent.toolbox`] property. + + + +The third part includes a set of curated examples that show the agent exactly what code it should produce +for what kind of user request. The large language models empowering the agent are extremely good at +recognizing patterns in a prompt and repeating the pattern with new data. Therefore, it is very important +that the examples are written in a way that maximizes the likelihood of the agent to generating correct, + executable code in practice. + +Let's have a look at one example: + +```` +Task: "Identify the oldest person in the `document` and create an image showcasing the result as a banner." + +I will use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer. + +Answer: +```py +answer = document_qa(document, question="What is the oldest person?") +print(f"The answer is {answer}.") +image = image_generator("A banner showing " + answer) +``` + +```` + +The pattern the model is prompted to repeat has three parts: The task statement, the agent's explanation of +what it intends to do, and finally the generated code. Every example that is part of the prompt has this exact +pattern, thus making sure that the agent will reproduce exactly the same pattern when generating new tokens. + +The prompt examples are curated by the Transformers team and rigorously evaluated on a set of +[problem statements](https://github.com/huggingface/transformers/blob/main/src/transformers/tools/evaluate_agent.py) +to ensure that the agent's prompt is as good as possible to solve real use cases of the agent. + +The final part of the prompt corresponds to: +``` +Task: "Draw me a picture of rivers and lakes" + +I will use the following +``` + +is a final and unfinished example that the agent is tasked to complete. The unfinished example +is dynamically created based on the actual user input. For the above example, the user ran: + +```py +agent.run("Draw me a picture of rivers and lakes") +``` + +The user input - *a.k.a* the task: *"Draw me a picture of rivers and lakes"* is cast into the +prompt template: "Task: \n\n I will use the following". This sentence makes up the final lines of the +prompt the agent is conditioned on, therefore strongly influencing the agent to finish the example +exactly in the same way it was previously done in the examples. + +Without going into too much detail, the chat template has the same prompt structure with the +examples having a slightly different style, *e.g.*: + +```` +[...] + +===== + +Human: Answer the question in the variable `question` about the image stored in the variable `image`. + +Assistant: I will use the tool `image_qa` to answer the question on the input image. + +```py +answer = image_qa(text=question, image=image) +print(f"The answer is {answer}") +``` + +Human: I tried this code, it worked but didn't give me a good result. The question is in French + +Assistant: In this case, the question needs to be translated first. I will use the tool `translator` to do this. + +```py +translated_question = translator(question=question, src_lang="French", tgt_lang="English") +print(f"The translated question is {translated_question}.") +answer = image_qa(text=translated_question, image=image) +print(f"The answer is {answer}") +``` + +===== + +[...] +```` + +Contrary, to the examples of the `run` prompt, each `chat` prompt example has one or more exchanges between the +*Human* and the *Assistant*. Every exchange is structured similarly to the example of the `run` prompt. +The user's input is appended to behind *Human:* and the agent is prompted to first generate what needs to be done +before generating code. An exchange can be based on previous exchanges, therefore allowing the user to refer +to past exchanges as is done *e.g.* above by the user's input of "I tried **this** code" refers to the +previously generated code of the agent. + +Upon running `.chat`, the user's input or *task* is cast into an unfinished example of the form: +``` +Human: \n\nAssistent: +``` +which the agent completes. Contrary to the `run` command, the `chat` command then appends the completed example +to the prompt, thus giving the agent more context for the next `chat` turn. + +Great now that we know how the prompt is structured, let's see how we can customize it! + +### Writing good user inputs + +While large language models are getting better and better at understanding users' intentions, it helps +enormously to be as precise as possible to help the agent pick the correct task. What does it mean to be +as precise as possible? + +The agent sees a list of tool names and their description in its prompt. The more tools are added the +more difficult it becomes for the agent to choose the correct tool and it's even more difficult to choose +the correct sequences of tools to run. Let's look at a common failure case, here we will only return +the code to analyze it. + +```py +from transformers import HfAgent + +agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder") + +agent.run("Show me a tree", return_code=True) +``` + +gives: + +``` +==Explanation from the agent== +I will use the following tool: `image_segmenter` to create a segmentation mask for the image. + + +==Code generated by the agent== +mask = image_segmenter(image, prompt="tree") ``` -The first part explains precisely how the model shall behave and what it should do. This part -most likely does not need to be customized. +which is probably not what we wanted. Instead, it is more likely that we want an image of a tree to be generated. +To steer the agent more towards using a specific tool it can therefore be very helpful to use important keywords that +are present in the tool's name and description. Let's have a look. +```py +agent.toolbox["image_generator"].description +``` +``` +'This is a tool that creates an image according to a prompt, which is a text description. It takes an input named `prompt` which contains the image description and outputs an image. +``` -TODO(PVP) - explain better how the .description and .name influence the prompt +The name and description make use of the keywords "image", "prompt", "create" and "generate". Using these words will most likely work better here. Let's refine our prompt a bit. + +```py +agent.run("Create an image of a tree", return_code=True) +``` + +gives: +``` +==Explanation from the agent== +I will use the following tool `image_generator` to generate an image of a tree. + + +==Code generated by the agent== +image = image_generator(prompt="tree") +``` + +Much better! That looks more like what we want. In short, when you notice that the agent struggles to +correctly map your task to the correct tools, try looking up the most pertinent keywords of the tool's name +and description and try refining your task request with it. ### Customizing the tool descriptions -The performance of the agent is directly linked to the prompt itself. We structure the prompt so that it works well -with what we intend for the agent to do; but for maximum customization we also offer the ability to specify a different prompt when instantiating the agent. +As we've seen before the agent has access to each of the tools' names and descriptions. The base tools +should have very precise names and descriptions, however, you might find that it could help to change the +the description or name of a tool for your specific use case. This might become especially important +when you've added multiple tools that are very similar or if you want to use your agent only for a certain +domain, *e.g.* image generation and transformations. + +A common problem is that the agent confuses image generation with image transformation/modification when +used a lot for image generation tasks, *e.g.* +```py +agent.run("Make an image of a house and a car", return_code=True) +``` +returns +``` +==Explanation from the agent== +I will use the following tools `image_generator` to generate an image of a house and `image_transformer` to transform the image of a car into the image of a house. + +==Code generated by the agent== +house_image = image_generator(prompt="A house") +car_image = image_generator(prompt="A car") +house_car_image = image_transformer(image=car_image, prompt="A house") +``` + +which is probably not exactly what we want here. It seems like the agent has a difficult time +to understand the difference between `image_generator` and `image_transformer` and often uses the two together. + +We can help the agent here by changing the tool name and description of `image_transformer`. Let's instead call it `modifier` +to disassociate it a bit from "image" and "prompt": +``` +agent.toolbox["modifier"] = agent.toolbox.pop("image_transformer") +agent.toolbox["modifier"].description = agent.toolbox["modifier"].description.replace("transforms an image according to a prompt", "modifies an image") +``` + +Now "modify" is a strong cue to use the new image processor which should help with the above prompt. Let's run it again. + +```py +agent.run("Make an image of a house and a car", return_code=True) +``` + +Now we're getting: +``` +==Explanation from the agent== +I will use the following tools: `image_generator` to generate an image of a house, then `image_generator` to generate an image of a car. + -### Customizing the single-execution prompt +==Code generated by the agent== +house_image = image_generator(prompt="A house") +car_image = image_generator(prompt="A car") +``` -In order to specify a custom single-execution prompt, one would so the following: +which is definitely closer to what we had in mind! However, we want to have both the house and car in the same image. Steering the task more toward single image generation should help: + +```py +agent.run("Create image: 'A house and car'", return_code=True) +``` + +``` +==Explanation from the agent== +I will use the following tool: `image_generator` to generate an image. + + +==Code generated by the agent== +image = image_generator(prompt="A house and car") +``` + + + +Agents are still brittle for many use cases, especially when it comes to +slightly more complex use cases like generating an image of multiple objects. +Both the agent itself and the underlying prompt will be further improved in the coming +months making sure that agents become more robust to a variety of user inputs. + + + +### Customizing the whole prompt + +To give the user maximum flexibility, the whole prompt template as explained in [above](#structure-of-the-prompt) +can be overwritten by the user. In this case make sure that your custom prompt includes an introduction section, +a tool section, an example section, and an unfinished example section. If you want to overwrite the `run` prompt template, +you can do as follows: ```py template = """ [...] """ @@ -112,31 +379,33 @@ template = """ [...] """ agent = HfAgent(your_endpoint, run_prompt_template=template) ``` - + -Please make sure to have the `<>` string defined somewhere in the `template` so that the agent can be aware -of the tools it has available to it. +Please make sure to have the `<>` string and the `<>` defined somewhere in the `template` so that the agent can be aware +of the tools, it has available to it as well as correctly insert the user's prompt. -#### Chat-execution prompt +Similarly, one can overwrite the `chat` prompt template. Note that the `chat` mode always uses the following format for the exchanges: +``` +Human: <> + +Assistant: +``` -In order to specify a custom single-execution prompt, one would so the following: +Therefore it is important that the examples of the custom `chat` prompt template also make use of this format. +You can overwrite the `chat` template at instantiation as follows. ``` template = """ [...] """ -agent = HfAgent( - url_endpoint=your_endpoint, - token=your_hf_token, - chat_prompt_template=template -) +agent = HfAgent(url_endpoint=your_endpoint, chat_prompt_template=template) ``` - + -Please make sure to have the `<>` string defined somewhere in the `template` so that the agent can be -aware of the tools it has available to it. +Please make sure to have the `<>` string defined somewhere in the `template` so that the agent can be aware +of the tools, it has available to it. @@ -176,7 +445,7 @@ It takes two inputs: `image`, which should be the image to transform, and `promp Name: 'image_transformer' ``` -The name and description is accurate and fits the style of the [curated set of tools](./transformers_agents#a-curated-set-of-tools). +The name and description are accurate and fit the style of the [curated set of tools](./transformers_agents#a-curated-set-of-tools). Next, let's instantiate an agent with `controlnet_transformer` and `upscaler`: ```py @@ -191,7 +460,7 @@ image_transformer has been replaced by as provided in `additional_tools` ``` -The set of curated tools already has a `image_transformer` tool which is hereby replaced with our custom tool. +The set of curated tools already has an `image_transformer` tool which is hereby replaced with our custom tool. @@ -201,7 +470,7 @@ as the overwritten tool in this case. -The upscaler tool was given the name `image_upscaler` which is not yet present in the default toolbox and is therefore is simply added to the list of tools. +The upscaler tool was given the name `image_upscaler` which is not yet present in the default toolbox and is therefore simply added to the list of tools. You can always have a look at the toolbox that is currently available to the agent via the `agent.toolbox` attribute: ```py @@ -248,7 +517,7 @@ image = agent.run("Transform the image: 'A frozen lake and snowy forest'", image ``` ==Explanation from the agent== -I will use the following tool: `image_transformer` to transform the image. +I will use the following tool: `image_transformer` to transform the image. ==Code generated by the agent== @@ -257,7 +526,7 @@ image = image_transformer(image, prompt="A frozen lake and snowy forest") -The new image processing tool is based on ControlNet which is can make very strong modifications to the image. +The new image processing tool is based on ControlNet which can make very strong modifications to the image. By default the image processing tool returns an image of size 512x512 pixels. Let's see if we can upscale it. ```py @@ -266,7 +535,7 @@ image = agent.run("Upscale the image", image) ``` ==Explanation from the agent== -I will use the following tool: `image_upscaler` to upscale the image. +I will use the following tool: `image_upscaler` to upscale the image. ==Code generated by the agent== @@ -278,11 +547,11 @@ upscaled_image = image_upscaler(image) The agent automatically mapped our prompt "Upscale the image" to the just added upscaler tool purely based on the description and name of the upscaler tool and was able to correctly run it. -Next, let's have a look into how you can create a new custom tool. +Next, let's have a look at how you can create a new custom tool. ### Adding new tools -In this section we show how to create a new tool that can be added to the agent. +In this section, we show how to create a new tool that can be added to the agent. #### Creating a new tool @@ -406,7 +675,7 @@ and generates the following audio. Depending on the LLM, some are quite brittle and require very exact prompts in order to work well. Having a well-defined -description of the tool is paramount to having it be leveraged by the agent. +name and description of the tool is paramount to having it be leveraged by the agent. diff --git a/docs/source/en/transformers_agents.mdx b/docs/source/en/transformers_agents.mdx index bddc908b020196..5343e93cf3386a 100644 --- a/docs/source/en/transformers_agents.mdx +++ b/docs/source/en/transformers_agents.mdx @@ -21,7 +21,7 @@ can vary as the APIs or underlying models are prone to change. Transformers version v4.29.0, building on the concept of *tools* and *agents*. -In short, it provides a natural language API on top of transformers: we define a set of curated tools, and design an +In short, it provides a natural language API on top of transformers: we define a set of curated tools and design an agent to interpret natural language and to use these tools. It is extensible by design; we curated some relevant tools, but we'll show you how the system can be extended easily to use any tool developed by the community. @@ -63,7 +63,7 @@ Before being able to use `agent.run`, you will need to instantiate an agent, whi We recommend using the [bigcode/starcoder](https://huggingface.co/bigcode/starcoder) checkpoint as it works very well for the task at hand and is open-source, but please find other examples below. -Start by logging-in to have access to the Inference API: +Start by logging in to have access to the Inference API: ```py from huggingface_hub import login @@ -79,8 +79,8 @@ from transformers import HfAgent agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder") ``` -This is using the inference API that Hugging Face provides for free at the moment, if you have your own inference -endpoint for this model (or another one) you can replace the url above by your url endpoint. +This is using the inference API that Hugging Face provides for free at the moment if you have your inference +endpoint for this model (or another one) you can replace the URL above with your URL endpoint. @@ -102,7 +102,7 @@ agent.run("Draw me a picture of rivers and lakes") -It automatically select the tool (or tools) appropriate for the task you want to perform and run them appropriately. It +It automatically selects the tool (or tools) appropriate for the task you want to perform and runs them appropriately. It can perform one or several tasks in the same instruction (though the more complex your instruction, the more likely the agent is to fail). @@ -121,7 +121,7 @@ Note that your `agent` is just a large-language model, so small variations in yo different results. It's important to explain as clearly as possible the task you want to perform. If you'd like to keep a state across executions or to pass non-text objects to the agent, you can do so by specifying -variables that you would like the agent to use. For example you could generate the first image of rivers and lakes, +variables that you would like the agent to use. For example, you could generate the first image of rivers and lakes, and ask the model to update that picture to add an island by doing the following: ```python @@ -133,17 +133,17 @@ updated_picture = agent.chat("Take that `picture` and add an island to it", pict This can be helpful when the model is unable to understand your request and mixes tools. An example would be: -```python +```py agent.run("Draw me the picture of a capybara swimming in the sea") ``` -Here, the model could interpret it two ways: +Here, the model could interpret in two ways: - Have the `text-to-image` generate a capybara swimming in the sea - Or, have the `text-to-image` generate capybara, then use the `image-transformation` tool to have it swim in the sea In case you would like to force the first scenario, you could do so by passing it the prompt as an argument: -```python +```py agent.run("Draw me a picture of the `prompt`", prompt="a capybara swimming in the sea") ``` @@ -177,15 +177,15 @@ This method can also take arguments if you would like to pass non-text types or ### ⚠️ Remote execution For demonstration purposes and so that this can be used with all setups, we have created remote executors for several -of the default tools the agent has access to. These are created using -[inference endpoints](https://huggingface.co/inference-endpoints). To see how to setup remote executors tools yourself, -we recommend reading the custom tool guide [TODO LINK]. +of the default tools the agent has access. These are created using +[inference endpoints](https://huggingface.co/inference-endpoints). To see how to set up remote executors tools yourself, +we recommend reading the [custom tool guide](./custom_tools). In order to run with remote tools, specifying `remote=True` to either [`~Agent.run`] or [`~Agent.chat`] is sufficient. For example, the following command could be run on any device efficiently, without needing significant RAM or GPU: -```python +```py agent.run("Draw me a picture of rivers and lakes", remote=True) ``` @@ -202,18 +202,18 @@ agent.chat("Draw me a picture of rivers and lakes", remote=True) The "agent" here is a large language model, and we're prompting it so that it has access to a specific set of tools. LLMs are pretty good at generating small samples of code, so this API takes advantage of that by prompting the -LLM to give a small sample of code performing a task with a set of tools. This prompt is then completed by the +LLM gives a small sample of code performing a task with a set of tools. This prompt is then completed by the task you give your agent and the description of the tools you give it. This way it gets access to the doc of the -tools you are using, especially their expected inputs and outputs and can generate the relevant code. +tools you are using, especially their expected inputs and outputs, and can generate the relevant code. #### Tools -Tools are very simple: they're a single function, with a name, and a description. We then use these tools description -to prompt the agent. Through the prompt, we show the agent how it would leverage tools in order to perform what was -requests in the query. +Tools are very simple: they're a single function, with a name, and a description. We then use these tools' descriptions +to prompt the agent. Through the prompt, we show the agent how it would leverage tools to perform what was +requested in the query. This is using brand-new tools and not pipelines, because the agent writes better code with very atomic tools. -Pipelines are more refactored and often combine several tasks in one. Tools are really meant to be focused on +Pipelines are more refactored and often combine several tasks in one. Tools are meant to be focused on one very simple task only. #### Code-execution?! @@ -271,13 +271,12 @@ directly with the agent. We've added a few - **Text to image**: generate an image according to a prompt, leveraging stable diffusion - **Image transformation**: modify an image given an initial image and a prompt, leveraging instruct pix2pix stable diffusion -The text-to-image tool we have been using since the beginning is actually a remote tool that lives in +The text-to-image tool we have been using since the beginning is a remote tool that lives in [*huggingface-tools/text-to-image*](https://huggingface.co/spaces/huggingface-tools/text-to-image)! We will -continue releasing such tools on this and other organization, to further supercharge this implementation. +continue releasing such tools on this and other organizations, to further supercharge this implementation. The agents have by default access to tools that reside on `huggingface-tools`. -We explain how to you can write and share your own tools as well as leverage any custom tool that resides on the Hub in [following guide](custom_tools). -[following guide](custom_tools). +We explain how to you can write and share your tools as well as leverage any custom tool that resides on the Hub in [following guide](custom_tools). ### Leveraging different agents @@ -307,7 +306,7 @@ agent = OpenAiAgent(model="text-davinci-003", api_key="") ### Code generation -So far we have shown how to use the agents to perform actions for you. However, the agent is really only generating code +So far we have shown how to use the agents to perform actions for you. However, the agent is only generating code that we then execute using a very restricted Python interpreter. In case you would like to use the code generated in a different setting, the agent can be prompted to return the code, along with tool definition and accurate imports. diff --git a/src/transformers/tools/agents.py b/src/transformers/tools/agents.py index e1ae12804b3a20..bdb45f50719a70 100644 --- a/src/transformers/tools/agents.py +++ b/src/transformers/tools/agents.py @@ -19,6 +19,7 @@ import os import time from dataclasses import dataclass +from typing import Dict import requests from huggingface_hub import HfFolder, hf_hub_download, list_spaces @@ -199,7 +200,7 @@ def __init__(self, chat_prompt_template=None, run_prompt_template=None, addition self.chat_prompt_template = CHAT_MESSAGE_PROMPT if chat_prompt_template is None else chat_prompt_template self.run_prompt_template = RUN_PROMPT_TEMPLATE if run_prompt_template is None else run_prompt_template - self.toolbox = HUGGINGFACE_DEFAULT_TOOLS.copy() + self._toolbox = HUGGINGFACE_DEFAULT_TOOLS.copy() if additional_tools is not None: if isinstance(additional_tools, (list, tuple)): additional_tools = {t.name: t for t in additional_tools} @@ -207,7 +208,7 @@ def __init__(self, chat_prompt_template=None, run_prompt_template=None, addition additional_tools = {additional_tools.name: additional_tools} replacements = {name: tool for name, tool in additional_tools.items() if name in HUGGINGFACE_DEFAULT_TOOLS} - self.toolbox.update(additional_tools) + self._toolbox.update(additional_tools) if len(replacements) > 1: names = "\n".join([f"- {n}: {t}" for n, t in replacements.items()]) logger.warn( @@ -219,6 +220,11 @@ def __init__(self, chat_prompt_template=None, run_prompt_template=None, addition self.prepare_for_new_chat() + @property + def toolbox(self) -> Dict[str, Tool]: + """Get all tool currently available to the agent""" + return self._toolbox + def format_prompt(self, task, chat_mode=False): description = "\n".join([f"- {name}: {tool.description}" for name, tool in self.toolbox.items()]) if chat_mode: