[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models

[K-Mistele]

DRAFT: OpenAI Tool Use Checklist

This (Draft) PR will add support for OpenAI-style tool calling in a way that is minimally opinionated about tool use formats & prompt formatting.

The following features are expected to be supported:

• Custom tool use system prompt template (if desired) -- prevent opinionation about if/how the model uses a system prompt to enable tool use
• Custom tool call return value prompt template (if desired) -- prevent opinionation about the format that tool return values should be passed to the model
• Support for tool_choice="auto" - named tool choice is already supported via guided decoding
• Streaming tool call responses from the chat completions API
• Verified Support & examples for at least the following models:
  • Mistral 7B Instruct v0.3
  • Nous Research's Hermes 2 Pro Models function-calling models including Hermes 2 Pro Llama 3 8B

I'd welcome anyone who wants to contribute on this, and would be happy to add you to the Constellate AI vllm fork that this PR is based off of - please just leave a comment!

Checklist/roadmap:

• validation of tools and tool_choice
• sending tools to the model with tool_choice="auto"
  • CLI Argument: enable auto tool choice
  • CLI argument: tool use system prompt template path - specify the template for the prompt that tells the model how to use the provided tools
  • render template with specified tools
  • prepend the rendered template to the existing system prompt OR use it as the only one if the client didn’t specify one.
  • verify that the model will return a tool call as the chat completion response
• returning tool calls to the client
  • Detect if the model is returning a tool call via the first token
    • CLI argument - specify the token / token ID for tool use responses
  • implement custom extractor class that can be used and implemented for different models to extract their tool call formats to OpenAi's format
  • non-streaming chat completion response: return the JSON as appropriate response.tool_calls
  • streaming chat completion: if the tool use token is called, start streaming the tokens to the client.
• providing tool call results to the model
  • enable specifying a custom chat template
  • support using huggingface transformers to select the tool_use chat template
• verify support:
  • Nous Research’s Hermes 2 Pro LLama 3 8B
  • Mistral 7B instruct v0.3
• Add documentation

FIX #3237 #4656 (link existing issues this PR will resolve)

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

---

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Model] for adding a new model or improving an existing model. Model name should appear in the title.
• [Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
• [Kernel] for changes affecting CUDA kernels or other compute kernels.
• [Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
• [Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• We adhere to Google Python style guide and Google C++ style guide.
• Pass all linter checks. Please use format.sh to format your code.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
• Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

• After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
• After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
• After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
• Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

[K-Mistele]

Progress! I as of current commits, I can now get the hermes 2 pro model to generate a tool call using the --enable-auto-tool-choice and --tool-use-prompt-template flags:

Server:

python -m vllm.entrypoints.openai.api_server --model NousResearch/Hermes-2-Pro-Llama-3-8B --tool-use-prompt-template examples/tool_template_hermes_2_pro.jinja --enable-api-tools --enable-auto-tool-choice

Client:

python examples/openai_chat_completion_client_with_tools.py

Result

Chat completion results:
ChatCompletion(id='cmpl-1354f3f373574d7aa0e1bf0b78915188', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='<tool_call>{"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"}</tool_call>', role='assistant', function_call=None, tool_calls=[]), stop_reason=None)], created=1718763539, model='NousResearch/Hermes-2-Pro-Llama-3-8B', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=33, prompt_tokens=367, total_tokens=400))

Now, working on getting it to work for non-streaming responses - then, streaming!

[K-Mistele]

A question I asked in the discord, with some open questions about how to handle configuration:

Setting up function calling models for an open model requires a lot of configurations if you want to be unopinionated about the model. Here is a brief list of all the parameters that would be needed:

• enable "auto" tool choice - allow models to choose the function to call (supported for some models) or ONLY tool_choice="" for named tool choice via guided decoding
• tool use prompt template - how do you render the list of tools provided in the request into the prompt / system prompt for the conversation?
• tool use prompt role - what is the role of that message? defaults to system.
• tool use response start token - different models (e.g. Hermes 2 Pro models vs. Mistral 7B Instruct v0.3) use different tokens in their tokenizer to indicate the start of a tool call response vs. a chat response. it's important that this is configured correctly so that we can know whether to send the model's response as a chat response or tool response, and how to stream the response if stream=True. Because there does not seem to be a specific convention for defining this in a uniform way across model tokenizers, it will be necessary for the user to inform the API server which token indicates a tool response is starting, in order to have fully OpenAI API-compatible tool responses.
• tool result response/return value message template the template for returning tool results to the model so it can generated based on results of the tool. - e.g. <tool_response>{"name": "function_name_here", "content": TOOL_RETURN_VALUE_HERE}</tool_response> for Hermes 2 pro models
• tool result response/return value message role - the ROLE to use for that message, e.g. tool for Hermes 2 pro models

The question is, it is better to have all of these as separate CLI flags, or would a JSON configuration file be preferable so that people can create (and track in version control!) configs that work for popular models?)

[K-Mistele]

Please see ongoing conversation with the Hugging Face team, Nous Research & transformers maintainer here - this will make it MUCH easier to implement OpenAI API-compatible tool calling into vLLM regardless of model prompt/tokenizer configs.

HF PR for Hermes 2 Pro: https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/discussions/13#66724ea9bd5875ad665f1416

HF PR for Mistral 7B instruct v0.3: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/35

Once these are merged, there will be a STANDARD way in transformers to handle templating in tool responses just like for templating chat conversations into prompts with a chat template, and hopefully to pull out tool calls from generated text.

[interstellarninja]

'<tool_call>{"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"}</tool_call>'

hey great initiative and nice to see Hermes Pro model's tool calls working.

there's a slight issue with this tool call -- our format requires new lines after <tool_call> XML tags:

<tool_call>
{"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"}
</tool_call>

Also tool choice should also work since it's basically passing the chosen tool only as part of

[K-Mistele]

Thanks! Tool choice is already working via guided decoding, but I will update the PR to fix the template

[K-Mistele]

Ok the most recent commit seems to fix it:
python examples/openai_chat_completion_client_with_tools.py

ChatCompletion(id='cmpl-48f019602ab64f4ab49c3563318c6d1f', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='<tool_call>\n{"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"}\n</tool_call>', role='assistant', function_call=None, tool_calls=[]), stop_reason=None)], created=1718896666, model='NousResearch/Hermes-2-Pro-Llama-3-8B', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=34, prompt_tokens=368, total_tokens=402))

[frankabc12341]

@K-Mistele
Hi. My vllm version is 0.5.0 post1. But, when I run the command

python -m vllm.entrypoints.openai.api_server --model /home/asus/autodl-tmp/qwen/Qwen2-7B-Instruct --tool-use-prompt-template /home/asus/autodl-tmp/examples/chatml.jinja --enable-api-tools --enable-auto-tool-choice
it shows api_server.py: error: unrecognized arguments: --tool-use-prompt-template --enable-api-tools --enable-auto-tool-choice

In addition, I can't find openai_chat_completion_client_with_tools.py in the example fille.

Can you give me some advice?

[K-Mistele]

@K-Mistele Hi. My vllm version is 0.5.0 post1. But, when I run the command

python -m vllm.entrypoints.openai.api_server --model /home/asus/autodl-tmp/qwen/Qwen2-7B-Instruct --tool-use-prompt-template /home/asus/autodl-tmp/examples/chatml.jinja --enable-api-tools --enable-auto-tool-choice it shows api_server.py: error: unrecognized arguments: --tool-use-prompt-template --enable-api-tools --enable-auto-tool-choice

In addition, I can't find openai_chat_completion_client_with_tools.py in the example fille.

Can you give me some advice?

Please see my reply here. This is a draft pull request, which means it has not been merged into vLLM's codebase, and it is not ready to be merged yet. It is still a work-in-progress. As such, none of its additions or features are available in vLLM yet. If you are interested in testing or contributing to this pull request, please see the origin fork at the tool-use branch. However, please be aware that the capabilities discussed in this PR are incomplete and have not been robustly tested.

[aw632]

if the tool use token is called, pull out the tool call JSON from the rest of the response (should be array)

Given that guided decoding is not enabled for "auto" tool use, what is the error handling planned in case the LLM does not output valid JSON?

[K-Mistele]

if the tool use token is called, pull out the tool call JSON from the rest of the response (should be array)

Given that guided decoding is not enabled for "auto" tool use, what is the error handling planned in case the LLM does not output valid JSON?

Great question! Unfortunately since each model that supports tool calling uses its' own format for function calls (as opposed to tool choice with guided decoding, where we're forcing the LLM to call a specific tool in a specific format at decode-time) the response format is up to the model and its' trainer.

At this point, we are still exploring ways to handle extraction of tool calls from disparate formats to OpenAI-compatible calls in a way that isn't opinionated (we want to support multiple formats including Mistral, Hermes 2 Pro, Firefunction, etc). Until we have a good answer on this, we probably won't have a good answer for how to handle errors. There are a couple possible options once we have attempted to extract the model's tool call format into OpenAI's:

• return the call to the client without validation, and allow the client to validate the tool call with libraries like Instructor, pydantic or Zod
• return an HTTP error if the model generates an invalid tool call
• automatically attempt to re-run the generation - perhaps only if the user specifies a specific CLI flag, since this could lead to performance issues or infinite retry loops depending on the model and other configurations like temperature, top_p and top_k

In the interim, we may try to solve this problem by "ignoring" it - detect if the model is generating a tool call, but avoid the destructuring/extraction issue by returning the tool call in the model's format to the client. Not exactly sure what this would look like, but it's a possibility.

[K-Mistele]

Copy/pasting from discord:

I'd really like to make progress on tool use and right now with hugging face adding support into transformers for passing tools into the chat template, that solves one of the main issues

I think the blocker now is figuring out how to handle decodiong "auto" choice tool calls from each model's specific format (mistral vs. Hermes pro vs. firefunction) and returning that to the client ESPECIALLY when streaming is requested.

Until there's a "canonical" way to do decode model-specific tool-calls into the OpenAI format e.g. through transformers or "reverse templates" or something, it might be best to approach this like chat templates which is to try & support it as good as we can where possible

Here's what I mean by this:

• mistral's format is very close to openAI's format and is probably close in terms of implementation. If a tool is being called, the first token from the response indicates this, and then a JSOn array is generated. I can use a partial JSON parser to stream tool arguments etc as they are generated to the client, basically without any translation
• Hermes 2 Pro's format is different and uses multiple XML tags <tool_call></tool_call> for each call with the JSON inside. I should also be able to handle some streaming here with XML extraction and regex.
• Firefunction v2 uses a totally different format as well.

So each one needs a very different implementation. I propose creating a ToolCallParser abstract class that can be implemented for different models like mistral and hermes. If a user is using a tool-calling model, they can use a CLI flag to toggle which parser they want, if they want one at all. If not, a tool call would be treated like a regular chat completion, and they can handle it client-side.

This way, we can ship support for tool calls incrementally in the absence of a commonly-accepted "best practice" on how to do this. People can also add support for other models that are important to them in a minimally-invasive way. Then, once a better way to implement this more broadly is available, we can deprecate this approach

I'd really appreciate feedback on this before moving forward and would especially love to hear if @mgoin and @simon-mo would consider this an appropriate approach that would be likely to be approved.

[simon-mo]

@br3no would love to get your feedback and review for this PR!

[mgoin]

FWIW I verified that spec decode tests pass locally 🔥

[wybartel]

Thank you all for your hard work! Especially you @K-Mistele! Great getting this into main!!!

[TimPietrusky]

I was following along the whole time, thank you so much @K-Mistele ❤️

[meetzuber]

How can I use it with llama 3.1 8b model?

[pbasov]

@meetzuber you have to write a prompt template for l3.1 if I understand the implementation correctly.

[gislerro]

@meetzuber you have to write a prompt template for l3.1 if I understand the implementation correctly.

From my understanding you have to write a ToolParser aswell.

I'm experimenting with a more general approach to tool parsing that generalizes over different models

Since OpenAI API compatible tool calls only require two string arguments name and arguments a regex for custom tool calling must only define two capture groups with those names, such a regex can be validated with a "meta" regex like:

(?=.*?\(\?P<name>.*?\))(?=.*?\(\?P<arguments>.*?\))

The tool calling regex must then be provided with a cli arg --tool-call-regex.

Then as an example for the Llama 3.1 model with JSON based tool calling your --tool-call-regex looks like:

{"name": "(?P<name>.*?)", "parameters": (?P<arguments>.*?)}

which would then extract the tool call: https://regex101.com/r/RhZ4zx/1

For a different custom tool calling format just provide another --tool-call-regex:

<function=(?P<name>.*?)>(?P<arguments>.*?)<\/function>

which also extracts tool call(s): https://regex101.com/r/9pM6IL/2

[K-Mistele]

How can I use it with llama 3.1 8b model?

I will be adding support in a separate PR

[K-Mistele]

@meetzuber you have to write a prompt template for l3.1 if I understand the implementation correctly.

From my understanding you have to write a ToolParser aswell.

I'm experimenting with a more general approach to tool parsing that generalizes over different models

Since OpenAI API compatible tool calls only require two string arguments name and arguments a regex for custom tool calling must only define two capture groups with those names, such a regex can be validated with a "meta" regex like:

(?=.*?\(\?P<name>.*?\))(?=.*?\(\?P<arguments>.*?\))

The tool calling regex must then be provided with a cli arg --tool-call-regex.

Then as an example for the Llama 3.1 model with JSON based tool calling your --tool-call-regex looks like:

{"name": "(?P<name>.*?)", "parameters": (?P<arguments>.*?)}

which would then extract the tool call: https://regex101.com/r/RhZ4zx/1

For a different custom tool calling format just provide another --tool-call-regex:

<function=(?P<name>.*?)>(?P<arguments>.*?)<\/function>

which also extracts tool call(s): https://regex101.com/r/9pM6IL/2

Unfortunately this approach does not work with streaming. Supporting tools in streaming mode was a core requirement of this PR since many applications use streaming mode by default if they are user-facing, and there is no way to “turn off streaming” if the model starts generating a tool call.

[pbasov]

@K-Mistele Documentation would be much appreciated, so the community can be quick to add support for future models with tool support.

[meetzuber]

Only below options are mentioned in tool/function calling. and there are only 2 options for tool call parser mistral and hermes.
There is no --tool-call-regex option there.

--enable-auto-tool-choice – mandatory Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it deems appropriate.

--tool-call-parser – select the tool parser to use - currently either hermes or mistral. Additional tool parsers will continue to be added in the future.

--chat-template – optional for auto tool choice. the path to the chat template which handles tool-role messages and assistant-role messages that contain previously generated tool calls. Hermes and Mistral models have tool-compatible chat templates in their tokenizer_config.json files, but you can specify a custom template. This argument can be set to tool_use if your model has a tool use-specific chat template configured in the tokenizer_config.json. In this case, it will be used per the transformers specification. More on this here from HuggingFace; and you can find an example of this in a tokenizer_config.json here

https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#automatic-function-calling

[gislerro]

There is no --tool-call-regex option there.

That's not implemented on the main branch

[meetzuber]

Only below options are mentioned in tool/function calling. and there are only 2 options for tool call parser mistral and hermes.
There is no --tool-call-regex option there.

--enable-auto-tool-choice – mandatory Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it deems appropriate.

--tool-call-parser – select the tool parser to use - currently either hermes or mistral. Additional tool parsers will continue to be added in the future.

--chat-template – optional for auto tool choice. the path to the chat template which handles tool-role messages and assistant-role messages that contain previously generated tool calls. Hermes and Mistral models have tool-compatible chat templates in their tokenizer_config.json files, but you can specify a custom template. This argument can be set to tool_use if your model has a tool use-specific chat template configured in the tokenizer_config.json. In this case, it will be used per the transformers specification. More on this here from HuggingFace; and you can find an example of this in a tokenizer_config.json here

https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#automatic-function-calling

That's not implemented on the main branch

These are implemented in v0.6.0 release.
https://github.com/vllm-project/vllm/releases/tag/v0.6.0

[vipulgote1999]

@K-Mistele can mentioned which is the pull request for llama3.1 tool support...ithink most people is been waiting for it.

[JackYangzg]

@K-Mistele can mentioned which is the pull request for Qwen2 tool support...ithink most people is been waiting for it.

[K-Mistele]

@K-Mistele can mentioned which is the pull request for llama3.1 tool support...ithink most people is been waiting for it.

There is not a pull request for it yet because it's still a work-in-progress and not ready for review. I can create a draft though.

[K-Mistele]

@K-Mistele can mentioned which is the pull request for Qwen2 tool support...ithink most people is been waiting for it.

There is not a PR for this, and at this point I hadn't planned on it. If this is something you're interested, please feel free to create an issue for a feature request, and tag me in it. If it gets enough interest, I'll add it.