feat(tools): support Tool calls in the API

[mudler]

Description

Part of #1712

Notes for Reviewers

still missing stream support

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	d44c8ba
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/65cfe68ed3fc8000080b7915

[stippi2]

As for the missing streaming support... Do you think it might be an idea to stream contents as implemented now and then wrap the entire function call output in one big streaming event? In my application, I would have to wait for the function anyway, before I can do anything. (I guess there are other use-cases where you force a JSON response and the chat response is one argument that should also stream. But one big event for the function would already enable my case.)

[mudler]

As for the missing streaming support... Do you think it might be an idea to stream contents as implemented now and then wrap the entire function call output in one big streaming event? In my application, I would have to wait for the function anyway, before I can do anything. (I guess there are other use-cases where you force a JSON response and the chat response is one argument that should also stream. But one big event for the function would already enable my case.)

yes, that's exactly what I'm looking at now - at least for the moment we provide compatibility, we can get back at this later on to try to stream the whole result from the LLM directly

[stippi2]

I'm amazed by your quick progress! This is the project I'm working on: https://github.com/stippi/voice-assistant
I've added a section to the README for what to configure in order to use LocalAI.

[mudler]

I'm amazed by your quick progress! This is the project I'm working on: https://github.com/stippi/voice-assistant I've added a section to the README for what to configure in order to use LocalAI.

that's super cool! I'd be happy to add it in the community section in the README :)

[mudler]

I didn't give it a shot yet at this PR, but I think most of the pieces should be in place by now, if you have some cycles to spend and test this out would be great!

Thanks for the quick feedback @stippi2, that really helps me through, really appreciated!

[stippi2]

Ok, I'll give it a shot. Maybe two questions.

1. When I reply with a tool result, the payload is also a bit different compared to the older functions. I didn't notice that in the diff yet.
2. GPT-4 will, depending on the situation, call multiple tools at once and optionally give a chat reply, too. Is that already supported/transported from the local LLMs in LocalAI?

[mudler]

Ok, I'll give it a shot. Maybe two questions.

1. When I reply with a tool result, the payload is also a bit different compared to the older functions. I didn't notice that in the diff yet.

This should be covered by dddd67d (non-streaming mode), for streaming mode we just reply with the tools format (as we didn't supported streaming functions before, I would not port deprecated APIs to the new SSE feature)

2. GPT-4 will, depending on the situation, call multiple tools at once _and_ optionally give a chat reply, too. Is that already supported/transported from the local LLMs in LocalAI?

Multiple tools at once is not supported (yet), see #1275, as it requires changes to the BNF grammar and would probably require few rounds of tests first, but replies in case a tool is not selected is supported - however, at the moment I didn't wired it up with streaming responses, but that's easy to get after this bunch of changes.

[stippi2]

This is an example messages array I see in the network tab of the Chrome dev tools:

  {
      "role": "user",
      "content": "Play songs from Peter Fox, please."
    },
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "call_C8tn5mmpCnlWgaSBvIYXv85p",
          "type": "function",
          "function": {
            "name": "find_artists_and_play_top_songs_on_spotify",
            "arguments": "{\"queries\":[\"Peter Fox\"]}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "name": "find_artists_and_play_top_songs_on_spotify",
      "tool_call_id": "call_C8tn5mmpCnlWgaSBvIYXv85p",
      "content": "{\"result\":\"playback started\"}"
    }

[mudler]

tool_call_id

Ok, I'll give it a shot. Maybe two questions.

1. When I reply with a tool result, the payload is also a bit different compared to the older functions. I didn't notice that in the diff yet.

🤦 I didn't read it right. I totally missed up the reply to a tool result. Will try to have a look at it later/tomorrow

Update:

well, thinking again, actually that should be covered already without any changes - just map the "tool" role in the model config. We should pass the name, but for a first implementation should be already working.

[stippi2]

Had a chance to try it out, but it crashes. I attached a debugger at the spot and this is what I see:

[stippi2]

well, thinking again, actually that should be covered already without any changes - just map the "tool" role in the model config. We should pass the name, but for a first implementation should be already working.

To what string should it be mapped?

[mudler]

well, thinking again, actually that should be covered already without any changes - just map the "tool" role in the model config. We should pass the name, but for a first implementation should be already working.

To what string should it be mapped?

It would ideally depend on how the model is fine-tuned, if it didn't saw any "tool" or function it might be problematic. The role is used when constructing back the prompt feeded back to the LLM

Had a chance to try it out, but it crashes. I attached a debugger at the spot and this is what I see:

Going to have a look soon

[mudler]

okay, this should be working now at least - I didn't give it a deep look into the API diffs, but I think we are very much closer now

[stippi2]

Very close, but not quite:

[mudler]

mm that looks more a model misconfiguration - or somehow the LLM output is not entirely JSON at the end.

Did you tried to set the stopwords on the model?

Example:

stopwords:
- "<dummy32000>"

[stippi2]

This is my config:

name: mistral
mmap: true
parameters:
  model: mistral-7b-openorca.Q6_K.gguf
  temperature: 0.2
  top_k: 40
  top_p: 0.95
template:
  chat_message: chatml
  chat: chatml-block
  completion: completion
context_size: 4096
f16: true
stopwords:
- <|im_end|>
threads: 4

[mudler]

This is my config:

name: mistral
mmap: true
parameters:
  model: mistral-7b-openorca.Q6_K.gguf
  temperature: 0.2
  top_k: 40
  top_p: 0.95
template:
  chat_message: chatml
  chat: chatml-block
  completion: completion
context_size: 4096
f16: true
stopwords:
- <|im_end|>
threads: 4

I think you should add the string you see in there in the final string in the output as a stopword, definitely should be not part of the response as we force JSON - but seems it is not entirely JSON in your case

[stippi2]

Will try, thanks.

[mudler]

For reference:

name: mistral
mmap: true
parameters:
  model: mistral-7b-openorca.Q6_K.gguf
  temperature: 0.2
  top_k: 40
  top_p: 0.95
template:
  chat_message: chatml
  chat: chatml-block
  completion: completion
context_size: 4096
f16: true
stopwords:
- <|im_end|>
- <dummy32000>
threads: 4

[stippi2]

With that change, the function call works. Still waiting for the chat response after sending the tool result to the model. Note that I didn't add a new role mapping anywhere. Should I?

[stippi2]

Hm. It keeps calling the tool. Are you sure the LLM is being forwarded the tool result? I'll try to add a mapping for the "tool" role, if I can find where it needs to be added.

    {
      "role": "user",
      "content": "Hi. How are you?"
    },
    {
      "content": "I'm doing well, thank you for asking. How can I assist you today?\n\n",
      "role": "assistant"
    },
    {
      "role": "user",
      "content": "Can you tell me the current weather?"
    },
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "9618f662-741e-49b3-8a92-87b6d2b4567e",
          "type": "function",
          "function": {
            "name": "get_current_weather",
            "arguments": "{\"latitude\":52.3497672,\"longitude\":13.3008244}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "name": "get_current_weather",
      "tool_call_id": "9618f662-741e-49b3-8a92-87b6d2b4567e",
      "content": "{\"result\":{\"weather\":\"light intensity shower rain\",\"temperature\":12,\"temperature_feels_like\":12,\"humidity\":93,\"wind_speed\":4.63,\"wind_direction\":270}}"
    },
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "9618f662-741e-49b3-8a92-87b6d2b4567e",
          "type": "function",
          "function": {
            "name": "get_current_weather",
            "arguments": "{\"latitude\":52.349785,\"longitude\":13.300821}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "name": "get_current_weather",
      "tool_call_id": "9618f662-741e-49b3-8a92-87b6d2b4567e",
      "content": "{\"result\":{\"weather\":\"light intensity shower rain\",\"temperature\":12,\"temperature_feels_like\":12,\"humidity\":93,\"wind_speed\":4.63,\"wind_direction\":270}}"
    }

[mudler]

   {
      "role": "tool",
      "name": "get_current_weather",
      "tool_call_id": "9618f662-741e-49b3-8a92-87b6d2b4567e",
      "content": "{\"result\":{\"weather\":\"light intensity shower rain\",\"temperature\":12,\"temperature_feels_like\":12,\"humidity\":93,\"wind_speed\":4.63,\"wind_direction\":270}}"
    },

It does, if you turn off streaming, you should be able to see the prompt being passed by, for instance:

9:49AM DBG Prompt (before templating): What is the weather like in Boston?
test result sent back from the client
9:49AM DBG Prompt (after templating): What is the weather like in Boston?
test result sent back from the client

And the request is:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{                                                                                                   
  "model": "phi-2", 
  "messages": [          
    {              
      "role": "user",                 
      "content": "What is the weather like in Boston?"                                                                                                                                                                             
    },    
   {            
      "role": "tool",      
      "name": "get_current_weather",
      "tool_call_id": "9618f662-741e-49b3-8a92-87b6d2b4567e",
      "content": "test result sent back from the client"
    }                                                                    
  ],          
  "tools": [         
    {                          
      "type": "function",                      
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },       

What you are seeing is the LLM model capacity to do follow-ups, which is a bit limited. For function calling I can tell you need at least models bigger than >30b to perform "good enough". But I can tell by experience if you want something meaningful you need to aim at 70b models or leverage speculative sampling with even bigger ones.

I've summed up some of my experience with it in https://github.com/mudler/LocalAGI - feel free to have a look in there, it basically forces the LLM to reason over the results and it improves results even with smaller models (but still, you need at least a 30b model, 7b won't be enough).

Edit: You can alternatively also play with roles mapping to check how the prompt is formatted back to the LLM. It improves accuracy by a long shot if the model can recognize correctly results. Proper templating helps in that, but still my suggestion above is relevant (use bigger models, 7b won't cut it)

[mudler]

I'm going to merge this as it looks good from a functional perspective, LLM/models limitation is out of scope of this PR. I'll add tests as soon as I get a bit more time to test this from master images in my home setup.

[stippi2]

Ok, thanks for the insight, and especially for implementing this so quickly!! Awesome!