Add OpenAI Function Calling

[adrienbrault]

See https://platform.openai.com/docs/guides/function-calling

Using https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF

This is a proof of concept! I am thinking that all the Hermes2Pro specifics should be modeled somehow in the Modelfile.

Script to compare `gpt-3.5-turbo-0125` with this branch using `adrienbrault/nous-hermes2pro:Q4_K_M`

export OPENAI_FUNCTION_CALLING_REQUEST='{
  "model": "${MODEL}",
  "messages": [
    {
      "role": "user",
      "content": "What is the weather like in Seattle today?"
    }
  ],
  "tools": [
    {
      "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },
            "format": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"],
              "description": "The temperature unit to use. Infer this from the users location."
            }
          },
          "required": ["location", "format"]
        }
      }
    }
  ]
}'

export MODEL_OLLAMA="adrienbrault/nous-hermes2pro:Q4_K_M"
export MODEL_OPENAI="gpt-3.5-turbo-0125"
export OLLAMA_HOST=127.0.0.1:11435

go build . && ./ollama serve

curl "${OLLAMA_HOST}/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "$(echo "${OPENAI_FUNCTION_CALLING_REQUEST}" | MODEL="${MODEL_OLLAMA}" envsubst '${MODEL}')" -s \
  | jq . \
  | tee ollama.json

curl "https://api.openai.com/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${OPENAI_API_KEY}" \
  -d "$(echo "${OPENAI_FUNCTION_CALLING_REQUEST}" | MODEL="${MODEL_OPENAI}" envsubst '${MODEL}')" -s \
  | jq . \
  | tee openai.json

git diff --no-index openai.json ollama.json

diff --git a/openai.json b/ollama.json
index 59d4977..faaa2d6 100644
--- a/openai.json
+++ b/ollama.json
@@ -1,33 +1,32 @@
 {
-  "id": "chatcmpl-95hyRBATFpkeeajKuAc1D4y8gw29w",
+  "id": "chatcmpl-395",
   "object": "chat.completion",
-  "created": 1711147703,
-  "model": "gpt-3.5-turbo-0125",
+  "created": 1711149126,
+  "model": "adrienbrault/nous-hermes2pro:Q4_K_M",
+  "system_fingerprint": "fp_ollama",
   "choices": [
     {
       "index": 0,
       "message": {
         "role": "assistant",
-        "content": null,
+        "content": "",
         "tool_calls": [
           {
-            "id": "call_DBgXGrqufbzrp7rDN0VspAdw",
+            "id": "call_YxkK26AReyBuOGUJiQbLNGHq",
             "type": "function",
             "function": {
               "name": "get_current_weather",
-              "arguments": "{\"location\":\"Seattle\",\"format\":\"celsius\"}"
+              "arguments": "{\"format\":\"celsius\",\"location\":\"Seattle, WA\"}"
             }
           }
         ]
       },
-      "logprobs": null,
       "finish_reason": "tool_calls"
     }
   ],
   "usage": {
-    "prompt_tokens": 93,
-    "completion_tokens": 20,
-    "total_tokens": 113
-  },
-  "system_fingerprint": "fp_3bc1b5746c"
+    "prompt_tokens": 349,
+    "completion_tokens": 45,
+    "total_tokens": 394
+  }
 }

[interstellarninja]

hi Adrien, thank you for implementing openai function calling format with Hermes 2 Pro, does it also parse multiple tool calls?

[adrienbrault]

@interstellarninja Yes

Script to compare `gpt-3.5-turbo-0125` with this branch using `adrienbrault/nous-hermes2pro:Q4_K_M`

export OPENAI_PARRALEL_FUNCTION_CALLING_REQUEST='{
  "model": "${MODEL}",
  "messages": [
    {
      "role": "system",
      "content": "Save all characters and locations."
    },
    {
      "role": "user",
      "content": "Gina, a witty barista at The Busy Bean Coffee Shop, serves Tommy, a clumsy delivery guy, a triple shot espresso before he makes a delivery to Mrs. Wigglesworths mansion. At the mansion, Tommy delivers a life-sized stuffed peacock to the eccentric elderly woman. Gina surprises them both by showing up with coffee, and Mrs. Wigglesworth invites them to stay for tea and a game of croquet."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "save_character",
        "parameters": {
          "type": "object",
          "properties": {
            "name": {
              "type": "string"
            },
            "role": {
              "type": "string"
            }
          },
          "required": ["name", "role"]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "save_location",
        "parameters": {
          "type": "object",
          "properties": {
            "name": {
              "type": "string"
            }
          },
          "required": ["name"]
        }
      }
    }
  ]
}'

export MODEL_OLLAMA="adrienbrault/nous-hermes2pro:Q4_K_M"
export MODEL_OPENAI="gpt-3.5-turbo-0125"
export OLLAMA_HOST=127.0.0.1:11435

go build . && ./ollama serve

curl "${OLLAMA_HOST}/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "$(echo "${OPENAI_PARRALEL_FUNCTION_CALLING_REQUEST}" | MODEL="${MODEL_OLLAMA}" envsubst '${MODEL}')" -s \
  | jq . \
  | tee ollama-parallel.json

curl "https://api.openai.com/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${OPENAI_API_KEY}" \
  -d "$(echo "${OPENAI_PARRALEL_FUNCTION_CALLING_REQUEST}" | MODEL="${MODEL_OPENAI}" envsubst '${MODEL}')" -s \
  | jq . \
  | tee openai-parallel.json

git diff --no-index openai-parallel.json ollama-parallel.json

diff --git a/openai-parallel.json b/ollama-parallel.json
index dfcd523..6508c08 100644
--- a/openai-parallel.json
+++ b/ollama-parallel.json
@@ -1,57 +1,56 @@
 {
-  "id": "chatcmpl-97jBOuU6lQHZ8PdWw1e210BBoWAjE",
+  "id": "chatcmpl-306",
   "object": "chat.completion",
-  "created": 1711629006,
-  "model": "gpt-3.5-turbo-0125",
+  "created": 1711629026,
+  "model": "adrienbrault/nous-hermes2pro:Q4_K_M",
+  "system_fingerprint": "fp_ollama",
   "choices": [
     {
       "index": 0,
       "message": {
         "role": "assistant",
-        "content": null,
+        "content": "",
         "tool_calls": [
           {
-            "id": "call_6k0gGQ8X6vdwxKH0Eit5Mr5d",
+            "id": "call_gCH2fvdvdb6DdkIwmBemeIxx",
             "type": "function",
             "function": {
               "name": "save_character",
-              "arguments": "{\"name\": \"Gina\", \"role\": \"witty barista\"}"
+              "arguments": "{\"name\":\"Gina\",\"role\":\"witty barista at The Busy Bean Coffee Shop\"}"
             }
           },
           {
-            "id": "call_dPEgQrkWLO3LZqEV7NuWB38z",
+            "id": "call_Q2qrHxnkpUQuKg8ALK8O2Pem",
             "type": "function",
             "function": {
               "name": "save_character",
-              "arguments": "{\"name\": \"Tommy\", \"role\": \"clumsy delivery guy\"}"
+              "arguments": "{\"name\":\"Tommy\",\"role\":\"clumsy delivery guy\"}"
             }
           },
           {
-            "id": "call_A9scqcYJbePu2ncyPk9kJGNW",
+            "id": "call_ntYobidp8bojUXbGeSPAltWz",
             "type": "function",
             "function": {
               "name": "save_location",
-              "arguments": "{\"name\": \"The Busy Bean Coffee Shop\"}"
+              "arguments": "{\"name\":\"The Busy Bean Coffee Shop\"}"
             }
           },
           {
-            "id": "call_EStwY2sQUymP4X7vDDenMS81",
+            "id": "call_aJw5z8hzyTqM03GoaFeMYWXY",
             "type": "function",
             "function": {
               "name": "save_location",
-              "arguments": "{\"name\": \"Mrs. Wigglesworths mansion\"}"
+              "arguments": "{\"name\":\"Mrs. Wigglesworth's mansion\"}"
             }
           }
         ]
       },
-      "logprobs": null,
       "finish_reason": "tool_calls"
     }
   ],
   "usage": {
-    "prompt_tokens": 150,
-    "completion_tokens": 99,
-    "total_tokens": 249
-  },
-  "system_fingerprint": "fp_b28b39ffa8"
+    "prompt_tokens": 122,
+    "completion_tokens": 169,
+    "total_tokens": 291
+  }
 }

[olafgeibig]

Great stuff 🙌 Can't wait for this to be merged.

[interstellarninja]

awesome work mate 👏 this makes Hermes 2 Pro swappable with OpenAI function calling use cases. @Benjoyo was also interested in opening a PR for OpenAI function calling support for llama.cpp with this model, hope this implementation can serve as a guide there.

[lucifer2288]

awesome, thank you!

[adrienbrault]

This functionality is now available as a docker container that proxies the Ollama OpenAI compatible API:

$ docker run -it --rm \
    -e OPENAI_BASE_URI=http://docker.for.mac.host.internal:11434 \
    -p 11440:80 \
    adrienbrault/hermes2pro-proxy

$ curl "http://localhost:11440/v1/..." 

https://github.com/adrienbrault/hermes2pro-proxy

[K0IN]

Can we get the prompt template (for the specific model) to be defined inside the model info (Modelfile)?

ollama/server/images.go

Line 50 in 4c7db6b

Template string

So we can extend this for a future models?