Skip to content

Added support for cloudflare Workers AI via the OpenAI endpoint #4846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Stono
Copy link

@Stono Stono commented Jul 11, 2025

Hi,
This PR adds support for Cloudflare Workers AI
It's quite a nice PaaS model environment.
https://developers.cloudflare.com/workers-ai/

I did originally add @langchain/cloudflare, however that seemed incomplete, didn't support tools etc. Cloudflare provide openai compatible endpoints for their chat models so I'm instead using that.

The model data is generated by hitting GET https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/models/search; some of them are 0 cost because "beta" models don't have a cost.

It was reasonably easy to add a new provider so kudos for the extensible pattern.

@Stono Stono force-pushed the add-cloudflare-workers-ai branch from 7009a1b to 99594bf Compare July 11, 2025 13:44
@Stono
Copy link
Author

Stono commented Jul 11, 2025

Hmm. I suppose one thing I don't like actually is that function calling does not work on all the models, it's only supported by 4 of them:

https://developers.cloudflare.com/workers-ai/models/?capabilities=Function+calling

Attempting it on another model and you get a 400.

Event with the "supported" models, it doesn't appear to work:

curl https://api.cloudflare.com/client/v4/accounts/account-id/ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer token" \
  -d '{
    "model": "@hf/nousresearch/hermes-2-pro-mistral-7b",
    "messages": [
      {
        "role": "user",
        "content": "What is the weather like in Boston today?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather for a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g., San Francisco, CA"
              },
              "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"]
              }
            },
            "required": ["location"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'

Gives:

{"id":"id-1752242603477","object":"chat.completion","created":1752242603,"model":"@hf/nousresearch/hermes-2-pro-mistral-7b","choices":[{"index":0,"message":{"role":"assistant","content":null},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

eg; null content, no tool calls, just no content. Same with @cf/meta/llama-4-scout-17b-16e-instruct.

I'll poke this over to some people I know at Cloudflare to see what they say.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant