Context usage count is always 0 for some local openai-compatible providers

[unkhz]

The openai-compatible models work nicely, but token counts are not calculated correctly, which may cause other issues. E.g. the auto summarization of the conversation will not happen when token limits are approaching.

Problem occurs at least with these local LLM servers:

• LM studio on Mac
• Ollama on Windows

Problem does NOT occur with

• llama.cpp

Below an example response from LM studio. The tokens are sent by LM studio, but they appear to be NaN when debug logging the response.

Not sure if this is ai-sdk or opencode issue.

Config

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "lmstudio": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:8080/v1"
      },
      "models": {
        "qwen/qwen3-30b-a3b": {
          "limit": { "context": 131072, "output": 4096 }
        },
        "mistralai/devstral-small-2505": {
          "limit": { "context": 131072, "output": 4096 }
        }
      }
    }
  }
}

Usage coming from ai-sdk

Raw query towards LM studio endpoint

$ curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen/qwen3-30b-a3b","messages":[{"role":"user","content":"hello"}]}'

Raw output from LM studio endpoint

{
  "id": "chatcmpl-nziofmbwmwbf87ynkt0bn",
  "object": "chat.completion",
  "created": 1750951509,
  "model": "qwen/qwen3-30b-a3b",
  "choices": [
    {
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "<think>\nOkay, the user sent \"hello\". I need to respond appropriately. Since it's a greeting, the best approach is to greet them back and offer help. I should keep it friendly and open-ended. Let me make sure the response is welcoming and encourages them to ask questions. Maybe something like, \"Hello! How can I assist you today?\" That should work.\n</think>\n\nHello! How can I assist you today? 😊"
      }
    }
  ],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 89,
    "total_tokens": 98
  },
  "stats": {},
  "system_fingerprint": "qwen/qwen3-30b-a3b"
}

[unkhz]

Possibly related to #295. Token cost is 0 if count is NaN. Perhaps all "@ai-sdk/openai-compatible" models are suffering from this.

[unkhz]

Gained a bit of extra info from step debugging. Seems the simpler generateText receives the usage just fine, but the complex streamText gets NaN's. It's also not LM studio issue, same happens with ollama running on different machine, when using the openai-compatible provider.

There's tiny chance opencode can fix this in configuration of the provider when instantiating it, but I'm fairly sure now the issue is in ai-sdk side. This troubleshooting doc suggests it's something they've dealt with: content/docs/09-troubleshooting/20-nan-token-counts-openai-streaming.mdx. However, the suggested config compatibility: 'strict' had no effect for me, so I think the openai-compatible provider has different manifestation of this.

I'll dig deeper during the weekend and might open an issue in the openai-compatible project.

[unkhz]

Ok got some clarity.

1. The LLM server is responsible for sending the usage, so naturally there's some variance between tools. For example llama.cpp always sends it. Current workaround is to use llama.cpp for serving local models
2. OpenAI spec has a request body parameter that can be set for ensuring the usage is reported. https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options. This is respected by some servers like Ollama and apparently vLLM as well.
3. The ai-sdk has a bug currently, which makes it impossible to enable this stream_options parameter. providerOptions.stream_options No Longer Effective in Latest SDK, Preventing Usage Data for DeepSeek/Volcengine vercel/ai#6774

When the ai-sdk bug is fixed we have a few options

1. Pass the stream_options as is from opencode.json config
2. Automatically set it to true in code, since the code relies on it

I'd lean towards option 2.

I verified that setting the stream_options parameter fixes the NaN usage for at least Ollama. I have the code in a branch, but it's not the best fix, as it might have nasty side effects dev...unkhz:opencode:fix/use-include-usage-parameter

[jessevdp]

I just went through almost this exact debugging experience just to end up in the same place. (I hadn't read this thread....) This needs to be fixed upstream!

vercel/ai#6774

[jessevdp]

There seems to be some PR that was merged into a v4 branch but I can't seem to figure out if that's shipped yet? It also seems to not be present on the v5 branch so who knows what that means..

vercel/ai#6849

[jessevdp]

I proposed a change to the ai-sdk that would allow users to simply configure includeUsage in their opencode provider config like so:

{
  "provider": {
    "your-provider": {
      "name": "YourProvider",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "https://your-provider.com",
        "apiKey": "...",
        "includeUsage": true
      },
      "models": {
        "your-model": {
          "name": "Your Model",
        }
      }
    }
  }
}

See: vercel/ai#7441

Let's hope it's merged!

[unkhz]

Nice one @jessevdp, the fix is included in latest @ai-sdk/openai-compatible beta release now 👍. If I understand correctly, the opencode provider install logic includes betas, so in theory the includeUsage option should be workable already via opencode.json. Trying it out at the moment, but could not yet get context usage count updating by adding the streamOptions: {includeUsage: true} into the model options. I'll try digging into it a bit more. https://github.com/sst/opencode/blob/c65e7aff86504a3386ed7ca137b8bc097439e2ce/packages/opencode/src/provider/provider.ts#L356

[unkhz]

Nevermind I was using wrong spell. Adding includeUsage: true (without streamOptions) to any model fixes ollama, so this one is the best workaround for now. Example provider config below:

    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://citra.local:11434/v1",
        "compatibility": "strict",
        "includeUsage": true
      },
      "models": {
        "llama3.1:8b-instruct-q8_0": {
          "limit": { "context": 131072, "output": 4096 }
        },
        "gemma3n:e2b": {
          "limit": { "context": 131072, "output": 4096 }
        }
      }
    },

[unkhz]

So we have the tools to fix this now. The remaining question is, how to avoid every user going to this issue to understand why context count is 0, should opencode do something like:

1. Force this includeUsage: true option by default in provider.ts.
2. Tell users that they need to specifically tell opencode to includeUsage: true e.g. "if context is always 0, try using this option"
3. Add openai-compatible as a new case in CUSTOM_LOADERS and set sane default options for it

Any thoughts on this @thdxr?

[jessevdp]

Yeah I didn't add a default value because I didn't want this to be a breaking change on their SDK and risk this taking even longer...

Perhaps we can start with an example in the docs?

How is "stream include usage" turned on for other providers? Or is it default for those?