Custom Providers with no Budget

[deanbouvier]

Perhaps I'm missing something but Custom Providers once added don't have a budget or way to assign rate limits when you create them, for some like local vLLM and Ollama they are my own compute so while slower and more limited have no budget essentially, also if I were to add a unique one that is not in the providers list but still could have limits and budgets I don't see a way to define these, all I can do is input the models.

The end result is that my local Models are returning a 429 error when using the routing, I can use them fine with CURL or Playground but from something like Claude Code terminal I can load a particular local model but they will generate an API Error when I try to use them in Chat.

Is there something I need to do that I'm missing, or how is Custom Provider meant to work within the routing? My assumption is I could either use my local as last step fall back, or I could force a route by using a particular name since cloud models tend not to exist in the smaller local flavours its easy to do, but it always falls with API error.

Any help would be appreciated!

[tashfeenahmed]

Thanks for the detailed write up, this is a good report and two separate things are going on.

On budgets: you are not missing anything, and the behaviour you want is already what happens. Custom models are created with no RPM, RPD, TPM or TPD limits at all, so the router treats them as unmetered. That is deliberate for exactly your case, local vLLM and Ollama on your own compute. If you do want limits on a custom model, open it from the Models page and set them in its settings, along with context window and the routing ranks.

On the 429: that is almost certainly not a rate limit, and I am sorry the status code sent you down the wrong path. When the router cannot find any candidate that satisfies a request, it reports 429 with "code": "routing_exhausted" in the body. Check that field first. If it says routing_exhausted rather than rate_limit_exceeded, nothing was throttled, the model was filtered out.

The most likely reason in your case is tool calling. Claude Code sends tools on every request, and the router skips any model not marked as supporting them. Your local models were probably registered with tool support off. Open the model and turn it on, then retry from Claude Code. That also fits your symptom exactly, since Playground and plain curl do not send tools, which is why those work.

To confirm before changing anything, look at the X-Fallback-Trail response header or the attempt trail in the error body. It names each candidate and why it was rejected, for example no tool-calling support or custom-key-mismatch.

If tool support is already on and you still get this, paste that trail here and I will take a look.