Gateway mode can't offer AI Gateway custom providers — and there's no other way for a deployment to give users a model

[Justin-Dearing]

TLDR - AI Gateway lets you register a custom provider. Cloudflare OS's gateway mode can't offer it: the catalog is a compile-time table and the provider routes are four hardcoded paths, so the gateway's own provider registry is never consulted. And outside gateway mode there's no deployment-level model at all — everything is per-user. Net result: a team deployment cannot give its users a model on a custom endpoint by any supported path. We think this would be a great capability for team deployments, and the nice part is the two halves compose naturally.

---

We run a team deployment from the unmodified starter with a self-hosted model (vLLM behind an Anthropic-compatible gateway). Ideally we'd register it as an AI Gateway custom provider and have gateway mode offer it — central key, per-user cost logs, zero user setup, exactly what gateway mode is for. But:

1. In gateway mode, offerable models come from the compile-time SUGGESTED_MODELS table (workshop-shared/src/api.ts) filtered by CF_AI_GATEWAY_PROVIDERS, and inference URLs are built from four hardcoded paths in ai-models.ts (/anthropic, /openai, /google-ai-studio/v1beta, /workers-ai/v1). Nothing can construct a /custom-{slug}/… route, and CFOS never asks the gateway what providers exist.
2. Gateway mode also drops apiUrl/apiToken from user-added models (AddModelModal.tsx), so enabling the catalog removes the only manual escape hatch. It's managed catalog or custom endpoint, never both.
3. Outside gateway mode, models are strictly per-user: the only write path is addModel in user.ts, authenticated as that user. No deployment default, no admin surface, no seeding — ten teammates hand-paste the same URL and key ten times.

What a fix could look like. Since an AI Gateway custom provider is just a base URL — the gateway doesn't know its models, wire formats, or context windows — even the gateway-side fix needs deployment config declaring the models. Something like: deployment config lists models (id, display name, wire format, context window, and either a custom-{slug} gateway route or a direct URL + secret-backed key), and those appear in every user's picker, optionally as the default. That one shape covers both avenues: gateway-mode deployments get custom providers with the gateway's key custody and cost logging; non-gateway deployments get the same models pointed directly at the endpoint.

This hole has a workaround ecosystem already, which is usually the sign of a missing feature: #29 (generic OpenAI-compatible provider), #50 (admin control over offered models), #219 (a ~1,400-line overlay patch series adding deployment gateway models with an admin UI), and ours — we inject a script into the built frontend that auto-adds our model to each user's account on load, over the same authenticated /api websocket the frontend uses. It works, but everyone's building bootstrap hacks around the same wall from different directions.

We prototyped the non-gateway half on a private fork and the seam is small (three call sites in user.ts, plus a modelTokenWindow branch for models outside the catalogs). Happy to share notes or test against our deployment — per CONTRIBUTING, deliberately not attaching a PR.

[dkattan]

This is a massive oversight and also likely a maintenance nightmare for you guys as model names from providers change constantly. You should be using discovery against the providers and offering the models in a dynamic fashion back to Cloudflare OS

[dkattan]

Like already it's behind: Where's GLM-5.3?

cloudflare-os/packages/workshop-shared/src/api.ts

Line 1191 in 6478a14

"@cf/zai-org/glm-5.2": {