Background
Cloudflare AI Gateway sits in front of provider APIs as a reverse proxy: network-level 429 interception, cross-Worker rate-limit aggregation, semantic response caching, and latency-based load balancing — all without touching app code. llm-providers handles semantic routing, workload-aware model selection, circuit breakers, and schema validation. These solve different sub-problems and compose cleanly.
Current state
Each HTTP provider (OpenAIProvider, AnthropicProvider, CerebrasProvider, GroqProvider, NvidiaProvider) already accepts a baseUrl option. A user can manually set this to a CF AI Gateway URL today, but:
- There is no typed
cfGateway shorthand — users must construct the URL string themselves and know the per-provider path suffix
- CF AI Gateway-specific request headers (
cf-aig-cache-ttl, cf-aig-cache-key, cf-aig-skip-cache) are not forwarded from request.gatewayMetadata, even though gatewayMetadata exists in LLMRequest
- The composable layering pattern is undocumented
Proposed change
Add an optional cfGateway block to each HTTP provider config:
interface CfGatewayConfig {
accountId: string;
gatewayId: string;
}
// Added to OpenAIConfig, AnthropicConfig, CerebrasConfig, GroqConfig, NvidiaConfig
cfGateway?: CfGatewayConfig;
Provider constructors: when cfGateway is set, derive baseUrl as:
| Provider |
Gateway base URL |
| openai |
https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/openai/v1 |
| anthropic |
https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/anthropic |
| cerebras |
https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/cerebras/v1 |
| groq |
https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/groq/openai/v1 |
| nvidia |
https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/nvidia-nim/v1 |
An explicit baseUrl overrides cfGateway (existing behaviour preserved).
Each provider's makeRequest should forward these headers when cfGateway is active and the values are present on request.gatewayMetadata:
cf-aig-cache-ttl ← gatewayMetadata.cacheTtl
cf-aig-cache-key ← gatewayMetadata.cacheKey
cf-aig-skip-cache ← gatewayMetadata.skipCache
What this does NOT change
buildProviderPlan / circuit breaker / schema drift / tool loop — unchanged
CacheObservability.aiGateway reporting — already wired, just ensure headers flow so CF GW actually acts on them
- Cloudflare Workers AI provider — uses the
env.AI binding, no HTTP base URL
Acceptance criteria
Background
Cloudflare AI Gateway sits in front of provider APIs as a reverse proxy: network-level 429 interception, cross-Worker rate-limit aggregation, semantic response caching, and latency-based load balancing — all without touching app code. llm-providers handles semantic routing, workload-aware model selection, circuit breakers, and schema validation. These solve different sub-problems and compose cleanly.
Current state
Each HTTP provider (
OpenAIProvider,AnthropicProvider,CerebrasProvider,GroqProvider,NvidiaProvider) already accepts abaseUrloption. A user can manually set this to a CF AI Gateway URL today, but:cfGatewayshorthand — users must construct the URL string themselves and know the per-provider path suffixcf-aig-cache-ttl,cf-aig-cache-key,cf-aig-skip-cache) are not forwarded fromrequest.gatewayMetadata, even thoughgatewayMetadataexists inLLMRequestProposed change
Add an optional
cfGatewayblock to each HTTP provider config:Provider constructors: when
cfGatewayis set, derivebaseUrlas:https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/openai/v1https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/anthropichttps://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/cerebras/v1https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/groq/openai/v1https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/nvidia-nim/v1An explicit
baseUrloverridescfGateway(existing behaviour preserved).Each provider's
makeRequestshould forward these headers whencfGatewayis active and the values are present onrequest.gatewayMetadata:cf-aig-cache-ttl←gatewayMetadata.cacheTtlcf-aig-cache-key←gatewayMetadata.cacheKeycf-aig-skip-cache←gatewayMetadata.skipCacheWhat this does NOT change
buildProviderPlan/ circuit breaker / schema drift / tool loop — unchangedCacheObservability.aiGatewayreporting — already wired, just ensure headers flow so CF GW actually acts on themenv.AIbinding, no HTTP base URLAcceptance criteria
cfGatewayoption accepted on all 5 HTTP provider configs with correct typesbaseUrlderivation is correct per-provider (unit-testable, no real HTTP needed)gatewayMetadatafields are setbaseUrloverride still works (no regression)