Skip to content

feat: first-class cfGateway config for per-provider AI Gateway passthrough #97

Description

@stackbilt-admin

Background

Cloudflare AI Gateway sits in front of provider APIs as a reverse proxy: network-level 429 interception, cross-Worker rate-limit aggregation, semantic response caching, and latency-based load balancing — all without touching app code. llm-providers handles semantic routing, workload-aware model selection, circuit breakers, and schema validation. These solve different sub-problems and compose cleanly.

Current state

Each HTTP provider (OpenAIProvider, AnthropicProvider, CerebrasProvider, GroqProvider, NvidiaProvider) already accepts a baseUrl option. A user can manually set this to a CF AI Gateway URL today, but:

  • There is no typed cfGateway shorthand — users must construct the URL string themselves and know the per-provider path suffix
  • CF AI Gateway-specific request headers (cf-aig-cache-ttl, cf-aig-cache-key, cf-aig-skip-cache) are not forwarded from request.gatewayMetadata, even though gatewayMetadata exists in LLMRequest
  • The composable layering pattern is undocumented

Proposed change

Add an optional cfGateway block to each HTTP provider config:

interface CfGatewayConfig {
  accountId: string;
  gatewayId: string;
}

// Added to OpenAIConfig, AnthropicConfig, CerebrasConfig, GroqConfig, NvidiaConfig
cfGateway?: CfGatewayConfig;

Provider constructors: when cfGateway is set, derive baseUrl as:

Provider Gateway base URL
openai https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/openai/v1
anthropic https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/anthropic
cerebras https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/cerebras/v1
groq https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/groq/openai/v1
nvidia https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayId}/nvidia-nim/v1

An explicit baseUrl overrides cfGateway (existing behaviour preserved).

Each provider's makeRequest should forward these headers when cfGateway is active and the values are present on request.gatewayMetadata:

  • cf-aig-cache-ttlgatewayMetadata.cacheTtl
  • cf-aig-cache-keygatewayMetadata.cacheKey
  • cf-aig-skip-cachegatewayMetadata.skipCache

What this does NOT change

  • buildProviderPlan / circuit breaker / schema drift / tool loop — unchanged
  • CacheObservability.aiGateway reporting — already wired, just ensure headers flow so CF GW actually acts on them
  • Cloudflare Workers AI provider — uses the env.AI binding, no HTTP base URL

Acceptance criteria

  • cfGateway option accepted on all 5 HTTP provider configs with correct types
  • baseUrl derivation is correct per-provider (unit-testable, no real HTTP needed)
  • CF AI Gateway headers forwarded when config is present and gatewayMetadata fields are set
  • Existing baseUrl override still works (no regression)
  • README documents the composable pattern (llm-providers for semantic routing + circuit breakers; CF AI GW for network-layer LB and 429 interception)
  • CHANGELOG entry under next minor

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions