Skip to content

Add multi-protocol Portkey gateway provider - #54

Merged
wch merged 5 commits into
mainfrom
portkey-provider
Aug 10, 2026
Merged

Add multi-protocol Portkey gateway provider#54
wch merged 5 commits into
mainfrom
portkey-provider

Conversation

@wch

@wch wch commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Adds a first-class portkey provider so organizations routing LLM traffic through a Portkey gateway can use it with each model speaking its natural wire protocol, mirroring the landed LiteLLM provider. Two deployment shapes are supported: hosted Portkey (https://api.portkey.ai/v1, Portkey API key, Model Catalog @provider/model ids with automatic discovery) and a self-hosted OSS gateway (the stored key is one upstream's key, models are user-declared). Claude models route over the Anthropic-shaped /v1/messages passthrough so explicit prompt-cache breakpoints and thinking signatures survive; other families route over the OpenAI-shaped endpoints. The provider is conservative where verification is still pending: hosted discovery currently stamps Claude-family models only, because hosted Portkey behavior could not be probed without an account.

Portkey can also back providers.custom.<name> entries with type: "portkey". The custom provider id owns model-fetch/cache identity and model stamping, while chat resolves through the factory registered under the portkey client kind, so live type changes follow the current catalog entry. Current user surfaces support a keyless, noncanonical self-hosted gateway or credential-injecting front proxy: models use bare upstream ids declared in models.custom, and the ordinary self-hosted path performs no HTTP discovery. A securely stored key for a custom id can reach the shared hosted fetcher, but the canonical hosted origin always requires a non-empty key and there is no custom-id credential UI or environment mapping yet.

Changes

  • ai-config: portkey added to the provider/client-kind vocabulary, built-in schema/catalog builder, and the base-only providers.custom discriminated union (with providers.schema.json regenerated); stripCatalogSlug exported as the single owner of the @slug/ id convention; model-capabilities/portkey-helpers.ts with classifyPortkeyModel (one per-entry decision object — family, capability model id, supported/excluded with reason, protocol — consumed by both discovery filtering and capability stamping) and getPortkeyModelCapabilities (Anthropic/OpenAI helper delegation, conservative fallback); canonical hosted base-URL constant in the base-URL policy module; PORTKEY_BASE_URL connection env mapping.
  • ai-provider-bridge: new providers/portkey-provider.ts built around a single resolvePortkeyConnection seam that both the client factory and model fetchers consume — required base URL, hosted-vs-OSS classification (exact canonical HTTPS origin only; lookalike hosts classify as OSS, http://api.portkey.ai is a local error before any request), mode-dependent key validation (hosted requires a non-empty key; OSS permits an empty key for an injecting proxy), /v1 normalization, and case-insensitive provider-local filtering of x-portkey-api-key/x-portkey-virtual-key from customHeaders. The chat client is a protocol-dispatching wrapper over one AnthropicClient and one OpenAIClient delegate (mirroring the LiteLLM dispatcher), with a same-gateway check that rejects per-request baseUrl overrides pointing at a different host before any request is made. Model discovery uses the fetchFresh seam on createCachedModelFetcher so Portkey owns its hosted pagination loop (with an atomic no-progress guard), the OSS no-fetch short-circuit, and the chat-vs-discovery header split (routing headers never sent to GET /v1/models). registerCustomPortkeyProvider registers a custom-id fetcher with independent cache/model identity and the shared kind-keyed client factory. Also fixes blank AI-SDK error messages (clarifyBlankRequestError in ai-sdk-helpers.ts) surfaced by Portkey's error shapes, benefiting all AI-SDK clients.
  • ai-credentials: PORTKEY_API_KEY env mapping for the built-in provider, plus the custom portkey auth descriptor (apikey, optional at catalog-synthesis time so a base-URL-only OSS/front-proxy connection can run).

Design decisions

  • Base URL is required, not defaulted. For Portkey the URL determines what the stored key is (a Portkey key vs. an upstream provider's key), so defaulting it would silently reinterpret the secret — a self-hoster's forgotten URL would send their upstream key to hosted Portkey. Key-only configs instead fail locally, before any network request, with an error naming PORTKEY_BASE_URL and the configure form.
  • Connection resolution owns mode validity. Custom catalog synthesis may produce an empty API key when a base URL is present, but resolvePortkeyConnection decides whether that credential is valid for the resolved deployment mode: hosted rejects empty/whitespace keys locally, while OSS accepts an empty key for a credential-injecting proxy. Discovery and chat consume the same rule.
  • Custom fetchers are id-keyed and the client factory is kind-keyed. Each custom gateway gets independent cache state and correctly stamped model ids. Chat follows the catalog's current clientKind, avoiding a stale custom-id factory after a live providers.json type change.
  • Secret headers are filtered provider-locally, never in the shared SDK-managed header filter: those names are Portkey credentials, and a global rule would strip them from existing non-Portkey gateway configurations. The stored key is re-added as a provider-owned header after sanitization so config can never override it.
  • Hosted discovery stamps Claude-family only until the hosted probe gate can run (no Portkey account was available); other families are excluded with a recorded reason rather than shipped unverified. Widening is additive when the probes run.
  • The dispatcher is mirrored from LiteLLM, not extracted: Portkey's per-mode credential wiring (hosted needs x-portkey-api-key on both delegates) does not parameterize cleanly; the convention stays documented in providerGuide.md.

Verification

Probed against a local OSS gateway (1.15.2): streaming /v1/messages passthrough, /v1/chat/completions with tools across OpenAI/Anthropic/Gemini upstreams, stateless encrypted-reasoning continuity over /v1/responses, and Bedrock passthrough with cache and thinking intact. Custom-provider tests cover schema/catalog preservation, custom-id model stamping and Portkey capability inference, operation without the built-in registrar, keyless OSS no-fetch behavior, and local rejection of keyless hosted discovery/chat. ai-lib check-types (including the shape guards), lint, and all package test suites pass.

Not included

  • Hosted Portkey verification (blocked on account access; the provider ships conservative until then).
  • A user-facing credential form or environment mapping for arbitrary custom provider ids. Custom Portkey is therefore documented as keyless/noncanonical on current user surfaces; use the built-in provider for one keyed gateway.
  • Multi-upstream OSS connections (needs a multi-secret credential shape).

Known limitations

  • OSS gateway 1.15.2 is unusable from a Node client without a shim: every streaming request 500s under Node 24 (gateway bug in undici header handling), and under Bun it mislabels Content-Encoding: gzip. A fixed gateway release resolves both upstream.

wch added 5 commits August 9, 2026 21:10
A first-class portkey provider routing each model over its natural wire
protocol against one gateway, in two modes derived from the required
base URL (the URL determines what the stored key is, so it can never
be defaulted):

- Hosted (exact https://api.portkey.ai origin only): key sent as
  x-portkey-api-key; paginated Model Catalog discovery of @slug/model
  ids via a new fetchFresh seam on createCachedModelFetcher.
  Conservatively stamps the Claude family only until the hosted Phase 0
  probes run (TODO(phase0-gate) markers); other families are excluded
  from discovery rather than shipped unverified.
- Self-hosted OSS (any other URL): the key is the single upstream's
  key, no discovery (GET /v1/models is broken on the OSS gateway,
  re-verified 2026-08-08); models declared via models.custom with
  optional per-model protocol; x-portkey-provider defaults to anthropic
  with user-supplied routing headers winning.

resolvePortkeyConnection is the single owner of URL classification,
/v1 normalization, secret-header sanitization (x-portkey-api-key /
x-portkey-virtual-key stripped case-insensitively from customHeaders),
and the chat/discovery header split (routing headers chat-only). The
protocol-dispatching client mirrors the litellm dispatcher (extraction
deferred until a third gateway provider) and rejects per-request
baseUrl overrides that don't normalize to the connection's gateway,
closing a credential-redirection hazard.

Also: stripCatalogSlug in ai-config as the shared @slug/ convention
owner; classifyPortkeyModel decision object + capability helpers with
an ID-only Claude-or-conservative rule (TODO(phase1));
clarifyBlankRequestError in the shared AI-SDK seam so blank
APICallError/RetryError messages (e.g. Portkey's {"html-message":...}
401 and {"status":"failure"} envelopes) surface status + body excerpt
for every provider; PORTKEY_API_KEY / PORTKEY_BASE_URL env mappings.

Backed by 2026-08-08 OSS gateway probes (1.15.2): tools/streaming/
images pass for anthropic, openai, and gemini upstreams; /v1/responses
stateless reasoning continuity passes for openai; Bedrock translation
preserves cache_control and thinking signatures.
… guard

Review findings on the initial portkey provider:

- Discovery headers now pass through the shared additive-header filter
  (additiveHeaderRecord). The provider-owned fetchFresh fetch bypasses
  the cached fetcher's merge, so SDK-managed headers (Authorization,
  x-api-key, ...) placed in customHeaders were reaching hosted
  discovery even though the chat delegates strip them. Mixed-case
  regression fixtures added.
- The pagination "no-progress guard" only detected an empty page. A
  server that ignores offset and repeats a non-empty page would
  duplicate models and keep requesting until the model bound. The
  guard now tracks seen model ids and stops when a page adds none,
  with an explicit page-count bound (100) and truncation at the
  10k-model bound for oversized pages.

Both regression tests verified failing on the unfixed code.
@wch
wch merged commit 9844ce3 into main Aug 10, 2026
4 checks passed
@wch
wch deleted the portkey-provider branch August 10, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant