Skip to content

feat (provider/neon): add Neon AI Gateway provider#15997

Closed
andrelandgraf wants to merge 5 commits into
vercel:mainfrom
andrelandgraf:add-neon-ai-gateway-provider
Closed

feat (provider/neon): add Neon AI Gateway provider#15997
andrelandgraf wants to merge 5 commits into
vercel:mainfrom
andrelandgraf:add-neon-ai-gateway-provider

Conversation

@andrelandgraf

@andrelandgraf andrelandgraf commented Jun 11, 2026

Copy link
Copy Markdown

Summary

Adds @ai-sdk/neon, a provider for the branch-scoped Neon AI Gateway.

Each Neon project branch gets its own gateway host, and a platform token authorizes requests for that branch. The provider routes each model to the best gateway endpoint based on its id, so a single neon('databricks-...') call (same base URL + token) reaches the full databricks-* catalog with the best available feature support.

Routing

Model family Endpoint Why
Anthropic (databricks-claude-*) native Messages API streaming structured output + native reasoning
OpenAI (databricks-gpt-*, *-codex) native Responses API Codex (only served natively), native reasoning, image-gen tool
Everything else (Gemini, Llama, Qwen, gpt-oss, ...) unified MLflow (OpenAI-compatible) endpoint broad coverage; Gemini is here because its native gateway endpoint can't stream

Routing reuses the official @ai-sdk/anthropic and @ai-sdk/openai model classes (no reimplementation), with thin Neon subclasses that handle gateway quirks:

  • OpenAI: sets forceReasoning for gpt-5* so reasoning handling works despite the required databricks- prefix.
  • Anthropic: disables eager_input_streaming (rejected by the gateway) so streaming tool calls work.

Configuration

NEON_AI_GATEWAY_BASE_URL (branch host root) + NEON_AI_GATEWAY_TOKEN (platform token), both from the Console's AI Gateway tab; or pass baseURL/apiKey to createNeon.

import { neon } from '@ai-sdk/neon';
import { generateText } from 'ai';

const { text } = await generateText({
  model: neon('databricks-gpt-5-3-codex'), // or claude / gemini / llama / ...
  prompt: 'Summarize Postgres for me.',
});

Verified live (Anthropic, OpenAI, Codex, Google, Meta)

generateText, streamText, system prompts, multi-turn, tool calling (single + multi-step + streaming), generateObject, streamObject, reasoning, and image (vision) input.

For MLflow-routed models, the provider detects the family and drops parameters a backend rejects (e.g. penalties/seed for Llama, reasoningEffort for Gemini) with an AI SDK warning instead of a hard error.

Known gateway limitations (not adapter bugs; documented)

  • Image generation (generateImage) and embeddings (embed/embedMany) have no gateway endpoint → NoSuchModelError. (Image gen exists only as the OpenAI Responses image_generation tool and is constrained by the gateway's response-size cap.)
  • gpt-oss-* returns a non-standard response shape on the unified endpoint.
  • databricks-gpt-5-mini + multi-step tools intermittently returns gateway 502s (upstream); Codex and other models are unaffected.

Test plan

  • pnpm --filter @ai-sdk/neon test (node + edge), type-check, build
  • oxlint + oxfmt clean
  • Live end-to-end capability matrix across Anthropic, OpenAI, Codex, Google, and Meta models
  • Verified as an external consumer (linked package) in a separate project

Note: per contributing/add-new-provider.md, first-party @ai-sdk/<provider> packages may warrant an issue to discuss first — happy to open one / adjust as maintainers prefer.

Add `@ai-sdk/neon`, a provider for the branch-scoped Neon AI Gateway. It
targets the gateway's unified, OpenAI-compatible MLflow endpoint, exposing the
full `databricks-*` foundation model catalog (Anthropic, OpenAI, Google, Meta,
and more) through a single language model.

Configure with `NEON_AI_GATEWAY_BASE_URL` (the branch host) and
`NEON_AI_GATEWAY_TOKEN` (the platform token), or pass `baseURL`/`apiKey` to
`createNeon`. Includes provider + chat-model unit tests, an example, and docs.
Verified the adapter against the live gateway across Anthropic, OpenAI, Google,
and Meta models for text, streaming, tools, structured output, and vision.

- Strip the JSON Schema `$schema` marker from tool parameters and structured
  output schemas via a request transform. Gemini's backend rejects unknown
  schema fields, which previously broke tool calling and object generation for
  Gemini; other backends ignore its absence.
- Enable `supportsStructuredOutputs` so `generateObject` uses
  `response_format: json_schema` instead of the `json_object` fallback (which
  required the prompt to contain the word "json" and failed on several backends).
- Document supported capabilities, per-backend sampling-parameter constraints,
  the streaming-structured-output limitation for Databricks-hosted Anthropic
  models, and that image generation / embeddings are not offered by the gateway.
The Neon AI Gateway proxies to heterogeneous backends that each accept a
different subset of OpenAI-style parameters; previously, sending a parameter a
backend didn't support failed the whole request with a hard 400.

Add prefix-based model-family capability detection (getNeonModelCapabilities)
and drop parameters a family is known to reject, surfacing an AI SDK warning
instead of erroring — mirroring how @ai-sdk/openai handles reasoning models:

- Anthropic: drops frequencyPenalty/presencePenalty/seed, drops topP when
  temperature is also set, drops reasoningEffort.
- OpenAI gpt-5* reasoning: drops penalties and stop; drops temperature/topP for
  gpt-5 / gpt-5-mini / gpt-5-nano (gpt-5.1+ keep them).
- Meta (Llama): drops frequencyPenalty/presencePenalty/seed.
- Google (Gemini): drops reasoningEffort.
- Unknown/untested models are passed through unchanged.

Verified live across Anthropic, OpenAI, Google, and Meta models: the full
capability matrix (text, system, sampling, penalties, stop, seed, streaming,
tools, generateObject, reasoningEffort, vision) now passes, with dropped
parameters reported as warnings.
…llback

Some Neon AI Gateway models (e.g. Codex) are only served via provider-native
endpoints, so route by model family instead of always using the unified
endpoint:

- Anthropic (claude) -> native Messages API (unlocks streaming structured
  output; disables `eager_input_streaming`, which the gateway rejects).
- OpenAI (gpt-*, codex) -> native Responses API (unlocks Codex, which is not
  available on the chat endpoint; sets `forceReasoning` for gpt-5 models so the
  Responses model's reasoning handling works despite the `databricks-` prefix).
- Everything else (Gemini, Llama, Qwen, gpt-oss, ...) -> unified MLflow
  endpoint. Gemini stays here because its native gateway endpoint does not
  support streaming.

Routing is transparent (same base URL + token). Reuses the official
@ai-sdk/anthropic and @ai-sdk/openai model classes rather than reimplementing
them. MLflow-routed models keep the prefix-aware capability handling.

Verified live: text, system, sampling, penalties, stop, seed, streaming, tools
(single + multi-step + streaming), generateObject, streamObject, reasoning, and
vision pass across Anthropic, OpenAI, Codex, Google, and Meta models.
Image generation works today via the OpenAI Responses `image_generation` tool
when using `streamText` (the image returns as a `tool-result` part). Streaming
avoids the gateway's non-streaming response-size cap and read timeout, so only
the single-shot `generateText` path is constrained. Clarify the docs accordingly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant