feat: Vercel AI Gateway as an opt-in alternative to per-vendor API keys - #120
Closed
Rodriguespn wants to merge 4 commits into
Closed
feat: Vercel AI Gateway as an opt-in alternative to per-vendor API keys#120Rodriguespn wants to merge 4 commits into
Rodriguespn wants to merge 4 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
AI-965 Explore Vercel AI Gateway for eval model requests
Explore using Vercel AI Gateway to proxy model requests in evals instead of the current per-API-key architecture. The goal is to use a single API key and make it simpler to add models from different providers. There is also interest in whether it would integrate well with Vercel Sandboxes if eval execution moves further in that direction. |
Rodriguespn
force-pushed
the
explore/ai-gateway-vendor-provider
branch
from
July 24, 2026 13:16
440266c to
d3e0abc
Compare
Rodriguespn
added
the
run-evals-through-gateway
Rodriguespn
commented
Jul 24, 2026
Rodriguespn
force-pushed
the
explore/ai-gateway-vendor-provider
branch
from
July 24, 2026 14:11
fb81599 to
d3f033a
Compare
Rodriguespn
removed
the
run-evals-through-gateway
One new module (agents/gateway.ts) owns all gateway knowledge: endpoints, AI_GATEWAY_API_KEY, typed GatewayModelId slugs, vendor-from-slug parsing, and the RUN_THROUGH_GATEWAY env flag that flips every harness's default routing without touching experiment files. An explicit per-experiment `gateway:` option pins a path either way, so the direct per-vendor route is byte-for-byte unchanged unless opted in — and the whole feature can be removed later by deleting this module and the few guarded call sites. - claude-code: exec env swaps to ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN (ANTHROPIC_API_KEY must be empty — Claude Code prefers it when non-empty) - codex: skips `codex login`, writes a [model_providers.vercel] block with wire_api="responses" and env_key auth; bare ids get openai/-prefixed - ai-sdk: models are rebuilt as gateway models under the equivalent slug (version dashes → dots); provider readiness handles "gateway" - judge deliberately stays on the direct OpenAI key in both modes so scores remain comparable across routing paths Covered by unit tests asserting both branches of each runner plus the slug/env-flag helpers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…way label eval-refresh.yml grows a run_through_gateway dispatch input (default false) and honors a run-evals-through-gateway PR label — either routes the whole run through the AI Gateway by writing RUN_THROUGH_GATEWAY=true into the eval env. In gateway mode ANTHROPIC_API_KEY is withheld from the run, so a passing eval is itself proof the gateway carried the traffic; OPENAI_API_KEY stays for the judge. Verified end-to-end on this branch: run 30093344664 (3 matrix jobs, all green, existing experiments env-flipped to the gateway). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eb app .env.example documents AI_GATEWAY_API_KEY + RUN_THROUGH_GATEWAY; the web app strips the vendor/ prefix from gateway model ids before formatting model labels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rodriguespn
force-pushed
the
explore/ai-gateway-vendor-provider
branch
from
July 24, 2026 14:23
3024258 to
e93d186
Compare
Rodriguespn
added
the
run-evals-through-gateway
Rodriguespn
marked this pull request as ready for review
July 24, 2026 14:25
This was referenced Jul 24, 2026
Rodriguespn
removed
the
run-evals-through-gateway
Contributor
Author
|
Closing in favor of #126 |
mattrossman
added a commit
that referenced
this pull request
Jul 28, 2026
…way (#126) Adds `opencode` as a CLI agent harness plus two experiments running Kimi K3 through the [Vercel AI Gateway](https://vercel.com/ai-gateway): `opencode-kimi-k3` and `opencode-kimi-k3-no-skills`. Consolidates parts of #56, #120, and #124 per [this thread](https://supabase.slack.com/archives/C0BAGJBL49E/p1784919389986449?thread_ts=1784890581.560609&cid=C0BAGJBL49E). AI gateway is used only for OpenCode w/ the new experiments. Uses OpenCode's [native `vercel` provider](https://vercel.com/docs/ai-gateway/coding-agents/opencode), so the only credential is `AI_GATEWAY_API_KEY` which is already configured in repo secrets. The CLI is pinned to 1.18.5 and must stay >= 1.17.0. Older versions don't await the run event loop ([opencode#31389](anomalyco/opencode#31389)) and intermittently exit mid-step with no final report, which was failing about half of sandbox runs regardless of model. Failures like that used to be invisible: results JSON just showed an empty transcript with `stoppedReason: "error"`. Running with `--debug` now prints opencode's actual errors, which is how this was diagnosed: ``` OpenCode error event: {"name":"UnknownError","data":{"message":"Unexpected server error. Check server logs for details.","ref":"err_1ffd1f34"}} ``` Incidental fix: eval refresh jobs now fail when the experiment writes no result instead of green-skipping (surfaced by a missing `AI_GATEWAY_API_KEY` in the workflow env, now also wired up). To test locally, add `AI_GATEWAY_API_KEY` to `.env` (create one in the [evals Vercel project](https://vercel.com/supabase/evals) under AI Gateway → API keys), then: ```sh pnpm eval -- --experiment opencode-kimi-k3 --eval investigate-db-001-table-row-counts ``` Closes AI-967, AI-964, AI-965 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Vercel AI Gateway as an opt-in routing path for all the current three agent harnesses (
claude-code,codex,ai-sdk), living side by side with the existing per-vendor keys.One env var (
AI_GATEWAY_API_KEY) unlocks every vendor, including ones we hold no direct key for (the included Gemini experiment below proves it).How to switch between the two paths
run_through_gateway: true, or add therun-evals-through-gatewaylabel to a PR (RUN_THROUGH_GATEWAY=true pnpm eval -- …gateway: true+ slug (CLI agents) orgateway("vendor/model")(ai-sdk);gateway: falsepins direct even under the flagValidation
scenario:build-cli-002-declarative-schemabuild-cli-002-declarative-schema)claude-code-sonnet-5(existing direct experiment, env-flipped)codex-gpt-5.4-mini(existing direct experiment, env-flipped)gateway-ai-sdk-gemini-3.6-flash(suiteless, named explicitly)Check the run results and the artifacts run 30093344664
Ref AI-965