feat: add OpenCode harness running Kimi K3 through the Vercel AI Gateway

[mattrossman]

Adds opencode as a CLI agent harness plus two experiments running Kimi K3 through the Vercel AI Gateway: opencode-kimi-k3 and opencode-kimi-k3-no-skills. Consolidates parts of #56, #120, and #124 per this thread. AI gateway is used only for OpenCode w/ the new experiments. Uses OpenCode's native vercel provider, so the only credential is AI_GATEWAY_API_KEY which is already configured in repo secrets.

The CLI is pinned to 1.18.5 and must stay >= 1.17.0. Older versions don't await the run event loop (opencode#31389) and intermittently exit mid-step with no final report, which was failing about half of sandbox runs regardless of model.

Failures like that used to be invisible: results JSON just showed an empty transcript with stoppedReason: "error". Running with --debug now prints opencode's actual errors, which is how this was diagnosed:

OpenCode error event: {"name":"UnknownError","data":{"message":"Unexpected server error. Check server logs for details.","ref":"err_1ffd1f34"}}

Incidental fix: eval refresh jobs now fail when the experiment writes no result instead of green-skipping (surfaced by a missing AI_GATEWAY_API_KEY in the workflow env, now also wired up).

To test locally, add AI_GATEWAY_API_KEY to .env (create one in the evals Vercel project under AI Gateway → API keys), then:

pnpm eval -- --experiment opencode-kimi-k3 --eval investigate-db-001-table-row-counts

Closes AI-967, AI-964, AI-965

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evals	Ready	Preview, Comment	Jul 28, 2026 2:12am

[linear-code]

AI-967 Add Kimi K3 to public evals benchmark

Add the Kimi K3 model to the public evals benchmark. The intended path is via Vercel AI Gateway and through OpenCode.

AI-964 AI spike for OpenCode support in evals

Prototype OpenCode support in evals.

Context from the discussion: this seemed worth exploring ahead of launch, especially since OpenCode is open source. One possible tradeoff mentioned was substituting GPT-5.4-mini for a different harness if needed.

AI-965 Explore Vercel AI Gateway for eval model requests

Explore using Vercel AI Gateway to proxy model requests in evals instead of the current per-API-key architecture. The goal is to use a single API key and make it simpler to add models from different providers. There is also interest in whether it would integrate well with Vercel Sandboxes if eval execution moves further in that direction.

Review in Linear

[Rodriguespn]

I let Opus do intensive search through Opencode v1.18.5 and docs and it flagged me some Opencode specific behaviors that i left as comments. In general the implementation is solid but there are still some caveats worth taking a second look at

[Rodriguespn]

LGMT. Thx for pushing for this