Introducing Supabase Evals

[mattrossman]

Supabase Evals is our open-source benchmark and framework for measuring how well AI coding agents build with Supabase. It runs agents including Claude Code, Codex, and OpenCode against real Supabase tasks and scores what they do. The results are published to a web app, and the repo is open. Available now.

What changed

Agents are now a primary way people build with Supabase, through our MCP server, CLI, agent skills, and docs. We wanted a clear, repeatable way to measure how well they do it: where agents already succeed, where they need a hand, and whether the changes we ship actually move the numbers. Supabase Evals is how we do that, and now it's open.

What's included

• A benchmark that runs coding agents against real Supabase tasks, such as building a schema, debugging a failed Edge Function, or fixing a broken RLS policy
• Every scenario runs against a real Supabase environment, so agents call the actual MCP server and CLI
• Scoring that combines deterministic checks with LLM-as-judge for cases that need semantic judgment
• A web app to browse and group results by agent, product, stage, or eval
• Results across current models, including Opus 5, GPT 5.6 Sol, and Kimi K3

Getting started

Read the announcement: https://supabase.com/blog/introducing-supabase-evals
Browse the benchmark results: https://supabase.com/evals
Explore the repo: https://github.com/supabase/evals

Feedback welcome

We built this to understand where agents do well with Supabase and where they don't. If you have thoughts on what we test or how we score it, reply here.