Agent Benchmarks

Manual benchmark harness and evidence bundle for comparing AI coding agents building the same full-stack project-management app against Convex, Supabase, and Firebase.

The current benchmark compares:

Agent	Backend
Codex	Convex, Supabase, Firebase
Claude Code	Convex, Supabase, Firebase

The headline writeup is runs/executive-summary.md.

Current Result

This repo now contains the completed manual benchmark evidence:

• Functional evaluation for all six primary runs.
• Source-only security review and human adjudication.
• Source-only code-quality review and human adjudication.
• Run timing, token, and cost accounting.
• Two extra timing/cost repeats per agent/backend combination.

The short version:

• Convex was strongest overall.
• Supabase was functionally strong but weaker on security.
• Firebase was weakest, mainly because both agents produced insecure Firebase rules and struggled more with the implementation.
• Claude Code produced more maintainable code.
• Codex was faster and completed more evaluated runs.

These results are for this Trello-like project-management app. They should not be read as a universal claim about every possible app shape.

Repo Map

Path	Purpose
runs/	Primary evaluated runs, final source snapshots, reports, adjudications, and accounting.
timing-runs/	Generation-only timing/cost repeats. These are not functionally evaluated.
docs/manual-benchmark-protocol.md	Manual evaluation protocol.
docs/functional-behaviors.md	Functional behavior checklist used by the human evaluator.
runs/security-human-adjudication-final.md	Final security grades and rationale.
runs/code-quality-human-adjudication-final.md	Final code-quality grades and rationale.
runs/manual-run-accounting-summary.md	Evaluated-run timing, token, and cost accounting.
timing-runs/timing-repeat-analysis.md	Timing-only repeat summary.
prompts/manual/	Backend-specific prompt packs used for manual runs.
config/manual-run-matrix.json	The six-run manual benchmark matrix.
src/	Orchestrator, preflight, reporting, evaluation, and accounting code.
docker/	Agent and evaluator Docker images.

Setup

npm install
npm run check

Docker is required for the isolated agent/evaluator flows:

npm run docker:build:agent
npm run docker:build:codex-agent
npm run docker:build:claude-agent
npm run docker:build:evaluator

Manual Benchmark Commands

List the six manual conditions:

npm run manual:list

Run backend preflights:

npm run manual:preflight:convex
npm run manual:preflight:supabase
npm run manual:preflight:firebase

Run a condition:

npm run manual:codex:convex
npm run manual:codex:supabase
npm run manual:codex:firebase
npm run manual:claude:convex
npm run manual:claude:supabase
npm run manual:claude:firebase

The exact agent-facing prompt is saved before the agent starts:

runs/<run-id>/prompt/final-prompt.md

Timing Repeats

The timing runs are one-shot generation-only runs. They skip functional evaluation, app startup checks, repairs, security review, and code-quality review.

npm run timing:list
npm run timing:codex:convex
npm run timing:codex:supabase
npm run timing:codex:firebase
npm run timing:claude:convex
npm run timing:claude:supabase
npm run timing:claude:firebase
npm run timing:accounting

Accounting

Regenerate run accounting:

npm run runs:accounting
npm run timing:accounting

The evaluated-run table includes repairs where they happened. The timing/cost section in the executive summary uses only the initial generation attempt plus the two timing repeats.

Credentials

Copy .env.example to .env and fill only the providers you need.

The harness uses scoped per-run backend environments where possible:

• Convex cloud dev deployment credentials.
• Supabase Management API project provisioning credentials.
• Firebase / Google Cloud project provisioning credentials.

Harness profiles pass only explicitly allowed environment variable names into Docker containers. Logs record variable names, not secret values.

Artifact Rules

runs/ is intentionally tracked because the benchmark evidence lives there.

Do not commit:

• .env or .env.local
• service-account JSON files
• node_modules
• bulky build outputs
• unredacted provider secrets

Generated run snapshots may contain large source trees. Keep them only when they are part of the benchmark evidence.

Verification

Useful sanity checks:

npm run check