Skip to content

feat: add Cursor CLI agent harness and experiments - #166

Open
vivek100 wants to merge 1 commit into
supabase:mainfrom
vivek100:feat/cursor-agent-support
Open

feat: add Cursor CLI agent harness and experiments#166
vivek100 wants to merge 1 commit into
supabase:mainfrom
vivek100:feat/cursor-agent-support

Conversation

@vivek100

@vivek100 vivek100 commented Aug 4, 2026

Copy link
Copy Markdown

Summary

  • Add a Cursor CLI harness (packages/core/src/agents/cursor/) with pinned install, headless stream-json runner, transcript parser, and cursorAgent() factory wired into the registry.
  • Add four experiments mirroring Claude Code: cursor-composer-2.5, cursor-composer-2.5-no-skills, cursor-grok-4.5-high, cursor-grok-4.5-high-no-skills (Supabase MCP on; skills vs none).
  • Document CURSOR_API_KEY in .env.example / README and label Cursor in the results UI.

Fixes #167

How to run

# .env needs CURSOR_API_KEY; OPENAI_API_KEY for rubric/judge checks
pnpm eval -- --experiment cursor-composer-2.5 --eval resolve-dataapi-001-empty-results

Model ids (from live cursor-agent --list-models): composer-2.5, cursor-grok-4.5-high.

Verification

  • Unit: pnpm exec vitest run src/agents/cursor/ (core package)
  • Typecheck / format: pnpm typecheck, pnpm format:check
  • Discovery: pnpm eval:dry for all four Cursor experiments
  • Tools-mode smoke: cursor-composer-2.5 × resolve-dataapi-001-empty-results — agent completed with Supabase MCP (list_tables, execute_sql, apply_migration, …) and skills loaded. Scorer judge step needs OPENAI_API_KEY (eval uses judge() in addition to deterministic checks).

Test plan

  • pnpm --filter @supabase-evals/core test (includes Cursor runner/parser tests)
  • pnpm typecheck
  • Dry-run one Cursor experiment
  • Optional: add run-evals label if maintainers want full benchmark CI
  • Optional: targeted eval-refresh for the new Cursor experiments

Enable Composer 2.5 and Grok 4.5 evals via cursor-agent with stream-json
parsing, MCP/skills support, and benchmark/no-skills experiment variants.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vivek100
vivek100 requested a review from a team August 4, 2026 20:39
@vercel

vercel Bot commented Aug 4, 2026

Copy link
Copy Markdown

@vivek100 is attempting to deploy a commit to the Supabase Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Cursor CLI agent support (Composer 2.5 / Grok 4.5)

1 participant