[Leaderboard] Pi Coding Agent — Claude Opus 4.6 — 62.6% Pass@1

[satish860]

Pi Coding Agent — Leaderboard Submission

Agent name: Pi Coding Agent
Backbone LLM: Claude Opus 4.6 (via OpenRouter)
Hints: Yes (db_description_withhint.txt)
Trials: 5 per query
Pass@1: 62.6%

Architecture

A TypeScript coding agent built on the Pi SDK that:

1. Pre-indexes each dataset via automated SQL/MongoDB introspection — schemas, sample rows, value ranges, and join key analysis. No manual input, no data leakage.
2. For each query, reads the index then writes and executes Node.js scripts that connect directly to PostgreSQL, SQLite, DuckDB, and MongoDB.
3. Iterates on errors — if the script crashes, the agent patches and reruns until it produces a final answer.

Key differences from the built-in agent

	Built-in Agent	Pi Coding Agent
Language	Python	TypeScript / Node.js
Tool interface	4 constrained tools	Full read/write/bash
DB context	Runtime exploration	Pre-built index (automated)
Script style	Iterative queries	One complete script per query

Results summary

Dataset	Pass@1
agnews	0.60
bookreview	0.87
crmarenapro	0.83
googlelocal	0.70
music_brainz_20k	0.67
stockindex	0.67
stockmarket	0.60
yelp	0.74
GITHUB_REPOS	0.40
PANCANCER_ATLAS	0.67
DEPS_DEV_V1	0.00
PATENTS	0.00
Overall	62.6%

[satish860]

@shreyashankar - Can you help with the PR.

[shreyashankar]

Hello! Sorry just got back to the US from traveling and am looking at the PRs. We will add your name to the leaderboard shortly!

[shreyashankar]

Hi @satish860 — verified with common_scaffold/validate/validate.py:

• Pass@1 = 0.5603 (stratified) — average across the 12 datasets of the per-dataset average across queries of c/n. This is the leaderboard metric.
• Pass@1 = 0.626 (micro) — total passes / total runs across all 270 trials. Equal weight per (query, run); matches the 62.6% in your PR title.

0.5603 puts you at #1 on the leaderboard, which now links back to this PR. Thanks for the submission! Closing.