Add LabRat (Claude Opus 5 + Cartographer + informed packs) — 0.8102 stratified Pass@1

[esagduyu]

Leaderboard submission — LabRat (Claude Opus 5 + Cartographer + informed packs)

Stratified Pass@1	0.810150 (mean of per-dataset per-query pass rates)
Model	claude-opus-5, high reasoning effort
Trials	5 per query, all 12 official datasets (54 queries)
Hints	✓ (db_description_withhint.txt, appended to base description)
Tuned prompt	✓ — please mark this column. Four benchmark-informed prompt rule packs are active; the complete opening prompt for every query, packs included, is published in labrat_opus5_opening_prompts.json in this PR.
Date	2026-08-03 (run window 09:55 → 21:08 PDT)
Prior LabRat entries	keep both existing rows; this is an additional row, our first Opus submission

Full disclosure — please read before scoring

1. Two trials are missing (268 scored of 270). agnews:4 trials 1 and 4 ended infra:timeout (1800 s per-trial wall) and every completion re-run wedged our CLI harness within seconds — three attempts, all hung, diagnosed as a subprocess-cleanup deadlock on our side, not a semantic failure. The three completed agnews:4 trials are all passes. Scored both ways: 0.8102 excluding the two trials (our formula and this JSON), 0.8018 if you prefer to count them as failures. Maintainers' choice; both numbers are honest.

2. The prompt is benchmark-informed (hence Tuned ✓), with a contamination gate. The four packs encode form rules (answer shape, value placement, analytical conventions, per-dataset schema quirks) derived from our own failure analyses. Two automated gates enforce that no ground-truth content leaks into prompt text: (a) a blocking CI test that extracts every value-shaped and alphabetic token from pack text and fails on any appearance in any ground_truth* / validate.py under this repo; (b) an independent audit in the reverse direction (all 1,069 GT/validator values searched against the emitted prompt text, with a planted-token self-test proving the gate fires). Both pass on the submitted prompt text, which you can verify against labrat_opus5_opening_prompts.json.

3. Sandbox + taint audit. Same architecture as our accepted PR #72: the agent runs claude --print with an MCP-only tool allowlist (--allowedTools mcp__labrat, all native tools disallowed), isolated cwd (this checkout off-path), hermetic per-trial HOME, and a contamination backstop that withdraws any trial whose output names answer-key/external-dataset markers. The per-trial taint audit over all 270 trace files returned all clean — zero withdrawn trials (unlike our internal packs-off Opus run, which had 2 agnews withdrawals and was therefore never submitted).

4. Reproducibility. LabRat commit 7ca04ed38538 (branch feat/informed-packs-v2, github.com/esagduyu/labrat); every shard's config records this commit with a clean tree via our provenance capture. Ground truth: validated against origin/main as of 9a55e3427 — including the updated crmarenapro/query9 validator, under which all five of our crmarenapro:9 trials still pass. Cartographer pre-pass is deterministic and GT-firewalled (reads DB metadata + sampled rows only; no LLM authoring). A full trace bundle (268 per-trial tool-call traces, hash manifest, secret-scan with three reviewed benign findings — npm scoped-package names colliding with a URL-credential regex) is available on request.

Method summary

Opus 5 drives our open-source LabRat data-agent tool suite over MCP (schema catalog, profiling, SQL with error-category repair hints, join verification, local-embedding classification for semantic tasks, a server-side result ledger, and Scent — deterministic pre-computed schema notes retrieved as tool output). Prompt-side: nine generic process levers plus the four published packs. An answer-shape gate makes one presentation-only corrective pass with all tools disabled.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TXDSfkcxNjELp48YQSsndd

[Ruiying-Ma]

Hi @esagduyu, thanks for the submission. Could you please attach the 268-trace bundle so we can check it before putting a score on the board?

[esagduyu]

@Ruiying-Ma Trace bundle attached: leaderboard_submissions/labrat_opus5_trace_bundle.tar.gz (3.2 MB, sha256 f6398b909dbdc680637131a6890e1f24bafe6d343807c814a760e8ced9b75814).

Contents (trace_bundle/ after extraction):

• traces/ — one tool-call trace per scored trial (268 files, {tool, input, ok, output, latency_ms} per line, full un-truncated tool outputs)
• manifest.json — per-file sha256 + counts, so every trace can be integrity-checked
• trials.jsonl / submission.json — the 268 scored rows (identical content to the results JSON already in this PR)
• taint.json — the per-trial audit verdicts (all clean)
• config.json — full run flags + git provenance (labrat commit 7ca04ed38538, clean tree)

Notes for review: the two agnews:4 infra-timeout trials disclosed in the PR body have no trace here by construction (no completed semantic attempt). The bundler's secret scan raised three findings, all reviewed benign — they're npm scoped-package names (@scope/pkg) in deps_dev_v1 tool output colliding with a URL-credential regex; no credential material. Happy to provide anything else you need.

[Ruiying-Ma]

Hi @esagduyu! Thanks for the submission and the detailed traces. We have added your results to the leaderboard.