Coding Agent Life v1 reproduce problem

[RichardWang11]

Hi, rohitg00. I want to reproduced the results from url: https://github.com/rohitg00/agentmemory/blob/main/docs/benchmarks/2026-05-20-coding-agent-life-v1.md. But I got a different results: grep P@5 0.227 R@5 0.967 agentmemory-hybrid P@5 0.240 R@5 1.000. So I want to know how to calculate the Precision@5 for the method ?

[rohitg00]

Hi @RichardWang11 — thanks for the careful repro.

Short answer: your numbers are correct; the published scorecard is wrong.

The current formula in eval/runner/score.ts is:

const topK = ranked.slice(0, k).map((r) => r.sessionId);
const gold = new Set(q.goldSessionIds);
const hits = topK.filter((id) => gold.has(id)).length;
const precisionAtK = k > 0 ? hits / k : 0;
const recallAtK = gold.size === 0 ? 0 : hits / gold.size;

Then the per-adapter aggregate is the arithmetic mean of precisionAtK across all 15 questions.

Math ceiling on this dataset

coding-agent-life-v1 has 15 questions: 12 with one gold session, 3 with two gold sessions (you can verify via jq '[.[] | .goldSessionIds | length] | group_by(.) | map({n: .[0], count: length})' eval/data/coding-agent-life-v1/queries.json).

At k=5, the per-question P@5 cap is:

• 1 gold + hit → 1/5 = 0.200
• 2 gold + both hit → 2/5 = 0.400

So the absolute upper bound for mean P@5 on this corpus, assuming every gold session lands in top-5, is:

(12 * 0.200 + 3 * 0.400) / 15 = (2.4 + 1.2) / 15 = 0.240

That is exactly your hybrid number: P@5 = 0.240, R@5 = 1.000. It means hybrid retrieved every gold session in top-5 for every question — the best possible result on this dataset under the current formula.

Your grep number (P@5 = 0.227, R@5 = 0.967) backs out to: one of the 3 double-gold questions returned only 1 of 2 gold sessions in top-5, everything else hit. That checks out too:

(12 * 0.200 + 2 * 0.400 + 1 * 0.200) / 15 = 3.4 / 15 = 0.227
(12 * 1.000 + 2 * 1.000 + 1 * 0.500) / 15 = 14.5 / 15 = 0.967

Why the doc reports 0.578 / 0.267

Those numbers are mathematically impossible with the current formula — P@5 = hits/k caps at 0.240 on this dataset. The doc was published from a pre-fix run where the denominator was topK.length (returned-result count) instead of k (requested cutoff). That fix landed during the same PR (#562) that introduced the harness, but the scorecard was generated before the fix and never refreshed.

I'm going to update the scorecard with current numbers so this stops misleading future repros — tracking that.

TL;DR

• Your output is correct
• Formula is P@K = hits / K, mean across all questions
• Doc's 0.578 / 0.267 are from a pre-fix run and need to be replaced
• Hybrid still beats grep on this corpus, but the lift is in recall (1.000 vs 0.967) and per-type P@K, not the headline aggregate P@5 — because the ceiling at K=5 on a 1-2-gold corpus is too low to differentiate

Thanks for catching this. Sorry the doc misled you.