docs: the full-300 Online-Mind2Web result — 31.0%#193
Conversation
Signed-off-by: Jonathan Bursztyn <jobur93@gmail.com>
NotASithLord
left a comment
There was a problem hiding this comment.
Adversarial review (docs-only change, single file, +39/-1): the numbers check out — 93/300 = 31.0%, and the failure-bucket table sums to exactly 207 (58+53+14+14+68), matching 300−93. The 22% step-cap figure (66/300) is also exact. Framing is appropriately hedged (calibration-slice number flagged as non-generalizing, positioned below browser-use's verified ~40%, no overclaiming vs. hosted agents).
One minor, non-blocking nit: the slice table's ranges ("1–99", "100–199", "200–299") sum to 299 tasks, not 300 (99+100+100), even though the derived correct-counts (30+40+23=93) are internally consistent with the headline number. Likely just an off-by-one in the label (probably meant 0–99/1–100 for the first slice) — worth a follow-up fixup, not worth blocking this PR over since it doesn't affect the reported totals.
CI is green across all 12 checks (bun test, typecheck/lint/boundary, in-browser + e2e, dweb harnesses, packaging). No production code touched. Approving.
Generated by Claude Code
Completes the web-thread report (#188) with the number it was waiting for: the full 300-task Online-Mind2Web run — 93/300 = 31.0% (Opus 4.8, WebJudge/o4-mini, July 8).
Honest framing baked in: the 46.7% calibration-slice reading did not fully generalize (as flagged); 31.0% sits below browser-use's verified ~40%; the value is the failure taxonomy at n=300 — a third of failures are prompt-addressable finish-and-report misses and a quarter are step-cap deaths with a known fix, i.e. a mapped path to 40%+. Positioning stays "first extension-native number on the benchmark," not "we beat the hosted agents."
🤖 Generated with Claude Code