Skip to content

docs: the full-300 Online-Mind2Web result — 31.0%#193

Merged
NotASithLord merged 1 commit into
mainfrom
docs/full-300-result
Jul 8, 2026
Merged

docs: the full-300 Online-Mind2Web result — 31.0%#193
NotASithLord merged 1 commit into
mainfrom
docs/full-300-result

Conversation

@jonybur

@jonybur jonybur commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Completes the web-thread report (#188) with the number it was waiting for: the full 300-task Online-Mind2Web run — 93/300 = 31.0% (Opus 4.8, WebJudge/o4-mini, July 8).

Honest framing baked in: the 46.7% calibration-slice reading did not fully generalize (as flagged); 31.0% sits below browser-use's verified ~40%; the value is the failure taxonomy at n=300 — a third of failures are prompt-addressable finish-and-report misses and a quarter are step-cap deaths with a known fix, i.e. a mapped path to 40%+. Positioning stays "first extension-native number on the benchmark," not "we beat the hosted agents."

🤖 Generated with Claude Code

Signed-off-by: Jonathan Bursztyn <jobur93@gmail.com>
@jonybur jonybur requested a review from NotASithLord as a code owner July 8, 2026 11:38

@NotASithLord NotASithLord left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review (docs-only change, single file, +39/-1): the numbers check out — 93/300 = 31.0%, and the failure-bucket table sums to exactly 207 (58+53+14+14+68), matching 300−93. The 22% step-cap figure (66/300) is also exact. Framing is appropriately hedged (calibration-slice number flagged as non-generalizing, positioned below browser-use's verified ~40%, no overclaiming vs. hosted agents).

One minor, non-blocking nit: the slice table's ranges ("1–99", "100–199", "200–299") sum to 299 tasks, not 300 (99+100+100), even though the derived correct-counts (30+40+23=93) are internally consistent with the headline number. Likely just an off-by-one in the label (probably meant 0–99/1–100 for the first slice) — worth a follow-up fixup, not worth blocking this PR over since it doesn't affect the reported totals.

CI is green across all 12 checks (bun test, typecheck/lint/boundary, in-browser + e2e, dweb harnesses, packaging). No production code touched. Approving.


Generated by Claude Code

@NotASithLord NotASithLord merged commit 02fa8b1 into main Jul 8, 2026
12 checks passed
@NotASithLord NotASithLord deleted the docs/full-300-result branch July 8, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants