Skip to content

v1.0 — MCP-layer browser-server benchmark

Choose a tag to compare

View all tags
@pleasedodisturb pleasedodisturb released this 28 May 18:11
· 13 commits to main since this release
v1.0
86ba1d8
This commit was signed with the committer’s verified signature.
pleasedodisturb Vitalik
SSH Key Fingerprint: NJ9RHKfdPEOlzK5yyA2vJ1OafAhKC23Bd7IGypEdNwE
Verified
Learn about vigilant mode.

First release. Public, reproducible benchmark of 7 browser-automation MCP servers scored on the same byte-for-byte loopback fixtures across 8 weighted dimensions.

The verdict

MCP Composite Cold-start Tier
Playwright 7.93 197 ms 🟢 PRIMARY
Lightpanda 6.31 13 ms ⚡ 🟢 PRIMARY (read-only specialist)
browser-use (direct) 5.87 668 ms 🟡 SECONDARY
Chrome DevTools MCP 5.60 358 ms 🟡 SECONDARY
Firecrawl 4.23 171 ms 🟡 SECONDARY
Cloakbrowser 8.33 235 ms 🔴 SANDBOX-ONLY
Obscura 3.27 158 ms ⚫ SKIP (macOS)
browser-use (agent) SKIPPED* — ⚫ SKIP

* browser-use agent mode SKIPPED for LLM_KEY_ABSENT; being filled in as v1.0.1.

One-line takeaway: Pair Playwright (interactive) with Lightpanda (read-only). Treat Cloakbrowser as a research sandbox. Skip Obscura on macOS pending the Linux A/B.

Highlights

  • 51× cold-start spread across the field (Lightpanda 13 ms vs browser-use direct 668 ms)
  • Locked rubric — scoring/rubric.md and scoring/score.py byte-for-byte unchanged from wave start, enforced at every commit boundary by bench/wave_close_check.py
  • Median of 3 retry passes per stage with N/A-aware composite (read-only MCPs marked N/A on interactive stages drop those cells from their weighted denominator instead of scoring zero)
  • Self-hosted byte-for-byte loopback snapshot fixtures — no live URL dependency, no flakiness
  • Reproducibility recipe at docs/REPRODUCIBILITY.md — anyone with the repo can clone, run, and confirm the scores
  • Two-layer goal-backward verification — every closed phase has a VERIFICATION.md with status: passed, plus an independent re-audit

Honest scope

v1.0 used job-application fixtures (Greenhouse server-rendered + Ashby React SPA) as a proxy for real, modern web pages. v1.1 will expand the fixture set to general-purpose web tasks — search, e-commerce, content extraction, multi-page navigation.

What's coming next

The follow-up wave (G-710) covers TLS fingerprint capture (JA3/JA4), bot-detection adversary set (Cloudflare, DataDome, Akamai, reCAPTCHA), cross-machine MacBook parity, Obscura Linux A/B, and general-purpose fixtures.

Full evidence

Assets 2
Loading