Harness sensitivity near the success ceiling: a WebMCP ON/OFF evaluator and BrowserGym replication question

[jo32]

I'm sharing a small evaluation tool and a question about transferring the experiment to BrowserGym/AgentLab.

I maintain DeepDeck, a desktop project built on DeepSeek Harness. I adapted WindTunnel's 49-task, eight-site corpus to run through a full agent harness with ordinary browser/DevTools tools available in both WebMCP ON and OFF arms. The runner uses a 600-second agent deadline and no step cap.

The result that motivated this discussion is near-ceiling scorer pass rates even without WebMCP:

Model	WebMCP OFF	WebMCP ON
GPT-5.6 Terra	49/49	49/49
DeepSeek v4.1 Flash	47/49	48/49
Hy3	48/49	49/49

Efficiency still separates the runs. On the same 46 tasks passed by all six configurations, ON reduced total tokens / agent elapsed time by 35.6% / 33.5%, 26.1% / 27.9%, and 22.7% / 22.9%, respectively.

My hypothesis is that harness/tool coverage can substantially change what we infer about model capability and the marginal benefit of structured browser tools. Once task success approaches the ceiling, interaction rounds, time and cost become more informative. Fast individual inference does not necessarily mean fast completed tasks.

Evidence boundary: this batch has one attempt per task per arm, fixed ON-then-OFF order, and provider-default reasoning with the effective level unrecorded. It includes local patch/environment differences and known scorer weaknesses; a scorer PASS is not a manual verification of every requested action. This is not yet a same-model, controlled comparison against WindTunnel's original harness. Its current code-execution configuration also achieves 49/49, so the claim is not that non-WebMCP interfaces inherently fail.

Results and methodology · Scorer audit

Tool available now

The URL runner accepts a site and task, repeats WebMCP ON/OFF with frozen model settings and fresh sessions, and writes answers, tokens/cache usage, timing, tool errors and actual WebMCP calls. It now alternates order across repetitions. It currently runs in DeepDeck; there is no BrowserGym adapter yet.

pnpm benchmark:webmcp ablate --url https://your-site.example --query "Find the latest release and summarize its changes" --n 3

Setup and usage. The site needs native WebMCP tools, or a custom built JS bundle passed via --webmcp-file. URL mode is unscored without an expected answer, and clean profiles do not reset remote backend state.

Feedback I'd value from this community

For a BrowserGym replication, I would like to compare the same model × harness configuration × WebMCP availability, holding tasks, resets, validators and budgets fixed. I'd also include an ordinary function-tool arm exposing equivalent operations, to separate task-specific tool abstraction from the WebMCP interface itself.

1. Is there an existing BrowserGym/AgentLab extension point or related work for exposing site-provided structured tools alongside the normal browser action space?
2. Which small task subset has sufficiently strong state validators to make a useful first independent replication?
3. Which trace fields would you require to distinguish fewer agent rounds from faster model responses or backend calls?

I welcome methodological criticism and independent replications. Credit to nekuda-ai/WindTunnel for the starter tasks, site recipes and WebMCP patches; these numbers are DeepDeck-local adaptations, not its leaderboard. DeepDeck is MIT-licensed; model-provider charges apply to experiments.