WebMCP Lab

Lighthouse for WebMCP: test whether an agent can actually use your site.

Your site registers WebMCP tools. An agent calls them and says the task is done. WebMCP Lab checks whether it really was, by asking the site itself instead of believing the agent.

Live: https://webmcp-lab.vercel.app (Chrome 149+ with WebMCP enabled, or the ChatGPT in-app browser). Nothing to install, and a reference target site is deployed alongside it so you can run a real evaluation immediately.

The problem

The way WebMCP breaks is quiet. A tool declares pillar as a closed enum. Months later an editor renames one category. Nothing throws. search_posts still returns HTTP 200 with an empty list, the agent reports success, and the task silently stops working. Nobody finds out until a user does.

DOM automation cannot catch this, because from the outside a successful empty result and a successful full result look identical.

WebMCP can, and this is the whole argument for it. Tools publish their inputSchema, so the Lab compares a task's literal arguments to the live schema before executing anything:

Zero tool calls. The site was never touched. The Lab read the published contract, found pillar: "neurodivergent" is no longer accepted, named the tool to blame, and suggested the value that replaced it. A DOM-driving agent would have run the entire task and still reported success.

Why not just an OpenAPI spec

Fair question, and the answer is the reason this is a WebMCP project rather than an API-contract tester.

An OpenAPI document is published: generated at build time, committed, and free to drift from the code actually running. listTools() is not published, it is observed. It returns what this page, in this browser, in this session, after its JavaScript ran, actually registered. Tools can appear conditionally, so the surface is a function of state, and no static document can express that.

More to the point: websites have never had a machine-readable contract for what a visitor can do on them. Backends have had OpenAPI for a decade; the page has had nothing. WebMCP is the first time that contract exists, which is why a tool for testing it could not have been built before.

The assertions are WebMCP too. The Lab checks the outcome by calling the site's own read-only tools and asking what the state is now. Without that it would have to scrape the DOM, which is the unreliable thing being replaced.

What it does

1. Pick a target site and a task fixture with explicit success criteria.
2. Read the tools the site publishes, live.
3. Check the task's arguments against those schemas. Stop here if they break.
4. Run the task through the site's WebMCP tools.
5. Probe the site's own state and check it against the fixture's assertions.
6. Attribute the failure to one tool and one named failure class.
7. Rerun after a change to prove the fix held, or catch the regression.

Assertions probe the environment. They never read the agent's transcript.

The Lab is itself agent-operable

A tool that measures agent-readiness would be a poor argument for WebMCP if only a human could drive it. The Lab registers five tools of its own:

Tool	Writes	What it does
list_targets	no	The sites and task fixtures available
run_eval	yes	Run one task against one target, return a verdict
inspect_run	no	Why it failed: contract, calls, assertions, blame
rerun_eval	yes	Run again and diff, to prove a fix or catch a regression
export_report	yes	Markdown report for a pull request or issue

An agent calling run_eval moves the same controls a person would, and the verdict says which tool started the run, so a human watching sees the agent work:

tools/agent-drive.mjs runs the entire narrative through navigator.modelContextTesting with zero clicks, and fails the build unless every step behaves. Its recorded transcript is in evals/runs/agent-drive/.

Run it

Needs Node 20+ and Chrome 149 or newer with WebMCP enabled.

git clone https://github.com/Citability/webmcp-lab
cd webmcp-lab
npm install --ignore-scripts
npx playwright install chromium     # Chrome for Testing 149, used by the checks
npm run serve                       # then open http://localhost:8080

In your own Chrome, launch with --enable-features=WebMCP,WebMCPTesting (or enable chrome://flags/#enable-webmcp-testing). Without it the bundled polyfill takes over and everything still runs, which the masthead tells you.

Verify every claim on this page

npm run verify

That runs four checks in order, each of which exits non-zero on failure:

Command	Proves
npm test	The contract checker, failure attribution and lint rules, in isolation (18 checks)
npm run probe	Chrome's WebMCP API is real and native, with no polyfill loaded
npm run capture	The human path: stable target passes, renamed target fails
npm run agent	The agent path: six WebMCP tool calls, no clicks

capture and agent write durable artifacts to evals/runs/ and screenshots to docs/screenshots/. Every number and screenshot in this repository comes from those commands. None is hand-written.

And against the deployed site, in a fresh browser profile:

npm run health

That is the same agent journey pointed at https://webmcp-lab.vercel.app. It exists because a green local suite proves less than it looks like it does. Two breaks shipped straight through npm run verify: a cleanUrls rewrite that 404'd the target's script, and a race where the Lab asked for the tool list before the target had finished registering. Both were invisible on localhost, where every file arrives instantly. The second one is why discoverTools polls to a deadline instead of asking once, and why a run records how long registration took: a site that takes three seconds to become agent-usable is genuinely worse than one that takes forty milliseconds, and the Lab should be able to say so.

Grade any site without asking it

Running a task needs the site's cooperation, for a good reason: a page cannot read another origin's tools, and it should not be able to. But grading a tool surface needs nothing at all.

npm run lint https://webmcp-lab.vercel.app/target/index.html

  3 tools published, registered in 18ms

  add_to_reading_list
    clean
  get_reading_list
    clean
  search_posts
    [warn] fragile-enum: Parameter "pillar" is a closed enum (ai-building,
      neurodivergent, automation, philosophy). Renaming a value here breaks
      callers without throwing. Worth a fixture.

It opens the URL in a WebMCP browser and reads what the page actually registered. No script tag, no permission, no change to the site. Undescribed tools are errors. Closed enums are warnings, because an enum is not a mistake, it is the place your contract will break first. That is where to point a fixture, and in this case it is exactly the parameter the demo goes on to break.

One rule is deliberately missing. We wrote a readOnlyHint check, and it accused all three of our own tools, which do declare it. Chrome 149's listTools() returns { name, description, inputSchema } and drops annotations entirely, so a compliant site is indistinguishable from a silent one. The rule was removed rather than kept as a plausible-looking accusation, and the gap is reported as unobservable instead: an agent reading that same surface also cannot tell whether calling a tool twice is safe.

Read against sites we did not write

The example above grades our own target, which proves the command runs and nothing else. Here it is against four WebMCP URLs belonging to other people, run 2026-08-25, cold, with no contact with any of their authors:

url	result
nearform.github.io/vector-search-web/	1 tool, search_nearform_knowledge, registered in 31-115ms over four runs, graded clean
mcp-b.ai/	no tools published on arrival
web-mcp.net/playground	no tools published on arrival
webmcp.dev/	no tools published on arrival

The second column is the point. Every one of these sites is about WebMCP, and the browser hands navigator.modelContextTesting to all four, so the API being present says nothing about whether a site published anything into it. Exactly one of them offers a tool an agent could call on arrival. Separating those two states, from outside, without asking, is the whole job of Tier 0.

Try it:

npm run lint https://nearform.github.io/vector-search-web/

A site that registers its tools behind a click reads the same as a site with no tools at all, and the output says so rather than pronouncing the site dead.

Instrument your own site

The Lab evaluates a site cross-origin, and registerTool is same-origin only, so the target opts in with one script tag:

<script src="bridge.js" data-lab-origin="https://your-lab-origin"></script>

bridge.js reads whatever WebMCP layer the page has, native or polyfill, and answers the Lab over an origin-allowlisted postMessage. It registers nothing and changes nothing. Your own origin is always trusted, so a same-origin embed needs no attribute at all.

Layout

Path	What it is
lab.js	The engine. Contract checking, execution, assertions, attribution. Knows nothing about the DOM.
ui.js, index.html, lab.css	The human surface
lab-tools.js	The agent surface. Five WebMCP tools over the same controller.
bridge.js	The one script a target embeds
webmcp-polyfill.js	Enough of the API to run in browsers that have not shipped it
target/	Field Notes, the reference site, in three builds
fixtures/	Task definitions with explicit success criteria
tools/lint.mjs	Tier 0. Grades any URL's tool surface, with no cooperation from it.
tools/	The verification commands
evals/runs/	Recorded run artifacts
docs/	Goal, autonomy manifest, session handoff, submission plan

Provenance

This repository was created on 2026-08-25, the first day of the WebMCP Challenge submission period. Every file in it was written after that date. The full history is git log, starting at f497ef3.

One piece of prior art informed the design and is disclosed here. In May 2026 I wrote section_webmcp_agent_readiness.py for a separate private project, a static rubric that scores a page for agent-readiness by reading its markup. That work is the origin of the manifest-lint idea. None of its code is in this repository, and it does no live evaluation: it never executes a tool, never checks a schema against a task, and never inspects the resulting state. The evaluation engine here is new.

License

MIT. See LICENSE.