dsh-verify — Agents self-test and pass. Real browsers tell the truth.

[263311487-ux]

TL;DR — We ran a 4-agent web team inside DeepSeek Harness. The team self-reviewed as "no issues found". In a real browser, the dark-mode toggle button did nothing: the .dark class was toggled by JS, but the CSS rule for .dark was never written. We turned that failure into a tiny open-source tool: dsh-verify — write a JSON spec of what a human would check, it drives a real headless Chromium, and returns an HTML report + exit code. Buggy build FAILS, fixed build PASSES, same page, one missing CSS rule.

Why this matters for the agent ecosystem

Agent evaluation today is dominated by macro metrics: task success rate, tool-call accuracy, benchmark scores. Those numbers are computed against what the agent believes it built — code the agent wrote, checked by the agent.

What almost nobody checks: does the delivered artifact work in the environment a real user actually uses?

Our reproduction of the failure chain:

1. Spec writer defines the page (counter + theme toggle)
2. Frontend dev implements both buttons
3. QA agent runs "self-tests" — there is nothing to run, page is static
4. Reviewer reads the code and the self-tests, approves
5. No one opened a browser → the toggle does nothing in the real DOM

The gap is structural, not a skill issue: agent teams verify against a shared belief, not against the user's experience. The only fix is an independent check that does not share the team's blind spots.

What dsh-verify does (now)

• JSON spec, no framework: goto → click → expect_text → capture_style → expect_style_changed
• Real Chromium via Playwright (headless by default)
• Verifies computed styles, not DOM class lists — the check that catches "class toggled but CSS never written"
• Visual regression: pixel-level screenshot baselines (expect_screenshot) with red-highlight diff images
• AI-drafted checklists: dsh-verify gen --url <url> --run — LLM drafts, deterministic browser enforces (the AI never judges)
• MCP server for Claude Code / Cursor / Copilot: verify_spec, verify_url, generate_and_verify
• GitHub Action for one-step CI acceptance with report artifact
• HTML report with screenshots, exit code 0/1, --json for CI

The check that would have caught our bug:

{ "action": "capture_style", "selector": "#page", "prop": "backgroundColor", "var": "bg_before" },
{ "action": "click", "selector": "#color-btn" },
{ "action": "expect_style_changed", "selector": "#page", "prop": "backgroundColor", "var": "bg_before" }

Live evidence (real Playwright output, committed in the repo)

Build	Agent self-review	Real browser verdict
demo/buggy — .dark rule missing	"No issues found"	❌ FAIL 10/11 — background never changes
demo/fixed — one CSS rule added	fixed	✅ PASS 11/11

Screenshots and full step-by-step reports live in evidence/ so anyone can re-run and reproduce.

Why nothing else does this

We researched the agent-tooling community before building. Existing "verification" for agent deliverables is:

• Install/plugin smoke checks — does a skill install, does a CLI exist
• Static lint / unit tests — validate code the agent wrote, not behavior the user sees
• Screenshot-only agents — look at a picture, don't assert behavior

None of them answer: "When I click this button, does the user see the change?" That's the niche dsh-verify fills — deliberately small, JSON in, browser verdict out.

Try it (2 minutes)

git clone https://github.com/263311487-ux/dsh-verify
cd dsh-verify
npm install
npx playwright install chromium
npm run demo:buggy   # FAIL — the exact bug from this post
npm run demo:fixed   # PASS

Ask

• Do you run acceptance checks on agent-delivered web artifacts today? How?
• What's the next check that should be in the box? (On the roadmap: firefox/webkit matrix, mobile viewports, natural-language spec generation without a running page)
• If you've hit the same "self-test green, browser red" wall, tell the story — we want receipts in the README.

[263311487-ux]

Agent Arena is live — we built a reproducible benchmark: same task, same prompt, same human checks, different agents, graded in a real browser (dsh-verify). No LLM judges; the browser is the judge.

Seed result: 11/12 runs passed.

Agent setup	Todo app	Pricing calc	Signup form	Pass
DeepSeek v4-flash · single shot	❌ 9/19	✅ 22/22	✅ 18/18	2/3
DeepSeek v4-flash · self-check loop	✅ 19/19	✅ 22/22	✅ 18/18	3/3
DeepSeek v4-pro · single shot	✅ 19/19	✅ 22/22	✅ 18/18	3/3
DeepSeek v4-pro · self-check loop	✅ 19/19	✅ 22/22	✅ 18/18	3/3

The one failure is the interesting one: v4-flash single-shot built a todo app that started with zero seeded todos (9/19 checks failed). The same model with a self-check loop — seeing the real-browser failures and fixing — passed 19/19.

Honest caveats: LLM output is nondeterministic (flash/single passed this task on another run), one run per cell, and the framework is open for anyone to run their own agent.

Live page + methodology: https://263311487-ux.github.io/dsh-verify/arena/ · source in arena/ — bring your own agent, the checks are the same for everyone.

[263311487-ux]

Agent Arena update — now 48 real-browser runs (4× per cell, same tasks, same human checks, temperature 0.7; verdict = real Chromium clicks/typing/computed-styles, never an LLM judge)

Agent setup	Todo app	Pricing calc	Signup form	Pass
DeepSeek v4-flash · single shot	3/4	4/4	4/4	11/12
DeepSeek v4-flash · self-check loop	4/4	3/4	4/4	11/12
DeepSeek v4-pro · single shot	3/4	3/4	4/4	10/12
DeepSeek v4-pro · self-check loop	4/4	4/4	4/4	12/12

Two counterintuitive takeaways from the data:

1. The pricier v4-pro single-shot scored below the cheaper v4-flash single-shot (10/12 vs 11/12). Paying more does not buy reliability.
2. The real-browser self-check loop lifted v4-pro to 12/12 — but v4-flash's self-check crashed once when its own verification report came back as corrupt JSON ("undefined" is not valid JSON), before the browser could even grade the page.

Every failure is reproducible and invisible to an LLM judge: the todo app opened but the seed todos never rendered in the DOM — the agent reported success while the browser saw an empty list. That is exactly the gap dsh-verify exists for.

• Live leaderboard (regenerated from committed runs): https://263311487-ux.github.io/dsh-verify/arena/
• Raw per-run results: https://github.com/263311487-ux/dsh-verify/tree/main/arena/results
• Bring your own agent: node arena/run.mjs --agent <model>/selfcheck --task all --repeat 4 → open a PR, join the table.

[263311487-ux]

Brand + traction update (Aug 19) — the project now goes by Witness (the product name; dsh-verify is the npm/GitHub package name).

What's happened since the Arena update:

• npm: 1,796 real installs in 3 days since first publish (Aug 16 → today), latest v0.8.8 with search-keyword optimization for deepseek/mcp/browser-acceptance queries.
• 3 listing PRs submitted: awesome-mcp-servers (#12303), awesome-dsh-plugin (会话归档后无法查看已归档的会话,并进行恢复. #40), awesome-deepseek-harness (pnpm install failed on mac #412) — all open, tracking merge.
• Agent Arena: 48 real-browser runs live at 263311487-ux.github.io/dsh-verify/arena/ — same tasks, same human checks, verdict = real Chromium, never an LLM.

The pitch in five words stays the same: the browser is the judge.

Try it in 2 minutes:

npx dsh-verify demo:fixed    # PASS
npx dsh-verify demo:buggy    # FAIL — one missing CSS rule the agent's self-review missed

Repo: https://github.com/263311487-ux/dsh-verify · npm: https://www.npmjs.com/package/dsh-verify

[263311487-ux]

Open entry is live — the Arena now accepts any model. Run your agent (any OpenAI-compatible endpoint) on the same 3 tasks, and your setup appears on the public leaderboard with your GitHub name:

git clone https://github.com/263311487-ux/dsh-verify && cd dsh-verify
npm install && npx playwright install chromium
export LLM_API_KEY=sk-...
node arena/run.mjs --agent "your-model/single" --task all --repeat 1 --submitter yourname

Full challenge thread: #3308