Design Harness

Evidence for AI-made interfaces.

Design Harness is an open-source, model-agnostic UI/UX QA loop for AI coding agents. Point it at a local URL and it captures desktop/mobile screenshots, runs conservative source-backed checks, and writes an agent-friendly report you can hand back to Codex, Claude Code, Gemini CLI, or a human reviewer.

local URL -> screenshots -> audit.json -> report.md -> agent loop

It is not a black-box design judge, a hosted LLM product, or a visual regression replacement. The goal is simpler and more useful: make UI feedback repeatable enough that humans and agents can improve the same screen from the same evidence.

Why

AI coding agents can produce interfaces that look plausible at first glance while still having ordinary, expensive problems:

• mobile overflow,
• clipped text,
• weak contrast,
• unlabeled controls,
• color-only state changes,
• hidden or ambiguous errors,
• unreadable dense layouts,
• CTA and state hierarchy that "feels off" but has no review artifact.

Design Harness turns those observations into structured output. Deterministic issues stay deterministic. Heuristic risks stay labeled as risks. Subjective or emerging design judgments stay as review prompts instead of being dressed up as objective truth.

Quickstart

Playwright needs a local Chromium binary. Install it once:

npx playwright install chromium

Then run your app locally and audit it — no checkout needed:

npx @design-harness/cli@latest audit --url http://localhost:3000 --out runs/demo

Expected output:

runs/demo/
  metadata.json
  audit.json
  report.md
  report-manifest.json
  screenshots/
    <viewport-name>.png

Partial audits still write artifacts but exit with code 2 by default. Use --allow-partial when a debugging workflow should treat partial artifacts as success.

The npm release is audit-CLI focused. The PR comment renderer, scenario audit runner, and MCP adapter are checkout-local — they run only from a clone of this repository, not from the npm packages — until they are promoted to shipped package APIs.

From a checkout (for contributors)

For contributing, or for the checkout-local recipes:

pnpm install
pnpm build
pnpm design-harness -- audit --url http://localhost:3000 --out runs/demo

Use pnpm playwright:install if Chromium is missing.

Agent Loop

The core workflow is meant to be boring in the best way:

1. Run your app locally.
2. Audit the local URL.
3. Give runs/<name>/report.md and the screenshots to your coding agent.
4. Ask it to fix the highest-confidence findings first.
5. Run the harness again.

Example follow-up prompt:

Use this Design Harness report as the source of truth.
Fix deterministic failures first, then medium-confidence risks.
Do not chase subjective polish unless the report marks it as needs-review.
After editing, rerun the same audit command and summarize what changed.

This keeps the loop grounded. The agent is not guessing from "make it better"; it is responding to screenshots, evidence references, criteria, and report copy that separate measurement from taste.

What Gets Checked

Checks are conservative and source-backed:

• Render and layout: render failure or blank render, horizontal overflow, likely text clipping, DOM-computed contrast risk.
• Semantic accessibility: missing accessible names, form labels, image alternatives, heading issues, landmarks, missing page language declaration.
• Responsive readability: wide content, sticky obstruction, excessive line length, target size.
• Interaction state: error association, color-only states, disabled controls, live status, dialogs, custom controls, moving content.
• Reference-derived hierarchy heuristics: repeated visual weight, saturated color noise, checklist state visibility.

Every finding includes severity, confidence, viewport, category, evidence references, and a recommendation. v0.3 findings may also include criterionId, sourceRefs, determinism, resultKind, runtime type, observed evidence, and expected behavior.

A real finding from an audit of the semantic-a11y-bad fixture (abridged):

{
  "id": "finding-desktop-missing-form-label-1",
  "category": "accessibility",
  "severity": "medium",
  "confidence": "medium",
  "viewport": "desktop",
  "selector": "#query",
  "problem": "Form control #query may not have a programmatic label.",
  "recommendation": "Associate the control with a visible label, aria-label, or aria-labelledby.",
  "criterionId": "a11y.form-label.present",
  "sourceRefs": ["wcag-2-2", "polaris-accessibility"],
  "determinism": "deterministic",
  "resultKind": "risk",
  "evidenceRefs": ["screenshot-desktop", "measurement-desktop"]
}

A complete failing run — the full report.md, audit.json, and screenshots for a fixture with 24 findings and a blocked advisory score — is committed at examples/reports/semantic-a11y-bad.

See Criteria And Checks, Output Contract, and Fixture Catalog.

How Findings Speak

Design Harness tries hard not to overclaim. Findings are graded on two axes — determinism (deterministic / heuristic / subjective) and resultKind (failure / risk / needs-review) — plus a separate low/medium/high confidence field. Reports group them into three reader-facing tiers:

Tier	Meaning	Example
Deterministic failures and risks	Measured evidence supports a concrete failure or risk.	Horizontal overflow at 390px.
Heuristic risks	The signal is probably useful, but context still matters.	A reading line wider than ~95 characters.
Review prompts (needs-review)	Visible to a reviewer; the harness does not pretend to prove it automatically.	Visual weight repeated so evenly that priority is unclear.

That distinction matters for design work. A contrast threshold and a weak visual hierarchy should not be reported with the same certainty.

Example Fixture

Run the merchant dashboard fixture in one terminal:

pnpm example:serve

Then audit it:

pnpm design-harness -- audit --url http://localhost:4173 --out runs/merchant-dashboard

The fixture gives new contributors a stable target for checking the full local loop.

Recipes

• GitHub Actions: run the harness in CI, upload artifacts, and optionally comment on pull requests.
• Pull Request Comment Bot: checkout-local recipe to render a compact PR comment from an audit run.
• Scenario Audit: checkout-local recipe to run multiple local URL scenarios and aggregate the results.
• npm Execution: verify the packed CLI and the npx execution path.
• Agent Loop Recipes: prompts for Codex, Claude Code, Gemini CLI, and human reviewers.
• Release Checklist: package checks to run before publishing a public version.

Packages

Package	What it does
@design-harness/cli	Command-line entry point for the local audit workflow.
@design-harness/core	Schemas, validation, scoring, report rendering, shared types.
@design-harness/visual-audit	Playwright screenshot capture and source-backed DOM, layout, accessibility, and interaction checks.

Status

Implemented:

• local CLI audit loop, published to npm,
• desktop/mobile screenshot capture,
• schema-valid audit artifacts,
• Markdown report generation,
• source-backed criteria registry,
• example fixtures,
• Midjourney Reference Lab manifest and policy validators.

Checkout-local recipes/scaffolding:

• PR comment renderer,
• scenario audit runner,
• MCP adapter manifest and local dispatcher.

In progress or planned:

• package-installed PR comment/scenario/MCP command surface,
• more fixture coverage,
• deeper hosted/action adapters,
• Open Design integrations beyond the current specs.

Midjourney Reference Lab

The repository also includes a manual, local-only calibration workflow: curated good/bad UI reference observations are distilled into hand-authored fixtures, manifest records, and expected findings. Design Harness does not call or automate Midjourney, does not require a Midjourney account, and never depends on generated assets at runtime. See Midjourney Reference Lab Workflow.

Contributing

Design Harness is Apache-2.0 licensed. See CONTRIBUTING.md.

The best first contributions are small and evidence-oriented:

• add a good/bad fixture pair,
• improve a report recommendation,
• map a check to a stronger source-backed criterion,
• add a regression test for a real UI failure mode,
• help turn a needs-review calibration signal into a measurable heuristic.