HTMLCraft — Taste-Driven Agent Harness Lab

Encode my HTML taste, turn it into an agent skill, and measure which model and reasoning level can execute it reliably.

HTMLCraft is my experiment field for agent-harness design. It combines:

• my own HTML visual taste and rejection criteria;
• the harness mental models developed in learn-agent-harness;
• small AAR-, CocoaBench-, and HiL-Bench-inspired evaluations for model selection.

The goal is not merely to generate an attractive page. The goal is to answer a practical question with evidence:

For this taste profile and workflow, which model, harness, and reasoning effort gives the best balance of visual quality, task completion, judgment, latency, and cost?

Current version: 1.3.0

The experiment stack

HTML taste
  └─ style catalog + design systems + rejection rules
       ↓
HTMLCraft skill
  └─ brief → art direction → implementation → runtime validation
       ↓
Mini benchmark
  ├─ Navigate: follow the right evidence path
  ├─ Compose: coordinate image + data + code
  └─ Escalate: recognize when human input is required
       ↓
Model decision
  └─ compare model × harness × reasoning effort × cost

HTML generation is therefore both the useful product and the observable test surface.

What HTMLCraft encodes

The skill captures preferences that are otherwise easy to lose between sessions:

• one clear visual thesis instead of a pile of effects;
• purposeful CSS, Canvas, or Three.js rather than mandatory spectacle;
• real copy and honest product proof;
• static-first structure with motion as progressive enhancement;
• responsive, keyboard-accessible, reduced-motion-aware output;
• offscreen pausing, bounded DPR, and renderer cleanup;
• deterministic checks plus browser-level visual review;
• rejection of generic gradient blobs, glass everywhere, ornamental bento grids, fake system labels, and copied reference identity.

Its default artifact is a portable single HTML file with inline project CSS and JavaScript. Remote libraries are used only when the selected technique earns the dependency.

Use the skill

Codex:

$htmlcraft create an editorial portfolio for a systems engineer; warm paper, dense typography, no decorative particles

Claude Code:

/htmlcraft create an editorial portfolio for a systems engineer; warm paper, dense typography, no decorative particles

The skill resolves the brief, writes a compact art direction, chooses the smallest sufficient stack, creates the page, and validates the result.

Validate an artifact directly with:

python3 scripts/validate_html.py output.html --strict

Benchmark models and reasoning effort

The bundled six-case suite is intentionally small enough for frequent regression runs.

Track	Inspiration	What HTMLCraft measures	Metrics
Navigate	The Amazing Agent Race	Correctly follow sequential and fork/merge evidence instead of using attractive decoys	FA, PVR, RCR
Compose	CocoaBench	Combine a raster reference, local CSV/JSON, computation, and accessible HTML	Success, vision, data, implementation
Escalate	HiL-Bench	Discover missing or contradictory requirements and ask precise questions without spamming	Precision, Recall, Ask-F1, task pass

These are HTMLCraft-specific mini evaluations. Their scores are not comparable to the official benchmark leaderboards.

Prepare a candidate run

python3 scripts/benchmark_htmlcraft.py prepare \
  --model gpt-example-medium \
  --out /tmp/htmlcraft-gpt-example-medium

Run each generated TASK.md in a fresh session with the same harness permissions, timeout, and HTMLCraft revision. Use the model label to record reasoning effort explicitly, for example:

gpt-example-low
gpt-example-medium
gpt-example-high

Score and compare

python3 scripts/benchmark_htmlcraft.py score /tmp/htmlcraft-gpt-example-medium

python3 scripts/benchmark_htmlcraft.py compare \
  /tmp/htmlcraft-gpt-example-low/result.json \
  /tmp/htmlcraft-gpt-example-medium/result.json \
  /tmp/htmlcraft-gpt-example-high/result.json

The summary keeps the three capability profiles visible and also provides a convenience aggregate. Record tokens, latency, and estimated cost from the surrounding harness; this mini suite deliberately does not invent cross-provider accounting.

For help-seeking experiments, run both blocked and full-info conditions. The difference in final-task success exposes the judgment gap between “can solve when told everything” and “knows when to ask.”

See the full benchmark protocol for case isolation, scoring definitions, repeated runs, and limitations.

Install

With the skills CLI:

npx skills add SunChJ/htmlcraft

Codex user-level installation:

git clone https://github.com/SunChJ/htmlcraft ~/.codex/skills/htmlcraft

Claude Code user-level installation:

git clone https://github.com/SunChJ/htmlcraft ~/.claude/skills/htmlcraft

Restart the agent or open a new session after installation.

Repository map

htmlcraft/
├── SKILL.md                        # Taste-driven generation and validation workflow
├── agents/openai.yaml              # Codex skill metadata
├── references/
│   ├── style-catalog.md            # Visual languages and selection guidance
│   ├── design-systems.md           # Type, color, layout, and detail systems
│   ├── quality-bar.md              # Taste, trust, accessibility, performance, QA
│   ├── threejs-patterns.md         # Lifecycle-safe Three.js kernels
│   └── canvas-patterns.md          # Lifecycle-safe Canvas 2D kernels
├── scripts/
│   ├── validate_html.py            # Deterministic artifact validation
│   └── benchmark_htmlcraft.py      # Prepare, score, and compare candidate runs
└── benchmarks/htmlcraft-mini/
    ├── cases.json                  # Hidden-from-candidate assertions and blockers
    ├── cases/                      # Six isolated task fixtures
    └── README.md                   # Evaluation protocol

Interpreting results

Do not choose a model from overall alone.

• High visual/task quality with low Ask-F1 usually means the model works when the brief is complete but guesses when it is not.
• High PVR with low FA usually means it found the right evidence but failed during composition or final delivery.
• High Cocoa vision with low implementation indicates good visual understanding but weak artifact execution.
• A larger model that ties a smaller one while using much more time or tokens is not automatically the better HTMLCraft model.

Use at least three repetitions for a serious decision. Six cases are useful for regression signals, not statistical certainty.

Versioning

HTMLCraft follows semantic versioning. See CHANGELOG.md for changes to the taste profile, workflow, validators, and benchmark contract.

License

MIT