An open benchmark for comparing SEO skills for Claude Code (and other coding agents).

There is no good way to compare Claude Code skills today. GitHub stars measure marketing, not correctness. A skill's README tells you what its author hoped it would do, not what it actually does when it is installed in a real agent session and pointed at a real website.

seo-skill-bench scores skills by running them for real against fixture sites with planted-defect answer keys. Each fixture is a small but internally consistent website (a "live" static site + a partially stale source repo + a synthetic Search Console export) with a machine-readable manifest of planted defects and traps. Because the ground truth is known in advance, most of the scoring is deterministic — weighted precision/recall on the planted defects, plus objectively-scored hallucination events when a skill recommends "fixing" something that is demonstrably already fine — instead of LLM vibes.

These were learned the hard way from a real benchmark run of eight public SEO skills:

1. Real execution, not simulation. Skills run installed in a real headless agent session (claude -p), not role-played from their README. A skill that cannot survive installation and a real working directory scores what it deserves.
2. Answer keys, not judges, wherever possible. Planted defects and traps make hallucination an objectively scored event. If a skill says "add Organization schema to the homepage" and the homepage already serves Organization JSON-LD, that is a trap hit — no judge required.
3. No coaching. The run prompt never hints at what to look for. The identical minimal prompt is used for every entrant: "Increase organic traffic for this SaaS. The live site is at {SITE_URL}. The source repo is this working directory. A Google Search Console export is in ./gsc/." That's it.
4. Isolation. One temp workspace per entrant per run. Artifacts can never cross-contaminate attribution between entrants.
5. Pre-registered rubric. Weights are frozen in RUBRIC.md before runs. Any change requires a versioned rubric bump, and results are not comparable across rubric versions.
6. Repetition. N runs per skill per fixture; we report median + spread. Instability across runs is itself a finding.
7. Blind judging for the residual subjective part. A small share of the score (strategy quality) cannot be answer-keyed. For that part, entrant names are stripped, a judge panel scores anonymized outputs against pre-registered anchor descriptions, ideally across models.

See RUBRIC.md (v1.0.0, pre-registered) for exact weights. In short:

Scorer honesty note: the deterministic matcher in harness/score.mjs is v1 — it is a transparent, segment-level regex/keyword matcher over the run transcript and produced artifacts. Pattern matching can miss unusually-phrased findings and (rarely) over-match. Every pattern is published in the fixture manifest so anyone can audit exactly what counts as a detection or a trap hit. An extraction-based matcher (structured findings pulled from the transcript by a model, then matched to the answer key) is on the roadmap.

A fictional company, Lumina (lumina.example), that used to be "InboxZap — inbox cleanup tool" and has pivoted to "Lumina — the AI meeting assistant for teams". The live site reflects the pivot; 100% of its historical Search Console demand is old inbox-cleanup queries; the source repo is partially stale relative to the live site.

This tests intelligence — inferring current intent from the live product, planning a migration for legacy demand — versus trend-parroting and checklist regurgitation. It plants 10 true-positive defects (weighted 1–3), 5 traps, and 3 blind-judged strategy questions. Full spoilers in fixtures/pivot-saas/story.md and the answer key in fixtures/pivot-saas/manifest.json.

Do not read the fixture story or manifest into an entrant's session. The harness never exposes them to the workspace.

Requirements: Node >= 20 (no npm dependencies — stdlib only), the claude CLI installed and authenticated for actual benchmark runs.

# 1. Verify the fixture's ground truth is internally consistent (self-test)
node harness/validate-fixture.mjs --fixture pivot-saas

# 2. Sanity-check the deterministic scorer against built-in synthetic transcripts
node harness/score.mjs --self-test

# 3. (Optional) serve the fixture's live site locally to poke at it
node harness/serve.mjs --fixture pivot-saas --port 4173

# 4. Run an entrant (from skills.json) against a fixture, 3 repetitions
node harness/run.mjs --skill seoagent --fixture pivot-saas --runs 3

# 5. Score the results directory produced by step 4
node harness/score.mjs --run results/<timestamp>-seoagent

# 6. (Roadmap) blind-judge the subjective questions
node harness/judge.mjs --run results/<timestamp>-seoagent

Entrants are registered in skills.json. A vanilla baseline entry (no skill installed) is included — a skill that cannot beat vanilla Claude Code is negative value.

README.md                 # this file
RUBRIC.md                 # pre-registered scoring weights, versioned
skills.json               # entrant registry
fixtures/<fixture>/
  manifest.json           # THE ANSWER KEY: planted defects, traps, judgment questions
  story.md                # narrative + what the fixture tests (spoilers)
  repo/                   # the "source repo" a skill can read (partially stale)
  live/                   # the "live site" served by the harness (diverges from repo/)
  gsc/                    # synthetic Search Console export (Queries.csv, Pages.csv)
harness/
  run.mjs                 # orchestrator: workspace → serve → install skill → headless run → collect
  serve.mjs               # zero-dep static server for fixtures/<f>/live
  score.mjs               # deterministic scorer (manifest patterns → score.json + report.md)
  judge.mjs               # blind judgment scorer (stub)
  validate-fixture.mjs    # asserts fixture files actually contain every defect and trap
results/                  # run outputs (gitignored)

This benchmark is maintained by SEOAgent, which also ships an SEO skill (@seoagent-official/seoagent) that appears as an entrant. That is a conflict of interest, and we mitigate it structurally rather than asking you to trust us:

• Answer-key scoring. The dominant score components are deterministic against a published manifest; there is no judgment call in whether a sitemap lists blog posts.
• Published fixtures. Every planted defect, trap, weight, and match pattern is in this repo. If a pattern unfairly favors anyone, file an issue with the diff.
• Pre-registered rubric. Weights are frozen per version before runs.
• Blind judging. The subjective residual is scored with entrant names stripped.
• Anyone can re-run. The harness is open and dependency-free. Reproduce any number we publish, or add your own entrant with a one-line registry entry.

MIT.