Add eval harness for verifying agent skill-following behavior by jeryj · Pull Request #80703 · WordPress/gutenberg

jeryj · 2026-07-24T19:56:44Z

What?

Follow up to #80650

Adds an eval harness that verifies AI agents actually follow the repository's skills: given a scenario's task prompt, it checks — from the agent's own transcript — that the right skill files and docs were read, the wrong ones weren't, in the right order. One command, runnable against any model you have access to.

Why?

The skills and AGENTS.md guidance introduced in #80650 only work if agents actually follow them, and until now the only way to verify that was manually: run a prompt, capture the transcript, and grep it. While developing the testing skill we saw that compliance is probabilistic and regresses silently — small wording changes flipped whether an agent read the e2e guide at all. That class of regression is invisible to every existing check in the repo, so skill changes need a repeatable behavioral test. (See also #80699, which adds a skill this harness can grow scenarios for.)

How?

test/ai-development/run.mjs — zero dependencies. Default run does static sanity checks (skill frontmatter, scenario files parse). --live runs each scenario's query through a headless agent session, normalizes the transcript into read/write/command events, and evaluates the scenario's declarative assertions (readsInclude, readsExclude, readsBeforeWrite, commandOrder, commandRules, …), reporting pass rates per assertion with the actual reads/commands printed on failure.
Scoped to agent behavior only: no running WordPress environment is required, and the agent is never granted write permissions — a denied edit attempt is still transcript evidence, and the checkout is never modified. Whether Gutenberg's tests themselves pass is deliberately out of scope.
--model passes through to the agent CLI; --agent selects an adapter (claude today — another CLI is one adapter entry that normalizes its transcript into the same events).
Three seed scenarios distilled from transcript-validated manual runs: running e2e tests (scoped/headless/env-checked), writing an e2e test (authoring guide read before the first edit attempt), and a negative control asserting trivial tasks read no skills or guides (guards against over-triggering).
docs/contributors/code/agents-and-skills.md: the add-a-skill checklist now includes adding an eval scenario.

Testing Instructions

Live runs spawn real agent sessions (minutes and tokens each); they require the claude CLI to be signed in.

Run node test/ai-development/run.mjs — confirm the terminal prints OK: 1 skill(s) and 3 scenario(s) passed sanity checks.
Run node test/ai-development/run.mjs --live --scenario negative-control-trivial-task (~1 minute) — confirm the terminal prints PASS (1/1) for all three reads exclude assertions.
Run node test/ai-development/run.mjs --live --scenario testing-write-e2e (~3–5 minutes) — confirm the terminal prints PASS (1/1) for all six assertions, and confirm git status afterward shows no modified files (only a new transcript under test/ai-development/artifacts/).
Optional: repeat step 3 with --model haiku appended — confirm the same assertions are evaluated against the cheaper model (the run header shows [claude:haiku]).

Use of AI Tools

Authored with Claude Code (Claude Fable 5) in an interactive session directed and reviewed by the PR author; the scenarios encode behaviors first validated in captured agent transcripts.

🤖 Generated with Claude Code

Verifies that an agent, given a scenario's task prompt, consults the right guidance: reads the skill files and docs it should, doesn't read the ones it shouldn't, in the right order, and reaches for the right kind of command — with the evidence coming from the agent's transcript. - eval/harness/run.mjs: static sanity checks (skill frontmatter, scenario files parse) plus a live mode that runs scenario queries through a real agent session headlessly, normalizes the transcript into read/write/command events, and evaluates declarative assertions with pass rates (--scenario, --model, --agent, --repeat). Needs no running environment and grants no write permissions — a denied edit attempt is still evidence, and the checkout is never modified. - eval/scenarios/: two scenarios distilled from transcript-validated manual runs against the testing skill, plus a negative control that guards against over-triggering (trivial tasks must not read skills or contributor guides). - docs/contributors/code/agents-and-skills.md: adding a skill now includes adding an eval scenario. Verified live: testing-write-e2e passes all six assertions end to end; the negative control passes all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

github-actions · 2026-07-24T19:56:56Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: jeryj <jeryj@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: scruffian <scruffian@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

github-actions · 2026-07-24T20:01:34Z

Size Change: 0 B

Total Size: 7.75 MB

_{compressed-size-action}

- Relocate eval/ to test/ai-development/ — test infrastructure lives under test/ — flattening the harness/ subdirectory. Scenario and artifact paths now resolve relative to the script itself. - Format run.mjs with the repo formatter and allow console output for this CLI script. The initial commit never met the formatter because lint-staged's patterns do not cover .mjs files; CI's lint job caught it. - Update path references in the scenarios README and the agents-and-skills.md checklist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

scruffian · 2026-07-27T19:37:37Z

I think we can probably remove:

• Generic multi-agent adapter and --agent
• Unused final-result assertions
• Unrelated skill-frontmatter validation
• Support for scenarios without assertions

ciampo · 2026-07-27T19:34:23Z

+			const args = [
+				'-p',
+				scenario.query,
+				'--verbose',
+				'--output-format',
+				'stream-json',
+			];
+			if ( model ) {
+				args.push( '--model', model );
+			}
+			return args;


It looks like the live run does not prevent the invoked agent from modifying the checkout? The "write-e2e" prompt explicitly asks the agent to add a test, so a configuration that allows edits can change the checkout even though the script and scenario README promise that live runs never do.

Should we enforce a safer / more limited boundary? Maybe run against an isolated/disposable checkout with an explicit read-only permission policy? At the moment, we'd be relying on the inherited user/project configuration

ciampo · 2026-07-27T19:39:35Z

+	fs.mkdirSync( artifactsDir, { recursive: true } );
+	const artifact = path.join(
+		artifactsDir,
+		`${ scenario.slug }-${ Date.now() }-${ runIndex }.jsonl`
+	);
+	fs.writeFileSync( artifact, out.stdout ?? '' );
+	if ( out.error ) {
+		throw new Error(
+			`${ adapter.command } failed to run: ${ out.error.message }`
+		);
+	}
+	return {
+		outcomes: checkAssertions(
+			scenario.assertions,
+			adapter.parse( out.stdout ?? '' )
+		),
+		artifact,
+	};


In the current iteration, only out.error aborts the run, although spawnSync can fail in other ways, too (eg returning a non-zero status, or a terminating signal).

This could lead to reporting successful sessions even though no agent session ran (eg authentication errors that fail before emitting a transcript).

We should probably check status/signal and fail the scenario with its stderr before parsing assertions. Requiring a completed transcript/result would also make the success signal more trustworthy, too.

ciampo · 2026-07-27T19:45:54Z

+			}
+		}
+	}
+	return failed;


If the scenario in the command is misspelled (or if there aren't any assertions), the script skips the scenarios and returns success (without any live evaluation), potentially sylently veryfying nothing? Should it fail, instead?

Not sure if it's a big deal, though

ciampo · 2026-07-27T19:48:36Z

+				"matching": "test:e2e",
+				"mustContain": ".spec.js",
+				"mustNotContain": [ "--headed", "--ui", "--debug" ]
+			}


The scenario claims to require this sequence:

Run npm run wp-env-test status.

Start the environment only if needed.

Run the scoped e2e test.

But its assertion only says: if the agent runs wp-env-test start, it must previously have run wp-env-test status.

So this behavior incorrectly passes: npm run test:e2e -- test/e2e/specs/editor/blocks/paragraph.spec.js

It runs the test without checking the environment first, yet never runs wp-env-test start, so the conditional ordering assertion passes. The scenario should separately require a wp-env-test status command.

jeryj · 2026-07-28T15:13:09Z

Closing in favor of #80758, which leans on Playwright instead of rolling our own eval framework.

jeryj requested review from ajitbohra, fabiankaegy and juanmaguitar as code owners July 24, 2026 19:56

jeryj added [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Code Quality Issues or PRs that relate to code quality and removed [Type] Code Quality Issues or PRs that relate to code quality labels Jul 24, 2026

jeryj and others added 2 commits July 24, 2026 16:15

Update docs to include live test run

0f7fdc2

jeryj self-assigned this Jul 27, 2026

jeryj mentioned this pull request Jul 27, 2026

Add agent skill evals via Playwright #80758

Open

ciampo reviewed Jul 27, 2026

View reviewed changes

jeryj closed this Jul 28, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add eval harness for verifying agent skill-following behavior - #80703

Add eval harness for verifying agent skill-following behavior#80703
jeryj wants to merge 3 commits into
trunkfrom
add/skills-eval

jeryj commented Jul 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

scruffian commented Jul 27, 2026

Uh oh!

ciampo Jul 27, 2026

Uh oh!

ciampo Jul 27, 2026

Uh oh!

ciampo Jul 27, 2026

Uh oh!

ciampo Jul 27, 2026

Uh oh!

jeryj commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jeryj commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Testing Instructions

Use of AI Tools

Uh oh!

github-actions Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

scruffian commented Jul 27, 2026

Uh oh!

ciampo Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ciampo Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ciampo Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ciampo Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

jeryj commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jeryj commented Jul 24, 2026 •

edited

Loading

github-actions Bot commented Jul 24, 2026 •

edited

Loading