Add progressive-discovery pattern for agent instructions and skills by jeryj · Pull Request #80650 · WordPress/gutenberg

jeryj · 2026-07-23T19:03:42Z

What?

Establishes a documented pattern for how AI-agent instructions are organized in the Gutenberg repository, and lands the first skill (testing):

Root AGENTS.md becomes a lean router with a Progressive discovery section: contributor docs, skills, and directory guides are read only when the task needs them.
New top-level skills/ directory: skills/testing/SKILL.md intro point with references/ files for Jest, PHP, and e2e specifics.
New contributor guide at docs/contributors/code/agents-and-skills.md.

Why?

Agent guidance in the repo is currently ad hoc (root AGENTS.md, one per-directory file in packages/components/), with no documented convention for contributors. By providing a general structure, we have a way to start contributing to improve agent success in coding in Gutenberg.

How?

Follows conventions from WordPress Agent Skills to progressively disclose skills to prevent context bloat and confusing agents:

Content is placed by trigger: always-loaded (root AGENTS.md: env setup, code quality, pitfalls, router) / location-triggered (directory AGENTS.md) / task-triggered (skills/) / on-demand (docs and references). Each fact has one home, at the highest level where it applies to everything below it.
The testing command block moved from root AGENTS.md into the skill's per-type references; the shared wp-env-test requirement lives once in SKILL.md. Non-discoverable trap rules (PHP build-time function prefixing, per-package CHANGELOG.md entries) moved to Common pitfalls.
The guidance doc's example is a skeleton (not a copy of the live skill) so it cannot drift; the live example is linked instead.

Testing Instructions

Start a fresh agent session with default configuration: claude (interactive), or for a captured run: claude -p "Run the e2e tests for the paragraph block" --verbose --output-format stream-json > run.jsonl.
Run grep -o '"file_path":"[^"]*"' run.jsonl to list every file read.
Confirm the file-read list includes skills/testing/SKILL.md and skills/testing/references/e2e.md, and that those reads appear before the first test command.
Confirm the file-read list does NOT include skills/testing/references/php.md or skills/testing/references/jest.md
Run grep -o '"command":"[^"]*"' run.jsonl to list every shell command.
Confirm the command list contains npm run wp-env-test status before any wp-env-test start.
Confirm it contains a spec path after npm run test:e2e -- (e.g. editor/blocks/paragraph.spec.js) and does not contain --headed, --ui, or --debug. Cross-check: no browser window opened during the run.
Start a new agent session: claude -p 'Add an e2e test for the paragraph block that checks that the `&` key is not output as `&` in the editor' --verbose --permission-mode acceptEdits --output-format stream-json > run.jsonl
Run grep -o '"file_path":"[^"]*"' run.jsonl to list every file read.
Confirm the file-read list includes the relevant docs and sklls, docs/contributors/code/e2e/README.md, skills/testing/SKILL.md and skills/testing/references/e2e.md, and that those reads appear before the first test command.

Use of AI Tools

Authored with Claude Code (Claude Fable 5) in an interactive session, with structure and content directed and reviewed by the PR author. The comparison against WordPress/agent-skills conventions was also AI-assisted and human-verified.

🤖 Generated with Claude Code

Establish how AI-agent guidance is organized in the repository: - Root AGENTS.md becomes a lean router: a Progressive discovery section routes to contributor docs, skills, and directory guides. The testing command block moves out (code-quality commands stay); non-discoverable trap rules (PHP build prefixing, package CHANGELOG entries) move to Common pitfalls. - New top-level skills/ directory with a testing skill: thin SKILL.md (frontmatter, shared planning gate, don't-weaken-tests rule, per-type routing) plus references/ for Jest, PHP, and e2e specifics. - New docs/contributors/code/agents-and-skills.md contributor guide: improve public docs first, one home per fact, skill conventions shared with WordPress/agent-skills, and a checklist for adding skills. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

github-actions · 2026-07-23T19:03:51Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: jeryj <jeryj@git.wordpress.org>
Co-authored-by: bgrgicak <berislavgrgicak@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

bgrgicak · 2026-07-24T14:46:33Z

+## Never make a failing test pass by changing the production code unless the production code is the source of a bug
+
+The e2e test passing is not the final task success criteria. The core goal is to verify that the production code works as expected.
+


How does this help? We could test it on a couple of examples where we introduce a bug into either the code or test and see if it makes the correct decision.

I'm worried that the current wording could be misleading. Maybe something like this phrase could work better, but I haven't tested it.

There might be a good skill already available somewhere for fixing failing tests.

## Fix test failure Before fixing a test failure, investigate the rot cause of the failure. If the test is failing due to it being outdated, update the test to match the code. If it's timing out, fix the timeout root cause, increase the timeout as a last resort. Update the code in case the test failure was introduced by a bug in the code.

I've had the agent try to fix a failing test by changing the core code. Trying to keep it focused on the important core task rather than immediate satisfaction of "Fixing the test." However, maybe this is not needing and the proper fix is the developer improving their prompting.

I'm unattached to this and happy to remove it. Core piece I want to accomplish is just the overall pattern.

ciampo · 2026-07-24T14:39:59Z

+
+Agent-specific guidance for PHPUnit tests. They require the wp-env test environment: check `npm run wp-env-test status` first, and run `npm run wp-env-test start` only if it is not already running.
+
+-   **Run**: `composer test` (all PHP tests), or `vendor/bin/phpunit <path_to_test_file.php>` (specific file or directory).


Should we recommend npm run test:unit:php (or npm run test:php when lint is desired), because those scripts run PHPUnit inside the configured wp-env test environment

ciampo · 2026-07-24T14:43:10Z

+## Never make a failing test pass by changing the production code unless the production code is the source of a bug
+
+The e2e test passing is not the final task success criteria. The core goal is to verify that the production code works as expected.


If we plan on doing some test-driven development, I'm afraid this section will prevent it?

Good point. Happy to remove it. Main point for the PR is scaffolding the overall structure of agents and skills. I'd rather be more minimal (remove this line) and discuss specific agent instructions in a follow-up PR.

ciampo · 2026-07-24T14:45:29Z

@@ -0,0 +1,21 @@
+# E2e tests: agent rules and routing
+
+Agent-specific guidance for Playwright e2e tests. They require the wp-env test environment: check `npm run wp-env-test status` first, and run `npm run wp-env-test start` only if it is not already running.


I noticed we're repeating the wp-env-test across a couple of files, should we unify it under the skill.md file for applicable environments?

I believe it's only relevant for php and e2e, so I put them in their separate files.

ciampo · 2026-07-24T14:50:54Z

+
+## Rules
+
+-   **Stay headless** (the default). Do not use `--headed`, `--ui`, or `--debug` — the human docs recommend them, but they open a GUI and block an agent session.


From the canonical e2e guide, it seems like headless mode is not supported in Linux. Maybe we can tweak the guidance here to say "use headless when supported, as per the canonical e2e testing guide"

Sure. It doesn't need to be as assertive as it is.

…files Co-authored-by: Bero <berislav.grgicak@gmail.com>

jeryj · 2026-07-24T15:04:41Z

@bgrgicak @ciampo - My main concern here is the overall architecture pattern and guidelines for adding the agent instructions and skills. The testing skill is meant to be minimal and improved in follow-ups, so the intention isn't a great final testing skill state right now.

github-actions · 2026-07-24T15:39:10Z

Flaky tests detected in 657ee4b.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/30108411927
📝 Reported issues:

[Flaky Test] should cut and paste individual blocks with collapsed selection #80248 in /test/e2e/specs/editor/various/copy-cut-paste.spec.js

ciampo · 2026-07-24T15:49:56Z

So far, no concerns from my end. I suspect we'll find out only by gathering evidence about how different agents do

aduth · 2026-07-24T18:11:12Z

 -   `@wordpress/build` (`packages/wp-build`) is a generic build tool used both in Gutenberg and by plugins targeting WordPress Core directly. Avoid Gutenberg-specific changes in it.
 -   Never invoke WordPress's forked or local CLIs through `npx` (e.g. `npx prettier`, `npx wp-scripts`). WordPress ships its own `wp-prettier` fork, and `wp-scripts` is the bin name of `@wordpress/scripts`. A bare `npx wp-scripts` can resolve to an unrelated third-party package on the public registry, not the local tool. Use the npm scripts instead (`npm run format`, `npm run lint:js`, `npm run lint:css` and so on), which run the binaries from local `node_modules`.
+-   PHP function and class names are renamed at build time (`gutenberg_*` prefix, `*_Gutenberg` suffix) to avoid conflicts with WordPress Core — the built names, not the source names, are what runs (and what tests must call). See `docs/contributors/code/build-system-function-prefixing.md`.
+-   Production code changes in a package require an entry in that package's `CHANGELOG.md`. See `docs/contributors/code/managing-packages.md`.


Nit: If we have a qualifier like "in a package" that refers to a specific place in the codebase, that's a good signal that it belongs in a localized AGENTS.md within the related folder, i.e. packages/AGENTS.md, so that this doesn't need to be loaded into agent context if an agent is working on files outside of packages.

(h/t @simison for his original suggestion to this in #78166 (comment))

I'm also finding that notes about discovery from the AGENTS.md loaded at runtime is pretty much ignored by the time the agent actually performs the task. Skills seem to be more reliable for procedural tasks than AGENTS.md. I say, let's give it a try in a couple ways and see which structure the agent more reliably follows.

jeryj requested review from ajitbohra, fabiankaegy and juanmaguitar as code owners July 23, 2026 19:03

jeryj requested a review from bgrgicak July 23, 2026 19:21

jeryj added [Type] Enhancement A suggestion for improvement. [Type] Developer Documentation Documentation for developers Developer Experience Ideas about improving block and theme developer experience and removed [Type] Enhancement A suggestion for improvement. labels Jul 23, 2026

jeryj added 3 commits July 24, 2026 10:24

Move specific information about environment into e2e and php tests

a7b208e

Prevent agent from changing production code to get an e2e test to pass

9f75737

Remove unnecessary guidance - add reading from README

b41f78e

bgrgicak reviewed Jul 24, 2026

View reviewed changes

ciampo reviewed Jul 24, 2026

View reviewed changes

bgrgicak reviewed Jul 24, 2026

View reviewed changes

Comment thread AGENTS.md Outdated

Improve reading contributor docs instructions to prevent reading all …

fff71d2

…files Co-authored-by: Bero <berislav.grgicak@gmail.com>

Improve reliability of docs reading

657ee4b

ciampo approved these changes Jul 24, 2026

View reviewed changes

jeryj merged commit 290fbff into trunk Jul 24, 2026
43 checks passed

jeryj deleted the add/skills-pattern branch July 24, 2026 18:02

github-actions Bot added this to the Gutenberg 23.7 milestone Jul 24, 2026

aduth reviewed Jul 24, 2026

View reviewed changes

aduth mentioned this pull request Jul 24, 2026

Design System: Add AGENTS guidance for changelogs #78166

Open

This was referenced Jul 24, 2026

Improve PR template guidance and add a pull-requests skill #80699

Open

Add a Gutenberg release knowledge skill #80627

Open

Add eval harness for verifying agent skill-following behavior #80703

Closed

scruffian mentioned this pull request Jul 27, 2026

Add agent skill for backporting PRs to WP Core #80594

Open

aduth mentioned this pull request Jul 28, 2026

Docs: Clarify agent guidance discovery #80753

Merged

ciampo mentioned this pull request Jul 28, 2026

Design System: Add portable agent skills #80597

Open

		## Never make a failing test pass by changing the production code unless the production code is the source of a bug

		The e2e test passing is not the final task success criteria. The core goal is to verify that the production code works as expected.


		Agent-specific guidance for PHPUnit tests. They require the wp-env test environment: check `npm run wp-env-test status` first, and run `npm run wp-env-test start` only if it is not already running.

		- Run: `composer test` (all PHP tests), or `vendor/bin/phpunit <path_to_test_file.php>` (specific file or directory).

		@@ -0,0 +1,21 @@
		# E2e tests: agent rules and routing

		Agent-specific guidance for Playwright e2e tests. They require the wp-env test environment: check `npm run wp-env-test status` first, and run `npm run wp-env-test start` only if it is not already running.


		## Rules

		- Stay headless (the default). Do not use `--headed`, `--ui`, or `--debug` — the human docs recommend them, but they open a GUI and block an agent session.

Uh oh!

Conversation

jeryj commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Testing Instructions

Use of AI Tools

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jeryj commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ciampo commented Jul 24, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jeryj commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 24, 2026 •

edited

Loading