test(agent-hosts): collapse duplicated plugin tree inventories and prose-shape assertions into behavioral acceptance

[ScriptedAlchemy]

Scope

Current master deletion/test-quality audit at b9204dcf7de8faac4ef05af367a546aa29c32e9f.

The shared plugin/ tree successfully removed three source bundles, but the test surface still carries parallel descriptions of that same tree. The result is substantial duplicated inventory/prose/schema maintenance that can fail on harmless wording/layout changes while missing installer/runtime regressions.

Evidence

tests/agent_suite/claude_plugin_bundle_test.rs explicitly says it mirrors sibling suites:

• plugin_manifest_schema_test.rs
• plugin_config_schema_test.rs
• plugin_skill_contract_test.rs

It then maintains its own hard-coded EXPECTED_SKILLS, EXPECTED_COMMANDS, and EXPECTED_AGENTS arrays, its own filesystem walkers/frontmatter helpers, and separate assertions over Claude manifest/MCP/hooks/skills/commands/agents.

plugin_skill_contract_test.rs independently walks plugin/skills, validates frontmatter, compares installed/source trees, and includes many tests that assert Markdown bodies contain specific phrases/tool names. Similar host/schema checks exist elsewhere in agent_suite.

The repo guidance already says acceptance should not use source-shape/string scans, giant inventories, or PR-specific bureaucracy when direct behavior exists. Plugin validation should follow that rule too.

Required design

Retain three kinds of evidence only:

1. Canonical source validation — validate each source artifact once against the maintained host/schema contract that genuinely owns its syntax.
2. Installer behavioral acceptance — install each supported host from the real production installer into an isolated home and assert the resulting owned files/registrations/components match the canonical component set and preserve unrelated peer files.
3. Runtime/host journey acceptance — where practical, exercise the installed configuration through the host-visible route or the production parser/dispatcher rather than checking that documentation text contains a magic phrase.

Delete duplicate test-local inventories when the production catalog/component-set authority already enumerates the same files/skills/agents. Tests should derive expected membership from the canonical production authority or compare installed output to the canonical source view, not maintain a second hand-written list.

Delete/merge candidates

• Fold claude_plugin_bundle_test.rs checks into the canonical schema/component/install tests and delete the mirrored per-host copies that add no distinct behavior.
• Replace local filesystem helpers/frontmatter parsing with the existing shared plugin_validation_support when the behavior is identical.
• Remove tests whose only contract is that a skill Markdown body contains a list of exact strings/tool names, unless the literal is itself a shipped wire token or safety requirement. Prefer testing dispatch/component metadata or parsing the structured frontmatter that drives behavior.
• Keep byte-identity tests only where byte identity is an actual installer contract. Do not duplicate them once at source, once per host, and once post-install if one production journey proves the same invariant.
• Keep host-specific tests for real host differences: manifest schema, registration path/format, overlays, permissions, rollback/peer preservation, lifecycle behavior.

Acceptance

• Produce a before/after inventory of agent-suite test files/test counts and delete redundant helpers/inventories/assertions rather than merely moving them.
• Every removed test must be covered by either a surviving canonical source validator or a production installer/host journey; list that replacement mapping in the PR.
• Adding/removing a canonical skill/agent/command should require changing one production authority/source, not multiple hard-coded expected arrays in tests.
• A harmless prose rewording in a skill document should not break tests unless the wording is itself a required external contract.
• Corrupt manifest/config/frontmatter, missing required component, wrong host registration, unexpected deletion of peer files, and installer rollback must still fail falsifiably.
• Do not replace removed Rust tests with shell source scans or snapshot-count gates.

Source review only; no test runtime measurements were performed in this audit.

[ScriptedAlchemy]

Follow-up skill-authoring audit adds two behavioral owners rather than expanding this test-cleanup issue:

• refactor(skill-writer): optimize routing behavior instead of forcing Use-when syntax #891 replaces mandatory Use when/length source-shape correctness with routing/outcome validation for managed and bundled skill descriptions.
• refactor(skills): shrink bundled workflows to progressive domain guidance #892 owns deleting tool-manual/process boilerplate from bundled skill bodies and moving genuinely useful detail behind progressive references/live tool help.

When implementing #888, keep real parser/frontmatter/host interoperability limits, but delete prose assertions made obsolete by #891/#892 instead of recreating equivalent string scans elsewhere.

[ScriptedAlchemy]

Status (2026-09-05): already implemented on Codex's unpushed branch codex/astra-takeover-sept5 (worktree /home/zack/.codex/worktrees/48a7/tracedecay), not on any pushed ref yet. Not duplicating on #707's branch. Close when that branch lands; residuals noted by the review lane: #888 removed the only 'every MCP tool taught by a skill' gate (deferred to #891), #890/#892 leave plugin/README.md 'Announce:' paragraph and shared_skill_contract_test MIN_DESCRIPTION_CHARS gate stale until #888 lands.