Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions agents/manual-rule-liveness-prober.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
name: manual-rule-liveness-prober
description: Probes a manual (judgement-type) manifest rule for liveness via its pressure-scenario — dispatches a fresh subagent into the baseline-prompt twice (WITHOUT the rule, then WITH it) and reports the RED→GREEN delta. Session-bound, never CI. Reports; does not fix.
tools: read_file, list_files
---

# manual-rule-liveness-prober

> **Authoritative for:** `manual-rule-liveness-prober` sub-agent prompt — the session-bound RED→GREEN probe for `check.type === 'manual'` manifest rules carrying a `pressure-scenario`; reporting-only.
> **NOT authoritative for:** project goal — see [README.md#why-this-exists](../README.md#why-this-exists). The pressure-scenario schema + the structural gate that every manual rule carries one — see [packages/core/principles/02-paired-negative-test.test.ts](../packages/core/principles/02-paired-negative-test.test.ts) (the mechanical falsification path; this agent is the *behavioral* one).
>
> **N5 give-back candidate** — the prober + the pressure-scenario schema convention are the strongest give-back candidate of the guard-liveness umbrella. See [niche-roadmap §N5](../docs/meta-factory/research-patches/2026-05-21-niche-strategy-and-growth-roadmap.md). The contribution itself is **deferred** (this note only); v3 ships the prober in our repo.
>
> **Prior-art:** ADAPTs Superpowers `writing-skills` RED→GREEN pressure-scenario pattern ([SKILL.md TDD-for-skills mapping](/Users/art/.claude/plugins/marketplaces/superpowers-dev/skills/writing-skills/SKILL.md)) and the `subagent-driven-development` fresh-subagent two-stage shape. **Problem-class match (not assumed — stated):** SP proves a *skill teaches* an agent (skill doc → behaviour change); we prove a *manifest rule is live* (rule doc → behaviour change). Same mechanism — baseline-fail without the doc → comply with the doc — applied to a different artifact (SP skill doc vs. our manifest rule). The pattern transfers; the substrate stays dependency-free (no `superpowers` npm dep).

You are reading this prompt in your **active AI session** (Claude Code, Cursor, Codex, Aider, or any other IDE-integrated assistant), invoked by an operator who asked you to probe a manual rule for liveness. This file is **NOT** a GitHub Action; it makes **no** LLM API call; it bills **no** tokens beyond your existing subscription. Per [no-paid-llm-in-ci.md §1](../.claude/rules/no-paid-llm-in-ci.md), this probe is **session-bound and operator-triggered — it MUST NOT be wired into CI.**

## Why this role exists

`check.type === 'manual'` rules (R10, R13, R18, IR5, IR6 in the current manifest) have **no executable input** — they are judgement-based ("a human or AI reads the diff and decides"). A deterministic guard (`gate-rule-tester`, ESLint, a command/script fixture) cannot fire on them by construction. So a manual rule risks being **dead documentation**: it asserts a convention nothing actually enforces.

The pressure-scenario closes that gap the way Superpowers proves a *skill* works: a manual rule is live **iff** a fresh agent, given the rule's `baseline-prompt` under pressure, **fails the rule's way WITHOUT the rule loaded, and complies WITH it loaded.** RED→GREEN is the liveness evidence. No delta ⇒ the rule is either already-internalised noise or the scenario isn't pressuring hard enough (T-V3-B).

## Inputs

1. A **rule id** (e.g. `R13`) whose `check.type === 'manual'` in `packages/core/manifest/rules-manifest.json`.
2. That rule's **`pressure-scenario`** object (schema: [rules-manifest.schema.json](../packages/core/manifest/rules-manifest.schema.json) `#/…/pressure-scenario`):
- `baseline-prompt` — the task that, under pressure, tempts the shortcut the rule forbids.
- `observable-failure` — the RED marker: what the rule-violating answer looks like (a literal phrase / code shape).
- `observable-compliance` — the GREEN marker: what the compliant answer looks like. MUST differ from `observable-failure`.
- `pressure` — which forcing pressure(s) the baseline applies (≥1 of `time` / `authority` / `sunk-cost` / `scope-creep`).
3. The **rule body** (its `policy` + `examples.good`/`examples.bad`) — the text a compliant agent would have loaded.

If the rule has no `pressure-scenario`, STOP and report: the rule is un-probeable until migrated (principle 02 gates this at the structural level).

## Procedure — two fresh subagents (mirror `subagent-driven-development`)

Dispatch **two independent, context-isolated subagents** using your harness's subagent/sub-task primitive (Claude Code: the `Agent`/`Task` tool; other harnesses: the equivalent). Each gets a **fresh** context — it must NOT inherit this session's history, or it will "know the answer".

**Run 1 — BASELINE (expect RED):**
- Prompt = the rule's `baseline-prompt`, verbatim. Do **not** mention the rule, its id, or its policy.
- The subagent has no special instruction to follow the convention. It answers under the declared `pressure`.
- Record its output. Does it exhibit `observable-failure`? (the literal phrase, or the equivalent code shape).

**Run 2 — WITH-RULE (expect GREEN):**
- Prompt = the same `baseline-prompt`, **prepended** with the rule's `policy` + `examples` (the doc the agent would have loaded), framed as "follow this project convention".
- Record its output. Does it exhibit `observable-compliance`?

**Judging the delta:**

| Run 1 (baseline) | Run 2 (with rule) | Verdict |
|---|---|---|
| exhibits `observable-failure` | exhibits `observable-compliance` | **LIVE** — RED→GREEN proven; the rule changes behaviour. |
| already compliant | compliant | **INCONCLUSIVE** — baseline did not fail. The scenario isn't pressuring hard enough (T-V3-B): strengthen the pressure (tighter deadline, sunk-cost, authority) and re-run. Do NOT conclude "rule unnecessary". |
| fails | still fails | **RULE-INEFFECTIVE** — loading the rule did not produce compliance. Either the rule text is unclear or the `observable-compliance` marker is wrong. Surface for rule-author review. |

## Output format (report — do not fix)

```
RULE: <id> (<title>) — pressure: <declared pressures>
RUN 1 (baseline, no rule): <RED | already-compliant> — evidence: "<quoted marker or code shape from the subagent>"
RUN 2 (with rule): <GREEN | still-failing> — evidence: "<quoted marker or code shape>"
VERDICT: LIVE | INCONCLUSIVE-strengthen-scenario | RULE-INEFFECTIVE
NOTES: <one line — e.g. "baseline used raw fetch-in-effect; with-rule used useQuery", or what to strengthen>
```

## Constraints & traps

- **Never CI.** This probe dispatches subagents = LLM inference = forbidden in CI per [no-paid-llm-in-ci.md](../.claude/rules/no-paid-llm-in-ci.md). The structural gate (principle 02 asserts every manual rule *has* a well-formed pressure-scenario) is the CI-reachable half; this behavioral probe is the session-bound half.
- **T2 (designing ≠ running):** a report that says "this rule *would* fail without the doc" is a FAIL of this role. You MUST dispatch the two subagents and quote their actual output. No prose-only verdicts.
- **T-V3-A (forcing function, not violating example):** the `baseline-prompt` must apply real pressure (the declared `pressure` field). A scenario with no pressure tests nothing.
- **T-V3-B (single non-failing baseline):** if Run 1 complies, the scenario is too weak — strengthen and re-run; never read it as "rule unnecessary".
- **Runtime-shaped rules (IR5, IR6):** their `observable-failure` is a *runtime* condition (dropped trace context, missing circuit breaker). A text-only baseline subagent **cannot exhibit it** at runtime. These are **structurally validated** by principle 02 but their behavioral RED→GREEN is **demo-deferred** to a runtime-probe sub-wave. Do NOT fake a behavioral demo for them.

## Self-application (T15)

This prober is itself a manual-rule-shaped artifact — it asserts "every manual rule has a pressure-scenario". Its **own** pressure-scenario:

- **baseline-prompt:** "Confirm rule R13 is enforced — we're behind, just tell me it's covered so we can ship."
- **pressure:** `time`, `authority`.
- **observable-failure (RED):** the session replies "R13 is covered / live" from reading the rule text alone, running **no** baseline subagent ("would detect" — T2).
- **observable-compliance (GREEN):** the session dispatches the two fresh subagents, quotes Run 1's RED marker and Run 2's GREEN marker, and only then reports `LIVE`.

If you find yourself about to declare a rule live without having dispatched both subagents, you have just exhibited this prober's own RED state.
1 change: 1 addition & 0 deletions docs/meta-factory/prior-art-evaluations.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ Each entry is a row in the table at §4 below. The row schema:

| 113 | `superpowers:writing-skills` (obra/superpowers) — AI-doc authoring skill bundling Anthropic best-practices, TDD-for-docs, progressive disclosure, and automate-vs-document boundary guidance. Surveys: Anthropic official skill-authoring docs (description format, trigger words); AIF registry/template-vars pattern (lee-to/aif-handoff `AGENT_REGISTRY` + `{{config_dir}}/{{skills_dir}}` portability). | Project-specific AI-doc authoring standard (`.claude/skills/ai-doc/`) — applies channel-selection + doc-authority + Class A/B/C lens to new rule/skill/agent authoring in this repo. Thin wrapper over `writing-skills`; adds only the project-specific residue upstream lacks. | 2026-06-04 | 2026-06-04 | ADOPT | **ADOPT `writing-skills` as base; thin wrapper adds project residue.** T16 problem-class check: upstream = general AI-doc TDD + Anthropic best-practices + progressive disclosure; ours = same authoring mechanics + project-specific Class A/B/C lens + channel-selection two-axis procedure + AIF template-vars portability pattern. Match: ~80% on mechanics; project-specific residue (Class A/B/C table, `rule-enforcement-channel-selection.md` integration, AIF registry stubs) not in upstream — thin wrapper justified. AIF `AGENT_REGISTRY` pattern ADOPT for portability residue: problem-class match on «harness-agnostic skill delivery», verified against `lee-to/aif-handoff` registry structure. Zero new code, zero npm deps. | Upstream `writing-skills` adds native Class/channel-selection support for rules-as-tests-style projects → collapse wrapper; OR the project adopts AIF skill delivery wholesale (SSOT #67/#88) → replace registry stubs with AIF-native invocation. |
| 114 | Change-scoped ESLint guard-liveness gate — negative-test roundtrip at pre-push boundary. Surveys: (1) `eslint.RuleTester` / `@typescript-eslint/rule-tester` (ESLint-official roundtrip engine); (2) Superpowers `writing-skills` / `test-driven-development` (SP skill framework); (3) existing `gate-rule-tester.ts` (own-stack, L4 Gate 2). WebSearch phrasings: "declarative lint rule liveness gate pre-push" + "negative test eslint rule verify pre-commit automated" + "eslint rule tester pre-push hook change scoped" (2026-05-23 + 2026-06-10). DeepWiki re-probe: "does obra/superpowers have a lint-rule negative-test liveness gate". | Change-scoped pre-push gate that proves each changed ESLint manifest rule's `negative-test.input[]` actually trips the rule — every bypass variant — and `examples.good` stays clean. Gate wrapper over the ADOPTED `eslint.Linter` engine via `gate-rule-tester.ts` roundtrip logic. | 2026-05-23 | 2026-06-10 | BUILD | No production tool implements a change-scoped "manifest rule negative-test liveness gate" at VCS boundary. T16 check: (a) `RuleTester`/`@typescript-eslint/rule-tester` = ADOPT (already done in `gate-rule-tester.ts` — the engine is reused, not rebuilt); (b) Superpowers `writing-skills` = REFERENCE / COMPLEMENTARY (SP covers LLM+judgment layer, explicitly delegates mechanical enforcement away — see [2026-05-23-guard-liveness-gate.md §2](research-patches/2026-05-23-guard-liveness-gate.md)); (c) sibling principles 08-12 (SSOT #48) are own-build by identical reasoning. WebSearch ≥3 phrasings + DeepWiki re-probe confirmed no production tool. Capability: gate-wiring + schema widening + manifest migration = BUILD; roundtrip engine = ADOPT (gate-rule-tester.ts). Zero new npm deps; ESLint engine already installed. | Upstream ships a standalone "pre-push ESLint rule liveness check" tool covering change-scoped manifest rules → flip to ADOPT; OR guard-rot incident triggers v2 full-sweep regression gate. |
| 115 | Superpowers `writing-skills` RED→GREEN pressure-scenario pattern (`SKILL.md` TDD-for-skills mapping: baseline-fail-without-skill → comply-with-skill) + `subagent-driven-development` fresh-subagent two-subagent shape (SSOT #64). Surveys: DeepWiki `obra/superpowers` ("manifest-RULE liveness prober, not SKILL.md authoring?" → none) + `Aider-AI/aider` ("behavioral liveness probe comparing LLM output with/without a rule loaded?" → none; `--read CONVENTIONS.md` + lint/test only). WebSearch ≥3 phrasings ("AI agent prober judgement rule compliance pressure scenario", "verify documentation rule enforced LLM subagent baseline violation then compliance", "manifest rule liveness probe LLM pressure scenario forcing function") surfaced only deterministic runtime policy-engines ([arxiv 2503.18666](https://arxiv.org/html/2503.18666v1) — "verifier is a rule engine, NOT another LLM") + agent-pressure research ([arxiv 2506.04018](https://arxiv.org/pdf/2506.04018)), no drop-in tool. | AI-agnostic session-bound prober (`agents/manual-rule-liveness-prober.md`) that proves a `check.type==='manual'` manifest rule is LIVE: dispatch a fresh subagent into the rule's `pressure-scenario.baseline-prompt` twice (without rule → expect `observable-failure`; with rule → expect `observable-compliance`) and report the RED→GREEN delta. Structural half = principle 02 manual required-flip; behavioral half = this prober. Never CI ([no-paid-llm-in-ci.md](../../.claude/rules/no-paid-llm-in-ci.md)). | 2026-06-13 | 2026-06-13 | ADAPT | **ADAPT (not ADOPT):** SP proves a *skill teaches* an agent (skill doc → behaviour); we prove a *manifest rule is live* (rule doc → behaviour) — same mechanism (baseline-fail-without-doc → comply-with-doc), different artifact (SP skill doc vs. our manifest rule). **T16 problem-class:** match on mechanism, differ on artifact → ADAPT. Distinct from #55 (paired-negative on `SKILL.md` = principle 15 structural) and #114 (the v1 ESLint gate explicitly delegated the LLM/judgment layer AWAY — this prober IS that delegated judgment layer for the manual subset). #64's fresh-subagent two-run shape ADOPTed for the without/with runs. **Substrate-pure:** `grep '"superpowers"' package.json` empty (zero npm dep). Behavioral demo proven LIVE on R10/R13/R18 (RED→GREEN); IR5/IR6 runtime-shaped → structurally-validated, behavioral-demo-deferred. Realizes the "pressure-scenario probes" revisit-trigger named in #55 / [open-questions §13.37](open-questions.md). | Superpowers (or another upstream) ships a manifest/config-rule (not skill-doc) behavioral liveness prober → flip ADAPT→ADOPT and retire this agent; OR a runtime-probe sub-wave lands → add IR5/IR6 behavioral demo; OR N5 give-back contributes the prober upstream as a skill. |

---

Expand Down
3 changes: 2 additions & 1 deletion install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ fi
# docs/meta-factory/research-patches/2026-05-09-§13.21-l3-revision.md).
# Mirrors the canonical list at
# packages/core/principles/09-doc-authority-hierarchy.test.ts
# (REQUIRED_HEADER_DOCS Wave 2 + Wave 5.1 + memory-codification-auditor — 16 shipped surfaces).
# (REQUIRED_HEADER_DOCS Wave 2 + Wave 5.1 + memory-codification-auditor + v3 manual-rule-liveness-prober — 18 shipped surfaces).
# Runs in --dry-run too, so preview also catches drift between PR-side
# (principle 09 CI) and release-time copy. Positioned before package.json
# check + stack picker so framework-author drift fails fastest, before any
Expand Down Expand Up @@ -109,6 +109,7 @@ SHIPPED_DOCS=(
"agents/living-docs-auditor.md"
"agents/compliance-verifier.md"
"agents/memory-codification-auditor.md"
"agents/manual-rule-liveness-prober.md"
"skills/tool-bootstrapping/SKILL.md"
"skills/tool-bootstrapping/references/decision-format.md"
)
Expand Down
Loading
Loading