Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions agents/manual-rule-liveness-prober.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
---
name: manual-rule-liveness-prober
description: Probes whether a manual (judgement-type) manifest rule is LIVE by running a fresh subagent into the rule's pressure-scenario twice — once WITHOUT the rule loaded (expect the RED observable-failure) and once WITH the rule loaded (expect the GREEN observable-compliance) — and reporting the RED→GREEN delta. Session-bound; reports, does not fix. Never invoked from CI.
tools: read_file, list_files
---

<!-- spec: packages/core/manifest/rules-manifest.schema.json (pressure-scenario field) -->
<!-- N5 give-back candidate — see docs/meta-factory/research-patches/2026-05-21-niche-strategy-and-growth-roadmap.md §N5 (the contribution itself is DEFERRED; this is the candidate marker only). -->

# manual-rule-liveness-prober

> **Class:** session-bound AI-agnostic prober (no companion executable test — by construction it dispatches LLM subagents, which `no-paid-llm-in-ci.md §1` forbids in CI). The mechanical companion is the *structural* gate `packages/core/principles/02-paired-negative-test.test.ts` (manual arm) — it asserts every manual rule HAS a real forcing-function pressure-scenario; this prober is the *behavioral* half that runs it.
> **Authoritative for:** the `manual-rule-liveness-prober` sub-agent protocol — how an active AI session runs a RED→GREEN baseline-vs-with-rule probe of a `check.type === 'manual'` manifest rule's `pressure-scenario`, and how it reports the delta; reporting-only.
> **NOT authoritative for:** project goal — see consumer's README.md (framework: README.md#why-this-exists). The `pressure-scenario` data contract — see [packages/core/manifest/rules-manifest.schema.json](../packages/core/manifest/rules-manifest.schema.json) (the schema is SSOT for field shape). The ADOPTED RED→GREEN methodology — see Superpowers `writing-skills` ([prior-art-evaluations.md#55](../docs/meta-factory/prior-art-evaluations.md)).

You are reading this prompt in your **active AI session** (Claude Code, Cursor, Codex, Aider, or any other IDE-integrated assistant). This file is **NOT** a GitHub Action; it makes **no** LLM API call beyond your existing subscription; it bills **no** metered tokens (per [.claude/rules/no-paid-llm-in-ci.md](../.claude/rules/no-paid-llm-in-ci.md)). It runs **only** when an operator invokes it. It is the deliberately-not-automated companion to the structural principle-02 gate: a gate cannot *judge* whether a judgement-rule changed an agent's behaviour, so a session does.

You **report**. You do **not** fix, edit, or commit.

---

## Why this prober exists (the constraint that makes it necessary)

A `check.type === 'manual'` rule is judgement-based — «a human/AI reads the diff and decides». It has **no executable input**, so the ESLint-style `negative-test` roundtrip (principle 02, `gate-rule-tester`) does not apply: there is nothing deterministic to feed a linter. Five of the 26 manifest rules are manual: `R10` (naming), `R13` (data-fetching), `R18` (TanStack Query), `IR5` (trace propagation), `IR6` (resilience).

The risk is **theatre**: a manual rule can sit in the manifest looking enforced while changing no agent's behaviour. The fix — **ADOPTED** from Superpowers `writing-skills` (`If you didn't watch an agent fail without the skill, you don't know if the skill teaches the right thing`, [SKILL.md:16](/Users/art/.claude/plugins/marketplaces/superpowers-dev/skills/writing-skills/SKILL.md)) — is a **pressure-scenario**: a forcing scenario where the rationalising-shortcut path is tempting and only the rule stops it. You watch a fresh agent **fail** without the rule (RED), then **comply** with the rule loaded (GREEN). The RED→GREEN delta is the rule's liveness evidence.

> **Problem-class match (T16 — do not pattern-match on name).** Superpowers proves a *skill TEACHES*; this prober proves a *manifest rule is LIVE*. **Same mechanism** (baseline-fail → with-doc-comply), **different artifact** (skill doc vs manifest manual rule). Superpowers' own guidance reinforces the fit: «Mechanical constraints (if it's enforceable with regex/validation, automate it — save documentation for judgment calls)» ([writing-skills/SKILL.md:59](/Users/art/.claude/plugins/marketplaces/superpowers-dev/skills/writing-skills/SKILL.md)). Manual rules ARE the judgment calls. This is ADOPT-of-methodology only — **zero** Superpowers dependency is added to the substrate (`grep -E '"superpowers"' package.json` stays empty).

## Input

A manual rule's manifest entry, carrying an entry-level `pressure-scenario` (schema: [rules-manifest.schema.json](../packages/core/manifest/rules-manifest.schema.json)):

```jsonc
"pressure-scenario": {
"baseline-prompt": "<scenario forcing a choice between the rule and a tempting shortcut>",
"observable-failure": "<what RED looks like — behaviour WITHOUT the rule>",
"observable-compliance": "<what GREEN looks like — behaviour WITH the rule; differs from RED>",
"pressure": ["time" | "authority" | "sunk-cost" | "scope-creep", ...]
}
```

Read it from `packages/core/manifest/rules-manifest.json` for the framework repo, or the consumer's installed manifest (`.ai-factory/…/rules-manifest.json`). Confirm the path with the operator if ambiguous.

## Protocol (mirrors Superpowers `subagent-driven-development` two-subagent shape)

For the target manual rule, dispatch **two fresh subagents** — each with isolated context, neither inheriting this session's history (the SDD discipline, [subagent-driven-development/SKILL.md:10](/Users/art/.claude/plugins/marketplaces/superpowers-dev/skills/subagent-driven-development/SKILL.md)):

1. **RED run (baseline, rule absent).** Dispatch a fresh subagent with **only** the `baseline-prompt` as its task — do **not** load the rule's `policy`, `title`, or any hint of the convention. Capture its output verbatim. Classify against `observable-failure`: did it take the shortcut? Document the **exact rationalisation** it used (this is load-bearing — it is what the rule must out-argue).
2. **GREEN run (rule present).** Dispatch a **second** fresh subagent with the `baseline-prompt` **plus** the rule's `policy` text prepended (the «rule loaded» condition). Capture its output. Classify against `observable-compliance`: did the loaded rule flip the behaviour?
3. **Delta.** Report one of:
- **RED→GREEN (LIVE):** baseline matched `observable-failure` AND with-rule matched `observable-compliance`. The rule demonstrably changes behaviour. ✅
- **GREEN→GREEN (INCONCLUSIVE — scenario too weak):** baseline already complied. Per **T-V3-B**, a non-failing baseline does **not** prove the rule unnecessary — it means the scenario is not pressuring hard enough. Recommend strengthening the `baseline-prompt` (add/intensify a declared `pressure`) and re-running. Do **not** report the rule dead.
- **RED→RED (RULE INEFFECTIVE):** with-rule still failed. The rule text does not out-argue the rationalisation. Report the surviving rationalisation; recommend the rule author close the loophole (Superpowers REFACTOR phase).

Run each rule's probe in its own fresh-subagent pair. Never reuse a subagent across RED and GREEN — shared context contaminates the baseline.

## Output (report only)

```text
RULE: <id> (<title>) — pressure applied: <pressure[]>
RED (baseline, rule absent): <verdict vs observable-failure> — rationalisation: "<verbatim>"
GREEN (rule loaded): <verdict vs observable-compliance>
DELTA: RED→GREEN | GREEN→GREEN (weak scenario) | RED→RED (rule ineffective)
EVIDENCE: <2-6 line excerpts of the two subagent outputs>
RECOMMENDATION: <none | strengthen scenario | close rule loophole> — for the rule/scenario AUTHOR, not applied here.
```

## Scope boundaries

- **Runtime-shaped rules (`IR5`, `IR6`).** Their `observable-failure` is runtime behaviour (dropped trace context; no circuit breaker). A text-only baseline subagent emits *code*, not a running trace — so their **behavioral** RED→GREEN demo is **deferred** to a runtime-probe sub-wave. They remain **structurally validated** by the principle-02 manual arm (a real forcing-function scenario exists), **not** un-falsified. Say so explicitly; do not fake a behavioral demo.
- **Never CI.** This protocol dispatches LLM subagents. Wiring it into pre-push/CI violates [no-paid-llm-in-ci.md §1](../.claude/rules/no-paid-llm-in-ci.md). It is session-bound and operator-triggered, by construction.
- **Report, do not fix.** Recommendations go to the rule/scenario author. You do not edit the manifest or commit.

## Recursive self-application (T15)

This prober is itself a manual-rule-shaped discipline — «every manual rule carries a real forcing-function pressure-scenario». Does its **own** pressure-scenario exist? Yes — it lives, in inverted form, as the **principle-02 manual arm** (`02-paired-negative-test.test.ts`): the RED baseline is «a manual rule ships with a placeholder / tautological / pressure-less scenario» (the mutation tests prove the assertion FAILS on each), the GREEN is «a real forcing-function scenario with RED≠GREEN and ≥1 declared pressure» (the positive test passes). The structural gate is the mechanically-checkable shadow of this behavioral protocol — the two halves are the same discipline at two channels (gate for «scenario is well-formed», prober for «scenario actually moves an agent»).

## See also

- [packages/core/principles/02-paired-negative-test.test.ts](../packages/core/principles/02-paired-negative-test.test.ts) — the structural manual arm (mechanical companion to this prober).
- [packages/core/manifest/rules-manifest.schema.json](../packages/core/manifest/rules-manifest.schema.json) — `pressure-scenario` data contract.
- [.claude/rules/no-paid-llm-in-ci.md](../.claude/rules/no-paid-llm-in-ci.md) — why this prober is session-bound, never CI.
- [.claude/rules/rule-enforcement-channel-selection.md](../.claude/rules/rule-enforcement-channel-selection.md) — judgement → injection, not gate: a manual rule cannot be mechanically gated on substance, so a session-read prober is the correct channel.
- [docs/meta-factory/prior-art-evaluations.md#55](../docs/meta-factory/prior-art-evaluations.md) — Superpowers `writing-skills` RED-GREEN-REFACTOR (ADOPTED methodology); [#64](../docs/meta-factory/prior-art-evaluations.md) — `subagent-driven-development` two-subagent shape; [#115](../docs/meta-factory/prior-art-evaluations.md) — this prober's BUILD verdict + search evidence.
1 change: 1 addition & 0 deletions docs/meta-factory/prior-art-evaluations.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ Each entry is a row in the table at §4 below. The row schema:

| 113 | `superpowers:writing-skills` (obra/superpowers) — AI-doc authoring skill bundling Anthropic best-practices, TDD-for-docs, progressive disclosure, and automate-vs-document boundary guidance. Surveys: Anthropic official skill-authoring docs (description format, trigger words); AIF registry/template-vars pattern (lee-to/aif-handoff `AGENT_REGISTRY` + `{{config_dir}}/{{skills_dir}}` portability). | Project-specific AI-doc authoring standard (`.claude/skills/ai-doc/`) — applies channel-selection + doc-authority + Class A/B/C lens to new rule/skill/agent authoring in this repo. Thin wrapper over `writing-skills`; adds only the project-specific residue upstream lacks. | 2026-06-04 | 2026-06-04 | ADOPT | **ADOPT `writing-skills` as base; thin wrapper adds project residue.** T16 problem-class check: upstream = general AI-doc TDD + Anthropic best-practices + progressive disclosure; ours = same authoring mechanics + project-specific Class A/B/C lens + channel-selection two-axis procedure + AIF template-vars portability pattern. Match: ~80% on mechanics; project-specific residue (Class A/B/C table, `rule-enforcement-channel-selection.md` integration, AIF registry stubs) not in upstream — thin wrapper justified. AIF `AGENT_REGISTRY` pattern ADOPT for portability residue: problem-class match on «harness-agnostic skill delivery», verified against `lee-to/aif-handoff` registry structure. Zero new code, zero npm deps. | Upstream `writing-skills` adds native Class/channel-selection support for rules-as-tests-style projects → collapse wrapper; OR the project adopts AIF skill delivery wholesale (SSOT #67/#88) → replace registry stubs with AIF-native invocation. |
| 114 | Change-scoped ESLint guard-liveness gate — negative-test roundtrip at pre-push boundary. Surveys: (1) `eslint.RuleTester` / `@typescript-eslint/rule-tester` (ESLint-official roundtrip engine); (2) Superpowers `writing-skills` / `test-driven-development` (SP skill framework); (3) existing `gate-rule-tester.ts` (own-stack, L4 Gate 2). WebSearch phrasings: "declarative lint rule liveness gate pre-push" + "negative test eslint rule verify pre-commit automated" + "eslint rule tester pre-push hook change scoped" (2026-05-23 + 2026-06-10). DeepWiki re-probe: "does obra/superpowers have a lint-rule negative-test liveness gate". | Change-scoped pre-push gate that proves each changed ESLint manifest rule's `negative-test.input[]` actually trips the rule — every bypass variant — and `examples.good` stays clean. Gate wrapper over the ADOPTED `eslint.Linter` engine via `gate-rule-tester.ts` roundtrip logic. | 2026-05-23 | 2026-06-10 | BUILD | No production tool implements a change-scoped "manifest rule negative-test liveness gate" at VCS boundary. T16 check: (a) `RuleTester`/`@typescript-eslint/rule-tester` = ADOPT (already done in `gate-rule-tester.ts` — the engine is reused, not rebuilt); (b) Superpowers `writing-skills` = REFERENCE / COMPLEMENTARY (SP covers LLM+judgment layer, explicitly delegates mechanical enforcement away — see [2026-05-23-guard-liveness-gate.md §2](research-patches/2026-05-23-guard-liveness-gate.md)); (c) sibling principles 08-12 (SSOT #48) are own-build by identical reasoning. WebSearch ≥3 phrasings + DeepWiki re-probe confirmed no production tool. Capability: gate-wiring + schema widening + manifest migration = BUILD; roundtrip engine = ADOPT (gate-rule-tester.ts). Zero new npm deps; ESLint engine already installed. | Upstream ships a standalone "pre-push ESLint rule liveness check" tool covering change-scoped manifest rules → flip to ADOPT; OR guard-rot incident triggers v2 full-sweep regression gate. |
| 115 | Manual-rule liveness prober — session-bound RED→GREEN `pressure-scenario` probe for `check.type === 'manual'` manifest rules (guard-liveness v3, the manual half of #114). Surveys: Superpowers `writing-skills` RED-GREEN-REFACTOR (#55) + `subagent-driven-development` two-subagent shape (#64), re-verified against installed v5.1.0 (`writing-skills/SKILL.md:14,16,36-37,59`); DeepWiki re-probe `obra/superpowers` (RED-GREEN methodology reusable for arbitrary rules/conventions, but shipped as an *authoring* methodology, not a per-manifest-rule prober) + `Aider-AI/aider` (`conventions.md` only *demonstrates* with/without ablation; no reusable prober); WebSearch ≥3 phrasings 2026-06-13 ("LLM agent compliance probe pressure scenario baseline vs with-rule" + "testing whether coding-agent rule actually changes behavior" + "eval framework prove guideline causes compliance ablation with/without"). | L-manual-rule liveness — an AI-agnostic, session-bound prober (`agents/manual-rule-liveness-prober.md`) that dispatches a fresh subagent into a manual rule's `pressure-scenario` baseline-prompt twice (rule absent → expect `observable-failure`/RED; rule loaded → expect `observable-compliance`/GREEN) and reports the delta. Methodology ADOPTED from SP #55; the prober + entry-level `pressure-scenario` schema wiring + principle-02 manual arm are ours. | 2026-06-13 | 2026-06-13 | BUILD | No production tool packages a per-manifest-rule judgement-rule liveness prober. **T16 problem-class:** SP `writing-skills` (#55) = ADOPT-methodology (its RED-GREEN proves a skill *teaches*; ours proves a manifest rule is *live* — same mechanism baseline-fail→with-doc-comply, different artifact); SP SDD (#64) = REFERENCE (two-subagent dispatch shape). DeepWiki confirmed SP ships this as a skill/methodology (incl. testing `CLAUDE.md` variants) but **not** a reusable prober reading a manifest field; aider's `conventions.md` only documents a with/without example. WebSearch surfaced research-grade ablation (`arxiv:2506.02357` "Principle ON/OFF" adherence benchmark; PrivaCI-Bench; ETH "Evaluating AGENTS.md"; promptfoo evaluate-coding-agents) = **REFERENCE** — they validate baseline-vs-with-rule comparison as a recognised technique, none is a drop-in per-rule prober. **Substrate-pure:** zero Superpowers npm dep (`grep -E '"superpowers"' package.json` empty). Session-bound, never CI ([no-paid-llm-in-ci.md §1](../../.claude/rules/no-paid-llm-in-ci.md)). | Superpowers (or any tool) ships a reusable per-rule/per-convention liveness prober that consumes a manifest field → flip BUILD→ADOPT; OR maintainer authorises a runtime-probe sub-wave giving `IR5`/`IR6` a behavioral demo; OR a manual-rule-rot incident motivates a v2 sweep. |

---

Expand Down
1 change: 1 addition & 0 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ SHIPPED_DOCS=(
"agents/living-docs-auditor.md"
"agents/compliance-verifier.md"
"agents/memory-codification-auditor.md"
"agents/manual-rule-liveness-prober.md"
"skills/tool-bootstrapping/SKILL.md"
"skills/tool-bootstrapping/references/decision-format.md"
)
Expand Down
21 changes: 21 additions & 0 deletions packages/core/hooks/checks/guard-liveness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,31 @@ export interface ManifestNegativeTest {
eslintRuleConfig?: unknown;
}

/**
* Pressure type a manual-rule scenario applies (T-V3-A: a pressure-scenario is a
* forcing function, not merely a violating example).
*/
export type PressureType = 'time' | 'authority' | 'sunk-cost' | 'scope-creep';

/**
* Liveness data for manual (judgement-type) rules — the entry-level analog of
* negative-test for rules with no executable input. Consumed by the session-bound
* manual-rule-liveness-prober (agents/manual-rule-liveness-prober.md), never CI.
* ADOPTED methodology: Superpowers writing-skills RED-GREEN pressure-scenario
* (prior-art-evaluations.md#55).
*/
export interface ManifestPressureScenario {
'baseline-prompt': string;
'observable-failure': string;
'observable-compliance': string;
pressure: PressureType[];
}

export interface ManifestRule {
check: { type: string; rule?: string };
examples: { bad: string; good: string };
'negative-test'?: ManifestNegativeTest;
'pressure-scenario'?: ManifestPressureScenario;
}

export type RuleLivenessStatus = 'pass' | 'fail' | 'skipped' | 'no-data' | 'n/a';
Expand Down
Loading