Skip to content

feat(pr-workflow): add PR-audit, reasoning-audit, and diagnostic skills - #106

Draft
MajorLift wants to merge 4 commits into
mainfrom
jongsun/add/audit-skills
Draft

feat(pr-workflow): add PR-audit, reasoning-audit, and diagnostic skills#106
MajorLift wants to merge 4 commits into
mainfrom
jongsun/add/audit-skills

Conversation

@MajorLift

@MajorLift MajorLift commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Adds eleven skills. Each encodes a failure mode observed in real validation runs, stated concretely enough to judge the guidance against the mechanism rather than the assertion.

pr-workflow — eight, and testing — one

skill the failure it encodes
falsifiers-first Falsifiers taken from the author's sentences probe the mechanisms the author already phrased clearly. The probe lands on the most legible line rather than the weakest, the method mirrors the author's so agreement carries little information, and there is no bucket for effects nobody claimed.
unintended-breakage A change breaks something it never mentions, and the diff looks intentional all the way through. Nine surfaces, each with a mechanical check; a candidate without a check is not reportable.
silent-failure A mechanism fails and nothing says so. Detectability is a property separate from correctness and is almost never tested, so the method induces the failure and watches for a signal that never arrives.
unmeasured-join A conclusion assembled from individually true facts, joined by a step nobody measured. The facts check out, so the join reads as prose and goes unchecked — and the conclusion is often correct, which is what lets it survive review.
instrument-check awk -v escape-processes its assignment, so a mutation runner wrote /^[sS]{1,4096}$/u where /^[\s\S]{1,4096}$/u was requested, narrowing a regex meant to be widened. A different test failed than the one targeted, the suite ran in full, every guard passed, and the run reported a falsification for a line it never wrote. The artifact echoed the requested mutation, so the two could not disagree.
scope-of-search A grep keyed on the variable name searchParams reported (no occurrence) directly above its own output showing five params.get('utm_*') reads. A negative is a fact about the pattern, not about the tree.
coverage-partition "The test has power" is a boolean; power is a distribution. Three mutations against one suite: 4 of 7 cases guard the parameter strip, 2 guard signature verification, 1 guards the value format. Five pass with verification entirely disabled.
selection-audit Two counts matched and were read as corroboration. They were different selection rules landing on the same number — one selecting changed test files, the other a set roughly twice as large.

| red-on-base | A test that passes on the branch proves the branch is green. Only one that fails on base and passes on branch proves the change is connected to the reported bug — and a base failure for the wrong reason (missing import, absent fixture, unrelated red) produces an identical exit code and proves nothing. Moved from #84; renamed from falsifying-test. |

coding — two

skill the failure it encodes
distinguishing-observation Instrumenting the mechanism you already suspect yields evidence consistent with your hypothesis and equally consistent with the ones you never wrote down. Names the pairs indistinguishable from outside — a swallowed error and an unreached path, a cache hit and a correct recomputation.
observability-gap Debugging a path before establishing what signal exists on it. Distinguishes absent from suppressed, since a log filtered by level, sample rate or a flag is a different problem from one never written.

Both sit beside flaky-test-detection and pair with the debug orchestrator in #98.

The decision that needs a call

falsifiers-first and unintended-breakage read the PR description in opposite orders, which is the substance rather than a style choice:

skill question description
falsifiers-first does the change do what it claims? sealed until the hypotheses are fixed
unintended-breakage what does it do that it was not meant to? read first, to fix the envelope

Sealing prevents the claims from steering which mechanisms get probed. Reading first is required because out of scope is not a property of code — it is a relation between code and a stated intent, and you cannot detect a departure from an envelope you have not read. Each skill names the other and states why the order differs, so neither gets harmonised into the other later.

Three properties worth knowing before reading the diff

The unclaimed bucket falsifiers-first sorts results into supported / contradicted / unclaimed. The third is the product, and a description-led run cannot produce it. It catches undersold changes as well as oversold ones.
The inverted verdict In silent-failure, a green suite under an induced failure means the failure is undetectable — the opposite reading from a mutation probe, where green means the test is vacuous. The artifact has to say which reading applies, because the output is identical.
Ranking by distance to discovery Not by severity. A small error nobody can detect outranks a large one that pages someone in a minute.

Reviewing this

Merge #84 first Seven of these link evidence, which ships there and is not on main. The relative links resolve on merge and are dead if this lands first. Alternative: drop the links, add them in a follow-up.
It splits cleanly Along the groups above — say so and I will split it.
Weakest guidance falsifiers-first's phase A dominates cost on a large diff and the skill only says to scope by mechanism, it does not solve it. Sealing is imperfect — CI check names, branch names and commit messages leak the author's framing — so the skill says to record what leaked rather than claim it did not.

Test plan

  • node .github/scripts/lint-skill-entry.mjs — 0 errors across 58 skills
  • No wiki-link syntax, no bare lane identifiers in any description
  • Every ## Related link resolves on this branch, except the six forward references to evidence
  • Reviewer check: does each skill name a failure specific enough to test the guidance against, or does any read as generic advice

`falsifiers-first` seals the description until its hypotheses are fixed. A run
that takes its falsifiers from the author's sentences probes the mechanisms the
author already phrased clearly, mirrors their method so agreement carries no
information, and has no bucket for effects nobody claimed — which is where the
findings worth having usually sit.

`unintended-breakage` reads the description first, because out-of-scope is a
relation between code and a stated intent rather than a property of code. It
fixes the envelope, enumerates effects against it, and requires a mechanical
check per candidate: an export removed still has importers, a locale key still
has lookups, a renamed event still has a dashboard. Its most valuable tier is the
one nothing fails on — renamed events, flipped defaults, changed state shapes.

Each names the other and says why the order differs, so neither gets harmonised
into the other later.
Asks whether a mechanism announces its own failure, which is a separate property
from whether it can fail and is almost never tested. Since a silent failure
cannot be observed, the method induces it and watches for a signal that never
arrives — and the verdict reads opposite to a normal mutation probe, where a
green suite means a vacuous test. Here green means the failure is undetectable,
which is the result.

Findings rank by distance to discovery rather than by severity: a small error
nobody can see outranks a large one that pages someone in a minute. Half the
listed shapes are instrument failures — a name that does not match what is
counted, a pattern that cannot match what it searches for — because a
measurement that fails quietly gets published.
Each encodes a failure that actually shipped, rather than a principle that
sounded right. `unmeasured-join` targets the costliest one — a conclusion
assembled from true facts through a step nobody measured, which survives review
because the facts check out and the join reads as prose. `instrument-check`
requires a positive and a negative control before a measurement counts, after a
mutation runner reported a falsification for a line it never wrote.
`scope-of-search` makes a negative carry the pattern that produced it, after a
grep keyed on a variable name declared absent what its own output printed two
blocks later. `coverage-partition` replaces "the test has power" with which cases
guard which mechanism. `selection-audit` separates a count from the rule that
produced it, after two different selections landed on the same number and the
match closed the question.

`distinguishing-observation` and `observability-gap` point the same discipline at
debugging: design the observation that separates candidates rather than confirms
the favourite, and establish what signal exists on a path before reading more of
it.

The `evidence` links are forward references — that skill ships in #84 and is not
on main yet.
@MajorLift
MajorLift marked this pull request as draft August 3, 2026 10:11
MajorLift added a commit that referenced this pull request Aug 3, 2026
Every other engine `evidence` names ships in its own pull request —
`memory-leak`, `race-condition-repro`, `supply-chain-audit`,
`react-render-delta`, `agent-run-cost`. This one was the sole exception, and the
reason was authoring order rather than design: it was written alongside the
orchestrator before the split between the instrument and the reasoning that
points it existed as a concept.

It lands in #106 with the other reasoning skills, whose substance is the same
kind — what counts as proof, and how a proof can look right while testing the
wrong thing. What stays here is the machinery: the runners, the run workflow, the
gate, the hooks.

The B3 engine cell now names a skill that arrives in #106, which is a dangling
name in a table rather than a broken link, and resolves whichever order the two
merge.
Its substance is the same as everything else here: what counts as proof, and how
a proof can look right while testing the wrong thing. Its falsifier — a base
commit that fails for the wrong reason produces an identical exit code and proves
nothing — is the same move `unmeasured-join` and `scope-of-search` make on
different material.

It sat in #84 because it was written alongside the orchestrator, before the
instrument and the reasoning that points it were separate ideas. Every other
engine already ships in its own pull request.

Renamed from `falsifying-test` before the move, because it and `falsify-probe.sh`
shared a root while answering different questions — base-against-branch asks
whether a test is connected to the reported bug, one-commit-with-a-mutated-line
asks whether a suite notices the mechanism going away.
@MajorLift MajorLift changed the title feat(pr-workflow): add PR-audit and reasoning-audit skills, plus two diagnostic siblings feat(pr-workflow): add PR-audit, reasoning-audit, and diagnostic skills Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant