feat(pr-workflow): add PR-audit, reasoning-audit, and diagnostic skills - #106
Draft
MajorLift wants to merge 4 commits into
Draft
feat(pr-workflow): add PR-audit, reasoning-audit, and diagnostic skills#106MajorLift wants to merge 4 commits into
MajorLift wants to merge 4 commits into
Conversation
`falsifiers-first` seals the description until its hypotheses are fixed. A run that takes its falsifiers from the author's sentences probes the mechanisms the author already phrased clearly, mirrors their method so agreement carries no information, and has no bucket for effects nobody claimed — which is where the findings worth having usually sit. `unintended-breakage` reads the description first, because out-of-scope is a relation between code and a stated intent rather than a property of code. It fixes the envelope, enumerates effects against it, and requires a mechanical check per candidate: an export removed still has importers, a locale key still has lookups, a renamed event still has a dashboard. Its most valuable tier is the one nothing fails on — renamed events, flipped defaults, changed state shapes. Each names the other and says why the order differs, so neither gets harmonised into the other later.
Asks whether a mechanism announces its own failure, which is a separate property from whether it can fail and is almost never tested. Since a silent failure cannot be observed, the method induces it and watches for a signal that never arrives — and the verdict reads opposite to a normal mutation probe, where a green suite means a vacuous test. Here green means the failure is undetectable, which is the result. Findings rank by distance to discovery rather than by severity: a small error nobody can see outranks a large one that pages someone in a minute. Half the listed shapes are instrument failures — a name that does not match what is counted, a pattern that cannot match what it searches for — because a measurement that fails quietly gets published.
Each encodes a failure that actually shipped, rather than a principle that sounded right. `unmeasured-join` targets the costliest one — a conclusion assembled from true facts through a step nobody measured, which survives review because the facts check out and the join reads as prose. `instrument-check` requires a positive and a negative control before a measurement counts, after a mutation runner reported a falsification for a line it never wrote. `scope-of-search` makes a negative carry the pattern that produced it, after a grep keyed on a variable name declared absent what its own output printed two blocks later. `coverage-partition` replaces "the test has power" with which cases guard which mechanism. `selection-audit` separates a count from the rule that produced it, after two different selections landed on the same number and the match closed the question. `distinguishing-observation` and `observability-gap` point the same discipline at debugging: design the observation that separates candidates rather than confirms the favourite, and establish what signal exists on a path before reading more of it. The `evidence` links are forward references — that skill ships in #84 and is not on main yet.
This was referenced Aug 3, 2026
MajorLift
marked this pull request as draft
August 3, 2026 10:11
MajorLift
added a commit
that referenced
this pull request
Aug 3, 2026
Every other engine `evidence` names ships in its own pull request — `memory-leak`, `race-condition-repro`, `supply-chain-audit`, `react-render-delta`, `agent-run-cost`. This one was the sole exception, and the reason was authoring order rather than design: it was written alongside the orchestrator before the split between the instrument and the reasoning that points it existed as a concept. It lands in #106 with the other reasoning skills, whose substance is the same kind — what counts as proof, and how a proof can look right while testing the wrong thing. What stays here is the machinery: the runners, the run workflow, the gate, the hooks. The B3 engine cell now names a skill that arrives in #106, which is a dangling name in a table rather than a broken link, and resolves whichever order the two merge.
Its substance is the same as everything else here: what counts as proof, and how a proof can look right while testing the wrong thing. Its falsifier — a base commit that fails for the wrong reason produces an identical exit code and proves nothing — is the same move `unmeasured-join` and `scope-of-search` make on different material. It sat in #84 because it was written alongside the orchestrator, before the instrument and the reasoning that points it were separate ideas. Every other engine already ships in its own pull request. Renamed from `falsifying-test` before the move, because it and `falsify-probe.sh` shared a root while answering different questions — base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether a suite notices the mechanism going away.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds eleven skills. Each encodes a failure mode observed in real validation runs, stated concretely enough to judge the guidance against the mechanism rather than the assertion.
pr-workflow— eight, andtesting— onefalsifiers-firstunintended-breakagesilent-failureunmeasured-joininstrument-checkawk -vescape-processes its assignment, so a mutation runner wrote/^[sS]{1,4096}$/uwhere/^[\s\S]{1,4096}$/uwas requested, narrowing a regex meant to be widened. A different test failed than the one targeted, the suite ran in full, every guard passed, and the run reported a falsification for a line it never wrote. The artifact echoed the requested mutation, so the two could not disagree.scope-of-searchsearchParamsreported(no occurrence)directly above its own output showing fiveparams.get('utm_*')reads. A negative is a fact about the pattern, not about the tree.coverage-partitionselection-audit|
red-on-base| A test that passes on the branch proves the branch is green. Only one that fails on base and passes on branch proves the change is connected to the reported bug — and a base failure for the wrong reason (missing import, absent fixture, unrelated red) produces an identical exit code and proves nothing. Moved from #84; renamed fromfalsifying-test. |coding— twodistinguishing-observationobservability-gapBoth sit beside
flaky-test-detectionand pair with thedebugorchestrator in #98.The decision that needs a call
falsifiers-firstandunintended-breakageread the PR description in opposite orders, which is the substance rather than a style choice:falsifiers-firstunintended-breakageSealing prevents the claims from steering which mechanisms get probed. Reading first is required because out of scope is not a property of code — it is a relation between code and a stated intent, and you cannot detect a departure from an envelope you have not read. Each skill names the other and states why the order differs, so neither gets harmonised into the other later.
Three properties worth knowing before reading the diff
falsifiers-firstsorts results into supported / contradicted / unclaimed. The third is the product, and a description-led run cannot produce it. It catches undersold changes as well as oversold ones.silent-failure, a green suite under an induced failure means the failure is undetectable — the opposite reading from a mutation probe, where green means the test is vacuous. The artifact has to say which reading applies, because the output is identical.Reviewing this
evidence, which ships there and is not onmain. The relative links resolve on merge and are dead if this lands first. Alternative: drop the links, add them in a follow-up.falsifiers-first's phase A dominates cost on a large diff and the skill only says to scope by mechanism, it does not solve it. Sealing is imperfect — CI check names, branch names and commit messages leak the author's framing — so the skill says to record what leaked rather than claim it did not.Test plan
node .github/scripts/lint-skill-entry.mjs— 0 errors across 58 skills## Relatedlink resolves on this branch, except the six forward references toevidence