research(slow-test-triage): R-phase audit — per-test triage + 3 maintainer DNs - #231
Merged
Conversation
…mendations
Measured pre-push test surfaces A (audit-ai-docs.test.ts) and B (test:principles)
twice each with --reporter=verbose. Found 7 tests ≥1000ms; all fire on every push.
Root causes:
- Group R4 (3 tests): execSync('npx --version') startup cost — warm cache 1.3s,
cold cache ~24s (explains the testTimeout escalation in PR #229).
- Group P11 (2 tests): git-log scan per capability artifact, O(n) git calls.
- Group P14 (1 test): spawnSync('bash', [check-skill-drift.sh]).
- Group P05 (1 test): execSync('npx tsx render-rules.ts --check').
Recommendations: Option 1 or Option 3-via-injection for Group R4 (DN-1);
Option 4 (leave-as-is) for P11/P14/P05 with re-evaluation trigger for P11.
§1.7: forward-check applied — vitest.config.ts:17-18 referenced (slow-test
origin comment pointing to this kickoff); audit-ai-docs.ts:155-176 cited
(probeR4 execSync root cause); backward-check sweep — no artefact superseded,
purely additive research-patch.
Prior-art: skipped — research-patch only, no new capability introduced
Merged
7 tasks
artyhoo
added a commit
that referenced
this pull request
Jul 31, 2026
…mbrella + S-A kickoffs (#1189) * docs(handoff): /arch v2 + context-pipeline session handoff — decisions, in-flight aif tasks, continuation protocol Prior-art: skipped — session handoff document only, no new capability; decisions it records cite their own SSOT rows (#231, #207, #179, #64). * docs(spec): /arch v2 + context-pipeline system design — layer model, pipeline arc, ADR-1..8 Step 5 of the 2026-07-31 handoff §4 protocol. Fable design authored on the Opus research distillate (spot-checked, freshness-barred) and the Opus cold critique (GO-WITH-PATCHES). All three critique blockers absorbed by re-derivation: #231 over-read retracted (ADR-5), 5/5-K1 incident count corrected to 2/5 and the primary/background split dropped (ADR-6), the calibration falsifier given an oracle via shadow-A/B + pre-declared threshold (ADR-5). M1-M7 absorbed as design constraints (population table, bounded drill-down + distillate K-pass, K6 candidate/adjudicate split, gate-channel re-route to pre-push/CI, operationalized bet falsifier, L1/L2 boundary re-drawn, option spaces spanned). Prior-art: skipped — design spec only, no new capability shipped * docs(arch-v2): umbrella kickoff S-A..S-F + S-A stage-scoped dispatch input Execution plan for the /arch v2 + context-pipeline track, derived from the 2026-07-31 design spec (ADR-1..8) by the Opus plan-writing seat. Umbrella kickoff: stage table S-A..S-F with per-stage scope, dependencies, tier classification (justified against CLAUDE.md's fixed criteria), acceptance and implemented ADRs; dispatch protocol (4-arm in-flight probe, Phase -1 cold review, bridge-profile marker rule with the fidelity-verdict precondition quoted verbatim and re-verified at dispatch); calibration-ledger bootstrap (ADR-5/6/8) with the ADR-8 token instrument named; cross-umbrella dependency on token-audit S1 (S-E only, two gates: merged AND content-read). The bottom seat + shadow-A/B station is marked active from S-B merge onward — S-A predates the contract implementation and is covered only by Phase -1 plus its own acceptance commands. Stated, not papered over. Plan-writer objections (§4, per «who must write the plan cannot rubber-stamp the design»): O-1 three wrapper drifts, not two, and one mis-described — upstream ships brainstorming/spec-document-reviewer-prompt.md in 5.1.0/6.1.1/ 6.2.0, night-mode:15's SDD roster does not match upstream, night-mode:29 cites stale upstream line numbers; O-2 the skill-exists-by-name smoke catches none of them and skips silently off-host; O-3 ADR-8's token metric had no named instrument (aif task tokenTotal/costUsd, verified live); O-4 the ledger principle test is vacuous before 5 rows; O-5 S-D's tier is a function of S-C's verdict; O-6 the spec's marker condition drops CLAUDE.md's «produced by /arch». S-A kickoff is stage-scoped and self-contained (handoff decision 11): W1-W6 with concrete file targets and a verification command each, host-verify contract, descopes, §1.7 obligation in enumeration format, T-enumeration plus three domain traps. Prior-art: skipped — kickoff/plan docs only, no new capability * docs(arch-v2): preserve track evidence artifacts — distillate, corrected idea, cold critique The design spec cites these three as its evidence chain (distillate → corrected idea → GO-WITH-PATCHES critique); they lived only in the session scratchpad under /private/tmp, which does not survive a reboot. Committed verbatim as session artifacts of the 2026-07-31 protocol run. Prior-art: skipped — evidence-record docs only, no new capability * docs(spec): point evidence-chain citations at the committed artifact files Prior-art: skipped — link fix in a design doc, no new capability * docs(spec): absorb token-audit S1 acceptance — fresh N2 numbers, ADR-3 falsifier fired S1 (task c781e8a9, accepted 2026-07-31) measured the repo-owned always-on set at 29-39% of the observed ~100k session-start total, firing ADR-3's pre-registered falsifier: the budget gate's asserted quantity is re-scoped to the repo-owned share (explicitly labelled), the harness remainder routes to settings-recommendations, and the InstructionsLoaded verification task doubles as the measurement-extension probe. N2 updated to the fresher script-reproducible per-environment numbers (140,216 B host vs 118,374 B container), replacing the older channel-level A7 pair. Prior-art: skipped — design-doc correction on fresh measurement, no new capability --------- Co-authored-by: Test <test@example.com>
This was referenced Jul 31, 2026
artyhoo
added a commit
that referenced
this pull request
Aug 1, 2026
…SOT #233 (#1197) * docs(arch-v2-s-c): S-C stage — L2 channel verdict (DEFER/null) + SSOT #233 arch-v2-context-pipeline S-C: BFR-disciplined verdict over the 5-option L2 channel space, on top of ADR-2's 4-row population table precondition. Verdict: DEFER the L2 build; null option adopted. Five independent lines of evidence converge — need not established (C10), no single option reaches all 4 population rows, the aif runtime already does partial per-role ambient shaping via effectiveSystemPromptAppend (subagentQuery.ts:681-684), the external evidence points subtractive (Anthropic 2026-07-24 80%-removal; obra/superpowers retired review loop per SSOT #231), and the falsifier is concrete and operational. R6 /app/ re-check: VERIFIED (not INCONCLUSIVE — plan's binding fact was reasonably inferred from "docker unavailable" but /app/ is directly readable; destination-environment-verification.md §3 governs container→HOST suite re-runs, not /app/ file reads). All six anchor sites match verbatim. BFR §3 mechanism run, not asserted: SSOT consult (#101/#228/#229/#231) + DeepWiki ×3 phrasings + WebSearch ×3 phrasings; context7 excluded per kickoff (problem-class, not library API). §1.7 backward-check delegated to agents/backward-sweep-auditor.md per T21 binding — cold agent enumerated 14 in-repo sibling surfaces + 2 NOT-IN-REPO, all SWEPT-CLEAN, 0 GAP-FOUND. Surface list ≠ diff's own files (conformant). S-D tier assignment: on the null verdict, S-D becomes a Tier-1 L2-closure PR (retirement note + done.md, no build). Marker decision recorded in §7 of the research-patch. Acceptance (kickoff §2): - population table complete (4 rows, file:line evidence per cell) ✓ - 5 options adjudicated; null argued ✓ - BFR mechanism run with queries + results quoted ✓ - T16 problem-class match per ADOPT/ADAPT ✓ - research-patch exists; SSOT row appended; principle 08/10/12 GREEN ✓ - S-D tier in PR body (§7 of patch) ✓ - §1.7 forward + backward ✓ - Zero build: no hook/resolver/skill/rule in diff ✓ - host-verify 3/3 PASS on Linux ✓ Prior-art: prior-art-evaluations.md#233 (this commit registers the L2 channel verdict capability class — row 233 IS the evaluation; no separate code artefact, markdown-only). * fix(arch-v2-s-c): renumber SSOT row 233 → 234 — ID collision with RTK (#1204) This branch and token-economy stage B (#1204, merged first) both claimed SSOT ID 233. prior-art-evaluations.md is an append-only register keyed by ID; principle 08 verifies that a cited ID EXISTS but not that it is UNIQUE, so both PRs were green in isolation and the duplicate would have been born at merge. RTK keeps 233 (first to land on staging); this branch's per-role-L2 row becomes 234, and the l2-channel-verdict patch's "Next free ID" note moves with it. Merged origin/staging in (merge-forward, never rebase a published branch) so both rows now coexist in this branch's copy of the register. The class-level fix (uniqueness check in principle 08) ships separately on the token-economy session branch — one concern per PR. Prior-art: skipped — ID renumbering + merge-forward bookkeeping, no new capability. --------- Co-authored-by: Test <test@example.com>
8 tasks
artyhoo
added a commit
that referenced
this pull request
Aug 10, 2026
…provenance repair (principle-11 F1 staging red) (#1375) ## Summary Two concerns, one invited scope: (1) NEW project skill `.claude/skills/reviewer/SKILL.md` — the interactive review-session protocol, until now living only in the operator's personal `~/.claude/commands/reviewer.md` where repo machinery cannot see or update it (the #1374 severity-contract change had to be hand-patched into it the same day — the incident this closes; in-repo, a project skill takes precedence over the same-named personal command per the documented skill-over-command rule). (2) SSOT entry #249 — provenance repair for `.claude/rules/effort-worthiness.md`: principle 11 F1 is red on staging because the #1374 squash rebuilt the introducing commit from a PR body that omitted the `Prior-art:` line (the #1094→#1097 class); the verbatim-path SSOT match fixes F1 for every subsequent PR. ## Changes - NEW `.claude/skills/reviewer/SKILL.md`: three modes + verification-vs-synthesis economy split adapted 1:1 from the operator command; verdict grammar bound to `reviewer-discipline.md` §6 (Failure-scenario, ESCALATED, notes lane, zero-finding legitimacy); explicit subordination to the cold agents it does not replace and an explicit not-a-registry-role note (seat-lifecycle.md §1 three-roles cut respected — no seat-lifecycle edit). - `.claude/skills/arch/SKILL.md:91`: stale «pending as of 2026-08-10» claim about the operator's global `/reviewer` resolved (hand-apply done same day; in-repo invocations now load the project skill). - `docs/meta-factory/prior-art-evaluations.md`: entry #249 (REFERENCE) — registers the §8 item-5 per-family consult (Conventional Comments, Google eng-practices, Bezos Type-1/2, CBR, WIP limits, ADR/spec-kit/Kiro) already folded into the rule's §4, and records the detector-disagreement root cause (pr-body-prior-art's diff detector calls new rule-markdown non-capability while F1 counts it) with a widening trigger. - Deliberately NOT changed: `setup.d/10-skills.sh` — consumer delivery of the reviewer skill is routed to advisor-pattern §8 item 9 (consumer-delivery stage with its own review), with an env+ tier recommendation (pairs with arch/pipeline at that profile). ## Prior-art consult Prior-art: skipped — project-internal interactive-review skill adapting the operator's own global /reviewer command into the repo; subordinates to reviewer-discipline.md §6 (severity contract SSOT) and to the cold agents it does not replace; no new capability, no packages/ code - [x] PR range is non-capability (one new skill markdown + one SSOT row + one line edit; zero `packages/` files, zero dependency changes); the trailer line above is carried in the PR body so it survives the squash into the introducing commit (the exact #1094→#1097 / #1374 lesson this PR also repairs). - [x] SSOT touch: entry #249 appended (append-only register; capability-commit-author write access per the ownership table). - [x] Cold `agents/capability-reuse-auditor.md` pass run before handoff (source-before-shape Layer B): verdict THIN-ADAPT → GO, 12-candidate overlap set, trailer↔body consistent per clause; its one notes-lane finding (sibling §6 digest cross-pointer) applied in-branch. ## Test plan - [x] §1.7-свод lands in squash-body (`gh pr merge --squash --body "$(gh pr view <N> --json body -q .body)"`) - [x] `npm run --prefix packages/core test:principles` — 41 files green locally after #249 (principle 11 F1 was the one red: 14/14 after; principles 09/14/15 cover the new skill dynamically, 46/46) - [x] `bash scripts/check-skill-drift.sh` — PASS (0 errors); doc-authority hook smoke on the new file — exit 0 - [x] Pre-push full substance sweep green at push time - [x] Manual smoke: `.claude/skills/reviewer/SKILL.md` paired-negative sections present (`## Without this skill` / `## With this skill`); frontmatter description carries concrete RU+EN triggers per skill-description-quality.md §2 ## Provenance n/a — dialog-invited repo-skill addition + CI-red repair; no stage kickoff, no dispatch substrate. ## Review findings Cold capability-reuse-audit (agents/capability-reuse-auditor.md, dialogue-blind on the just-authored file + intended trailer): verdict THIN-ADAPT, overall GO. Overlap set = 12 candidates (SSOT #64/#231/#236/#249 + aif-review family; skills/agents siblings; upstream Superpowers requesting-code-review + SDD). Key clears: the PR #858 class does not recur (the SDD executor/dual-reviewer loop is not re-described; spawning subagents without approval is forbidden by the skill's Hard bounds); subordination lines verified per file:line for all four owners named in the header. One notes-lane finding, fixed same round per the §6 contract: the repo now holds two §6 digests (this skill + agents/reviewer-discipline.md) — a cross-pointer with a drift rule was added to the skill's See also. Coverage honestly partial (headers-only for the non-overlapping skill/agent tail; effort-worthiness.md body unread by the auditor). ## Fidelity verdict FIDELITY: skipped — dialog-invited non-stage PR (repo-skill addition + staging F1 repair); no kickoff/spec basis to audit against; cold reuse-audit GO recorded under Review findings. ## Parked questions - Consumer tier for the reviewer skill: recommendation env+ (contour surface, pairs with arch/pipeline in setup.d/10-skills.sh:120-122); decision + consumer-generic rewording (the skill's Origin references the operator's home path) belong to advisor-pattern §8 item 9 — recorded in the memory card as an item-9 input. - SSOT #249's «Trigger to revisit» carries the detector-widening trigger: a second squash-trailer F1 incident → widen pr-body-prior-art's detector to F1's artifact classes (rule/skill/agent markdown). ## §1.7 Self-discipline check (REQUIRED if PR touches discipline-bearing files) ### §1.7 Forward-check applied New skill checked against every active layer: channel selection per rule-enforcement-channel-selection.md — on-demand skill load at the review-ask trigger, not always-on (description triggers per skill-description-quality.md §2, RU+EN); doc-authority header present with subordination lines (principle 09 dynamic skill enumerator green, packages/core/principles/09-doc-authority-hierarchy.test.ts); paired-negative sections present per packages/core/principles/15-skill-paired-negative.test.ts:50; provenance per principle 11 F1 — the `Prior-art:` trailer lives in this PR body (squash-safe) AND the artifact has SSOT keyword coverage, while the F1 red this PR repairs is closed by the verbatim path in docs/meta-factory/prior-art-evaluations.md entry #249; language-discipline — repo artifact in English; seat-lifecycle NOT extended — the three-roles cut at .claude/rules/seat-lifecycle.md:42 is respected by an explicit not-a-registry-role subordination line instead of a paths: edit; source-before-shape §1 — SSOT + .claude/skills/ + agents/ grepped before the body was written, and Layer B (agents/capability-reuse-auditor.md) run before handoff with verdict THIN-ADAPT/GO. ### §1.7 Backward-check applied Class = artifacts carrying the interactive-reviewer protocol or a §6 severity-contract digest. Surfaces enumerated (grep over .claude/skills/**, agents/**, setup.d/, plus the out-of-repo command): ~/.claude/commands/reviewer.md — out-of-repo sibling, hand-patched to §6 grammar 2026-08-10, now shadowed in-repo by this skill (skill-over-command precedence), SWEPT-CLEAN; agents/reviewer-discipline.md:37 — sibling run-moment §6 digest, GAP-FOUND (no cross-link between the two digests) → FIXED this PR (See-also drift rule in .claude/skills/reviewer/SKILL.md); .claude/skills/arch/SKILL.md:91 — stale «pending» claim about the global command, GAP-FOUND → FIXED this PR; .claude/rules/reviewer-discipline.md:56 — the operating SSOT itself, untouched by design (both digests subordinate to it); packages/core/templates/shared/skill-context/aif-review/SKILL.md + aif-orchestrator-discipline — shipped consumer surfaces, deliberately DEFERRED to advisor-pattern §8 item 9 (never a silent copy; ownership-table read-only for sessions); setup.d/10-skills.sh:108-127 — the delivery manifest, deliberately untouched, tier decision recorded as an item-9 input. No other surface in the class (agents/fidelity-auditor.md + agents/review-sidecar.md carry the cold-protocol grammar shipped by #1374, different altitude, already current).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
R-phase audit per
.claude/orchestrator-prompts/slow-test-triage/kickoff.md: identifies pre-push slow tests, classifies their slow-path cause, recommends Option 1/2/3/4 per-test, surfaces 3 maintainer decisions. R-phase only — no source/config edits; I-phase awaits maintainer GO on the DN-X items.Changes
docs/meta-factory/research-patches/2026-05-25-slow-test-triage.md(361 lines) — single new research-patch fileaudit-ai-docs.test.ts+principles/).legacy-trailer-checks.shconfirmed absent (Wave 10.3 deletion verified viapre-push.ts:17)reviewer-discipline.md §2(«Option A → consequence X / Option B → consequence Y» format, no strategy decided):ATTN — observations for maintainer (not blocking merge)
60_000(a subsequent commit bumped it). Worker noted transparently in §0.1; patch reflects current 60s.packages/core/principles/14-skill-drift-detection.test.tsvia the conditionalexistsSync('scripts/check-skill-drift.sh')guard; if the script is removed in future the test silently drops from pre-push coverage. Flagged in §1; not a DN since no current action.Prior-art consult
Prior-art:trailer — N/A, this PR adds 1 file (markdown research-patch) and is not a capability commit perCLAUDE.mddefinition (no dep added, no ≥50 LOC under newpackages/core/<new-dir>/, no ≥80 LOC underpackages/). The commit body documents the audit explicitly.Test plan
npm test --workspace=@rules-as-tests/core --run— pre-push hook fired full vitest suite during push: 111 passed, 4 skipped (verified in push output)npm run --prefix packages/core test:principles— included in the pre-push run above; principle 10 (research-patch annotation) and principle 13 (§1.7 substance) both pass for the new filemake self-auditgreen — not run locally (kickoff §1 marks audit-yml out of scope); will run on CIwc -l= 361 and section grep.claude/worktrees/agent-a29438b4af1885362/; verifiedgit rev-parse --show-toplevel≠ main repo path; satisfiesparallel-subwave-isolation.md §1§1.7 Skipped: mechanical R-phase output — adds one research-patch markdown file, introduces no new discipline rule (the patch's own §5 carries Forward+Backward+T15 self-reflexive content per principle 13)