[Spec 1280] PR-1: correct the prompt-surface measurement instrument (no prompt surface touched) - #1319
Merged
Merged
Conversation
Judgment-not-rules rewrite of the always-on prompt surface, targeting >50% reduction for a frontier-model fleet. Carries the three architect-mandated designs: the per-surface cut plan with word targets, the A/B non-inferiority eval design with a pre-registered decision rule, and the scar-rule carriage plan; plus the rollback story. Also records a measurement defect found during the spec phase and endorsed by the architect as criterion M0: scripts/measure-prompt-surface.sh derives its phase-task term from codev-skeleton/porch/prompts/, a dead Ralph-SPIR-era tree no code reads, while the live resolver loads protocols/<name>/prompts/. The metric was structurally blind to this project's largest target. Corrected always-on baseline is 33,519 served words, not 21,702.
CMAP round 1 (Codex + Claude, both REQUEST_CHANGES, both HIGH) fully incorporated; no rebuttal filed — every finding verified against source and accepted. Plus the architect's full-surface scope directive. Key corrections: - All per-surface ceilings restated GROSS (scar carriage counted inside them). Iteration 1 stated ceilings net-of-scar while M1 was gross; the gross landing would have been 50.2-51.4%, failing M1 while every row passed. - Baseline 33,519 -> 34,255: CLAUDE.md @imports the hot tier since #1119, so wc -w excludes 736 always-loaded words. Inherited from the measurement script's own stale comment. - Retracted iteration 1's 'no code reads porch/prompts' claim: drawn from a truncated grep. review-prompt-routing.test.ts:29 reads it (Spec 987 protection). M6's verification method rewritten. - New M10: ~25 test files pin exact prose in the surfaces being cut, incl. a pure-addition diff on builder-prompt.md incompatible with 824 -> 420. Each assertion is a prior spec's protection; retiring one is a governance act. - M0 extended: per-file four-tier resolution, role file, hot-tier transclusion, and per-audience segment reporting. - A/B: contamination controls, eligibility exclusion for issues touching surfaces under test, O1 demoted to advisory-with-tripwire. - Rollback restated as seven groups with a scar-registry dependency rule. Scope directive: full prompt surface inventoried by audience (architect / builder / phase / consultant) with resolver paths and served word counts. Segment targets: builder -53.2%, architect -61.2%, consultant -33.8%.
…none disputed Records the round-1 verification and resolution of all 13 findings from Codex and Claude, plus the architect's full-surface scope directive. No rebuttal filed - every finding was verified against source and accepted, including two factual errors in the spec's own Current State.
Architect-directed. Specifies PR-1's exact contents (corrected script + its first-ever tests + corrected segmented baseline + in-place annotation of the 1252 artifacts), what is deliberately excluded (any prompt-surface edit, the scar registry, the dead-tree deletion), timing (end of first implement phase, before any cut), and the safety verification: no test asserts on either 1252 word-count artifact, so the annotation carries no re-baselining cost.
…out contract added
…uted Architect-directed re-review of the revised spec. Two arithmetic errors caught that would otherwise have shipped: - M1's HOLD-at-50-52% branch was unreachable. M1 is fully derived from the M2 ceilings (met => <=16,016 = -53.2%); the HOLD band needs 16,442-17,128, i.e. M2 already failing. Recomputed and confirmed; replaced with the reachable contingency (denominator movement). - M5 proved nothing about prompts: it extracted gate/check names from an unchanged protocol.json and notifications from unchanged call sites, so every capability would report present even if every instruction vanished from the served prompts. Now inventories the resolved, expanded prompt surface with a contract-presence assertion per capability. Also: - M0c/M0(g)/T15: the metric could not distinguish deleted words from words relocated to skills - phantom savings on an axis T2 does not cover. - Coverage restated per-surface (enumerated from disk, unioned) - 'all ten protocols in both trees' is impossible: release is project-local (10 vs 9) and four protocols intentionally lack prompts/consult-types. - Inventory gap fixed: codev/protocols/release/protocol.md (1,626w) was missing because I enumerated the skeleton, where release does not exist. - M5 removal severity resolved on M10's committed-retirements pattern. - A/B execution: M7 gates verify-approval not the PR; treatment arm merges and control closes; cost restated as ~6 of 12 runs producing merged work; architect load (~24 gates + 12 PR reviews) named as the binding constraint. - O3 split: pre-merge findings gate SHIP, post-merge defects are a 14-day rollback signal. - M2b: nothing protected CLAUDE.md's human readability at 5,815 -> 1,900. Three of this phase's four self-audit findings share one root cause: enumerating from a convenient source instead of the authoritative one.
Architect-directed pre-gate compression. Folded, not dropped: every M-criterion (14), every test (16), the full inventory, the rollout section, the A/B design, the rollback groups, and the Baked Decisions verbatim all survive unchanged. Cut: the narration of how the spec got here — round-by-round finding tables, '(round-2 finding)' asides, the superseded 33,519 figure, and the withdrawn HOLD-band arithmetic. That record lives in the iter1/iter2 rebuttal files and the thread, which is where it belongs. Self-verified: all 20 template headings present and ordered; 14/14 criteria, 16/16 tests, 5/5 outcomes, 7/7 rollback groups; every load-bearing number reproduced; porch's spec_has_required_sections passes.
Gate verdict NOT APPROVED; Waleed redirected the acceptance model: 'I don't think the goal should be a particular size. That's not the right criteria. It should be to stick to the principles outlined in the blog post.' Acceptance is now principle conformance, judged per file: - Fetched the blog and quoted its principles VERBATIM as P1-P7, each restated as a per-file conformance question answerable from a diff. P5 (auto-memory) is declared N/A with reason rather than silently skipped: it is a Claude Code harness feature and Baked Decision 1's fleet includes GPT/Gemini consumers. - P7 (delete worst-case guardrails) collides with the scar-rule exemption. Named the collision and resolved it explicitly rather than papering over: the blog's guardrails protected against bad output, which judgment now handles; scar rules protect against irreversible acts, where being wrong once is unbounded. DEMOTED from acceptance to observability: the >50% target, all per-surface word ceilings, per-segment reduction goals, and the thin-margin analysis. M1/M2 are now reporting obligations that cannot fail on a number; T3 asserts report completeness, not ceilings. M0/M0b/M0c survive - they keep the project honest about what happened. NEW M11 - architect personal inspection of every changed file: per-file manifest (path, old/new wordcount, principles applied, rationale), architect reviews actual diffs file by file before each phase advances, batches <=12 files. Sized it honestly: 131 prompt-bearing files exist, but 60 codev/protocols copies are byte-identical mirrors of their skeleton twins, so inspection covers ~66 distinct content decisions with twin sync verified mechanically by T7 - otherwise half the human review is re-reading identical bytes. New T16 fails a phase if a changed file is missing from its manifest. Approach 2 (generated prompts) now also rejected as incompatible with M11: the architect cannot inspect old-vs-new diffs of files that no longer exist. UNCHANGED per directive: A/B mandatory, scar exemption, Baked Decisions, PR-1 early landing, M10 assertion-retirement-as-governance.
Codex REQUEST_CHANGES (5) + Claude COMMENT (6) on the acceptance-model revision. Both caught the same error in my own inventory: I reported '3 codev/protocols copies that differ'. Zero differ - all three are local-only with no skeleton twin. My cmp -s loop treated nonzero exit as 'differs', but cmp also exits nonzero when a file is absent. T7 now runs on the intersection of twinned files. Fifth self-audit finding this phase; fourth sharing one root cause (trusting a convenient signal over the authoritative thing). Structural fixes: - M5 conflicted with P6 (P6 permits replacing narrated names with a reference to structured truth; M5 demanded the names stay in prose, so a conformant rewrite would fail). Representation now means name-in-text OR an explicit resolvable reference to the source that still defines it. M5's detection limit is also stated: it detects deletion, not inversion - gap assigned to M11/O4 plus a short set of semantic invariants asserted as behaviour. - A/B arms could not be both 'same base commit' and 'pre-/post-rewrite commits'. Rebuilt as a prompt-only overlay: both arms branch from source commit S, control applies one overlay reverting G2-G6 and nothing else, each run records source hash AND prompt-surface hash. - Added the issue-mandated per-surface cut plan as a disposition table: every category marked rewritten / inspected-but-unchanged / excluded-with-reason. Resolves the scope contradiction (hot tier, .claude/skills) too. - M12: no release between the rewrite merge and the SHIP verdict. - A/B control isolation depends on tier-2 shadowing tier-4 (verified: 0 skeleton files lack a codev/ twin). Stated as a precondition; T14 asserts it pre-flight and voids the pair on failure. - M0(g) 'total authored surface' defined as physical files, no dedup, no expansion - deliberately a different basis from the always-on buckets. Not actioned by me: issue #1280's title/Goal still state the superseded >50% target measured by the disqualified script. That is the architect's artifact; flagged rather than edited.
Phase boundaries drawn by M11 inspection load rather than subsystem grouping, since the architect's per-file review is the throughput constraint. Enumerated the 67 decisions against disk rather than trusting the spec's '~66': protocol.md 10 (9 skeleton + release codev-local) + builder-prompt 9 + prompts 18 + templates 8 (6 skeleton + 2 codev-local maintain) + consult-types 18 + roles 3 + CLAUDE/AGENTS 1. Batches: 4,10,9,11,10,10,9,4 = 67, max 11 against the cap of 12. Sequencing rationale: - P0 (corrected instrument) ships as PR-1 before any prompt word changes - rewriting first would make every later measurement unfalsifiable. - P1 kept small (4) - highest blast radius, and it calibrates the conformance standard for the phases that follow. - P3 is only 9 decisions but carries the entire M10 test-retirement burden (baked-decisions.test.ts pure-addition diff vs rewriting builder-prompt.md), which is why it is not merged with P1. - P8 rebuilds the scar registry last, against the settled surface. Named two capability risks that P2 (interfaces-not-examples) could plausibly destroy: the plan template's machine-readable phases JSON (porch requires it) and the consult verdict format (consult parses it). Both get live integration checks rather than fixtures.
Both reviewers reproduced the 67-decision count independently - the accounting
held. What failed was the supporting work riding alongside the decisions, which
a plan drawn purely by inspection load under-specifies.
Two blocking mechanism gaps, both verified against source:
- P6 had no adopter-resolvable mechanism. protocol.md is inlined at spawn via
{{protocol_reference}}; protocol.json is inlined NOWHERE and doesn't exist on
disk in fresh adopter projects, so 'reference protocol.json' was the
fetch-by-path CLAUDE.md forbids. Resolved by verifying resolveCodevIncludes is
extension-agnostic: a fenced json block with {{> protocols/<p>/protocol.json}}
resolves through all four tiers, no porch change. New T18 covers strict AND
soft mode.
- Skill relocation is a FOUR-tree sync (.claude, .codex, and both skeleton
copies, with existing drift). M0(g) counted one, so relocated content would
have left Codex agents and adopters without it and been reported as DELETED -
inverting the honesty artifact. Phase 0 widens the basis; new T17 asserts parity.
Also: pre-rewrite capability inventory frozen in Phase 0 (was extracted last
while Phase 2 asserted against it); verify phase given an explicit home for
M7/M8/M12/T13/T14; T3/T16 into Phase 0; 'review batch' defined to include tests
and registry, with Phases 4 and 9 declaring two batches each; scar canonicals
verified against the ratified YAML in Phase 1 with T4 deferred to Phase 9;
commit-level group purity replacing the broken phase-level mapping; per-phase
green suite; M10 reassigned to where each collision actually lands.
Phase count 10 -> 11; decisions 67; max batch 11.
Any skill this project touches gets four-tree parity (T17 enforces, scoped to the touched set). Pre-existing drift on untouched skills and the skeleton-absence product question go to a separate architect-filed issue and are recorded-known-state here - they must not fail T17.
…e measurement instrument Spec 1280 M0/M0b/M0c. No prompt-surface file is touched by this commit - the instrument must be correct and public before anything is scored by it. THREE DEFECTS CORRECTED (the 1252 instrument was committed, deterministic, and wrong): 1. Phase-task term was derived from codev-skeleton/porch/prompts/, a dead Ralph-SPIR-era tree with no runtime consumer, while loadPromptFile loads protocols/<p>/prompts/. Real SPIR phase prompts average 1,396 words; the dead tree averaged 400. The metric was blind to the largest always-on surface. 2. roles/builder.md (1,837w) was omitted despite spawn-worktree.ts:854 injecting it into every builder spawn. 3. The script's own comment claimed CLAUDE.md 'already inlines' the hot tier. Since #1119 it @imports, which Claude Code transcludes - so wc -w excluded 736 always-loaded words. Reported baseline: 21,702 -> 34,235. ALSO FIXED, found while testing (a fourth inaccuracy, mine): include expansion was ADDITIVE - it counted a {{> path}} directive's own tokens plus the content substituted for them, over-reporting ~2 words per include. expand_text now does real substitution mirroring resolveCodevIncludes (regex replace in place, recursive, depth-guarded, unresolved -> empty), which also makes the T2 phantom-savings property exact. This is why the figure is 34,235 rather than the 34,255 quoted in the spec; size is reporting-only, so no criterion moves. New in this instrument: - per-file four-tier resolution (.codev > codev > skeleton), replacing two-tier directory-level selection that mis-measured single-file overrides - exclusive buckets (which sum) reported separately from derived audience loads (which overlap) - never presented as a sum - TOTAL_AUTHORED on a deliberately different basis (physical files, no dedup, no expansion) across ALL FOUR skill trees, so relocation cannot be reported as deletion (M0c) Frozen pre-rewrite capability inventory (M5): 57 capabilities, 47 present in served prompts, 10 absent pre-existing (porch delivers those via task JSON, not authored text). Extraction is over served prompt text, not protocol.json - extracting from unchanged config would report every capability present even if every instruction vanished. Representation accepts a resolvable {{> }} include, so a conformant P6 rewrite cannot fail M5. A pipefail/grep -q bug in the extractor initially reported all 57 as absent: grep -q exits on first match, printf takes SIGPIPE, and the pipeline reports failure BECAUSE the match succeeded. Fixed with a here-string. Same failure class this project keeps recording: an exit code with two causes, read as one. Tests: T1, T1b, T2, T3, T11, T12, T15 (instrument) and T16 (manifest completeness, written before the first manifest exists because the guard must predate what it guards). 22 passing. 1252-word-baseline.md and 1252-word-after-phase7.md annotated in place - originals preserved, marked superseded, with the reason and a pointer.
…f-inflicted bugs recorded
…ndition; skeleton is a build copy
…orm-independent CI caught what the local suite could not: PR #1319 measured 34,231 while my Mac measured 34,235. Same commit, same tree - verified by exporting the PR merge ref and measuring it locally, which reproduced 34,235. ROOT CAUSE: wc -w is not portable for this corpus. macOS/BSD wc in a UTF-8 locale counts the emoji U+26A0 WARNING SIGN + U+FE0F VARIATION SELECTOR-16 as TWO words; GNU wc on Linux, LC_ALL=C wc, and Python's str.split() all count ONE. spir/protocol.md carries four such banners, and 4 x 1 = the exact delta. Located by diffing per-line counts between BSD wc and Python rather than by guessing. WHY THIS IS A REAL DEFECT, NOT A TEST NUISANCE: this instrument exists to produce an honest before/after comparison. A count that depends on which machine ran it means measuring 'before' on a laptop and 'after' in CI yields a fictional delta - the phantom-savings failure the whole instrument was built to prevent, wearing a different hat. FIX: word counting is now DEFINED by the script (whitespace-delimited tokens of the UTF-8 decoded text, via Python str.split) rather than delegated to the host's wc. Verified identical across worktree, a clean export of the PR merge ref, and LC_ALL=C: all three now report 34,231. Two new regression tests: the script must not delegate to wc -w, and the total must be identical under LC_ALL=C and UTF-8. Baseline restated 34,235 -> 34,231 with both corrections documented in the artifact header (-20 additive-include model, -4 wc portability). Size is reporting-only under the charter amendment, so no acceptance criterion moves. Noted for the record: the characters that broke portability are the 'WARNING BLOCKING' worst-case-padding banners that principle P7 exists to delete. 24 tests passing.
Collaborator
CI failure on this merge — diagnosed: runner-speed flake, not a regressionThe Verified locally against the same main HEAD: all 20 tests pass, file total 23.5s. That test invokes the measurement script twice ( Suggested one-line fix for a follow-up (or PR-2 of the spec): give the script-invoking tests an explicit timeout budget, e.g. Leaving the fix to the spec lane rather than hot-patching main. (Diagnosed from the main-architect session after the CI cron alert.) |
Collaborator
|
Rerun of the failed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-1 of Spec 1280 — the corrected measurement instrument, landing early
No prompt-surface file is touched by this PR. That is the point: the instrument must be
correct and public before anything is scored by it. Spec 1280 criterion M0b, precedent
#1290.
Why the old instrument couldn't score this project
scripts/measure-prompt-surface.sh(Spec 1252) was committed, deterministic, and wrong:codev-skeleton/porch/prompts/— a dead Ralph-SPIR-era tree with no runtime consumer — whileloadPromptFile(commands/porch/prompts.ts:78) loadsprotocols/<p>/prompts/roles/builder.md(1,837w), whichspawn-worktree.ts:854injects into every builder spawnCLAUDE.md"already inlines" the hot tier. Since #1119 (managed-block.ts) it carries@importlines, which are transcluded at session launchwc -w CLAUDE.mdexcluded 736 always-loaded wordsReported baseline: 21,702 → 34,235.
A fourth inaccuracy, found while writing the tests — and it was mine
Include expansion was additive: it counted a
{{> path}}directive's own tokens plus thecontent substituted for them (~2 words per include).
expand_textnow performs realsubstitution mirroring
lib/skeleton.ts resolveCodevIncludes— regex replace in place,recursive, depth-guarded, unresolved → empty.
This is why the figure is 34,235 and not the 34,255 quoted in the approved spec: a
20-word delta across 10 iterations. Size is reporting-only under the charter amendment
(2026-08-01), so no acceptance criterion moves — but the spec's number is superseded, and
1280-word-baseline.mdsays so in its header rather than quietly disagreeing.What's new in the instrument
.codev→codev→ skeleton), replacing two-tierdirectory-level selection that mis-measured single-file overrides
overlap by design) — never presented as a sum
TOTAL_AUTHOREDon a deliberately different basis (physical files, no dedup, noexpansion) across all four skill trees, so relocation can never be reported as deletion
release, which iscodev/-only and was missed by every earlier inventoryFrozen pre-rewrite capability inventory (M5)
codev/resources/1280-capability-inventory.json— 57 capabilities, 47 present in servedprompts, 10 absent. The 10 are pre-existing: porch delivers those gates and checks via task
JSON rather than authored prompt text. That asymmetry is captured as baseline state, not
something the rewrite must invent — and it confirms the soft-mode concern the plan raised for P6
is already real.
The inventory is extracted over served prompt text, not
protocol.json. Extracting fromunchanged config would report every capability present even if every instruction vanished from
the prompts. Representation accepts a resolvable
{{> }}include, so a conformant P6 rewritecannot fail M5.
A
pipefailbug in the extractor initially reported all 57 as absent:grep -qexits onfirst match,
printftakes SIGPIPE, and the pipeline reports failure because the matchsucceeded. Fixed with a here-string.
Tests — the script had none, which is how three defects survived in a "committed and reproducible" artifact
22 passing: T1 (sources the served directory, asserted against the real loader), T1b
(per-file four-tier, fixture with one overridden prompt), T2 (phantom-savings: moving text
into an include is exactly neutral; non-markdown includes expand), T3 (report completeness),
T11 (buckets vs audience loads), T12 (determinism), T15 (relocation visible, never
reported as deletion), T16 (manifest completeness — written now, before the first manifest
exists, because the guard must predate what it guards).
Full suite: 203 files, 4,063 tests, 0 failures.
Record correction
1252-word-baseline.mdand1252-word-after-phase7.mdare annotated in place — originalspreserved unaltered, marked superseded, with the reason and a pointer. They are shared knowledge
other work reads; the record gets corrected while this project builds, not after.
1252-behavior-baseline.mdis unaffected (different instrument) and remains the valid"before" for M8.
Review note
Per M11, this batch is: 2 scripts · 2 test files · 3 artifacts · the manifest format. The
spec and plan artifacts also ride along (both gate-approved) so
maincarries the approveddocuments early.