docs(token-economy): round-2 GO close-out — seat nits + ledger note - #1202
Merged
Conversation
added 6 commits
August 1, 2026 03:56
…l → Fable decide funnel
… candidate survey against a measured billing profile Both stages carry the measured host-side profile inlined in §2: the aif container cannot reach ~/.claude/projects/, so the corpus travels with the dispatch input (same pattern as the anthropic-engineering-prior-art kickoff). Headline measurement over 247 transcripts / 58,341 assistant turns: 85.5% of weighted spend is context re-submission (cache read 53.3% + cache write 32.2%), output only 14%. Median session is 213 turns, so an always-on token costs 21.2x a one-shot input token, 51.6x at p90. That reframes the research: the head is resident-context size x turn count, not generation. Stage A attributes the always-on payload to artefacts and ranks by cost units. Stage B runs the BFR six-item mechanism per candidate against the same profile. Both are Tier 1 with the executor-tier bridge-profile marker; neither adjudicates a winner — an Opus distillation seat merges them into the patch the operator reads. Prior-art: skipped — dispatch inputs only, no new capability, dependency or module introduced by this commit
…rofile, inlined always-on population, de-collided host-verify + ledger Rows 2-3 Round 1 of the dispatch-input station (2 bottom seats + 2 Opus shadow arms, first shadow-present cohort runs) returned STOP on both kickoffs. Every finding is addressed: - MEMORY.md 7,738 B was unmeasured-at-authoring; corrected to the measured 16,504 B (bottom-seat K2 catch). - "Bash+Read = 96%" was arithmetically impossible (own top-10 rows sum past the stated denominator); real share is 87.9% of the full 40,101,993-char 52-tool total (shadow K2 catch). - All Section 2 tables now come from ONE aggregator run and cross-foot exactly (58,345 turns incl. the synthetic row); earlier drafts mixed two runs minutes apart. - W1's index-minus-claudeMdExcludes derivation replaced by an inlined host-measured always-on population, because observed session state contradicts both sources: 3 of 7 excluded rules load anyway (35,197 B) — carried into stage A as a pre-found finding (shadow K4 catch). - Host-verify contracts de-collided (per-stage globs + test -f guards + filename binding), wrong test runner fixed, first-line scope-comment contract surfaced into acceptance. - claude-code-guide replaced with container-reachable fallbacks; sibling task 53c2ecdd state corrected to done-not-harvested. - RTK seed framing neutralised (K6 candidate adjudicated REAL). - Calibration ledger: Rows 2-3 appended, cohort 2/5 with both runs carrying Opus-only K1/K2 finds; proxy-bottom-seat caveat recorded; counters synced. Prior-art: skipped — dispatch-input fixes and ledger rows only, no new capability introduced
…my-research-9524ba # Conflicts: # .claude/orchestrator-prompts/token-economy-research-s-a/kickoff.md # .claude/orchestrator-prompts/token-economy-research-s-b/kickoff.md
… round-2 note Prior-art: skipped — one-line presentational fixes from round-2 seat findings, no capability
…my-research-9524ba # Conflicts: # .claude/orchestrator-prompts/arch-v2-context-pipeline/calibration.md # .claude/orchestrator-prompts/token-economy-research-s-a/kickoff.md
9 tasks
artyhoo
added a commit
that referenced
this pull request
Aug 1, 2026
…sts (#1203) ## Summary Stage A of the token-economy research umbrella: attribute this repository's **always-on context payload** to concrete artefacts and rank them by measured cost, against the billing profile inlined in the kickoff §2 (247 transcripts / 58,345 assistant turns, measured host-side because `~/.claude/projects/` does not exist in the aif container). Zero build — the patch implements nothing, proposes only. Its consumer is the Opus distillation seat that will merge stage A with stage B into the single artefact the operator reads. The central result: **always-on documentation and accumulated tool output are co-dominant**, 754,884 vs ~430,243 cost-units in a median session (~1.75 : 1), so an intervention that touches only always-on docs leaves at least ~35% of the resident bill untouched. Session length is *not* a lever on that ratio (both classes are linear in N; the ratio moves <0.05% from N=213 to N=833) — the levers are arrival-position distribution and per-tool payload volume. ## Changes - New research-patch `docs/meta-factory/research-patches/2026-08-01-token-economy-s-a-profile.md` (559 lines) — the only file this PR touches. - **W1** — 11-row always-on attribution (bytes, est-tokens at the stated 4 B/t assumption, loading channel), each repo-side size carrying its own `wc -c` output; plus the edit-time-injection class (15 rules) as a separate resident-once-fired class. - **Carried finding** — `.claude/settings.json` `claudeMdExcludes` lists 7 rules; only 4 were actually evicted in the observed live session, while 3 (35,197 B combined) loaded anyway. Recorded as a measured finding with a proposed next-stage root-cause investigation, deliberately **not** fixed here. - **W2** — ranked cost table with cumulative share and a named head/tail boundary (head = top 7, 86.3%, ends at `cold-seat-economy.md`). The per-prompt bootstrap digest is rank 3 by cost and last by byte size — the T-TokenA-A guard demonstrated rather than asserted. - **W3** — tool-output cost class computed under a stated uniform-arrival assumption with per-factor direction of error, compared head-to-head against the always-on head, with a two-directional falsifier and its break-even arithmetic. - **W4** — progressive-disclosure inventory, one line per artefact. ## Prior-art consult - [x] Every commit carries a `Prior-art:` trailer; all are `skipped — <rationale>` because this PR adds one research patch and no capability (no dependency, no module, no code). - [x] No new capability area surfaced — no SSOT entry is due from this stage. (Stage B owns the candidate survey and carries its own SSOT row.) - [x] No existing SSOT entry matched, so no `Last reviewed` touch applies. - [x] context7 n/a — no library/API surface is involved; this stage measures this repository's own context payload. ## Test plan - [x] `bash scripts/host-verify.sh token-economy-research-s-a` → **4/4 PASS on Darwin (host, not container)** — the kickoff's declared contract, run where the work lands per `destination-environment-verification.md §1`. - [x] `bash scripts/run-local-ci-sweep.sh` → PASS (`md-ci-only`, diff-scoped). - [x] `npx vitest run packages/core/principles/10-research-patch-annotation.test.ts` → 5/5 (inside host-verify). - [x] `wc -l` = 559 < the 600-line pre-commit ceiling. - [x] Manual smoke: `git diff --name-only origin/staging...HEAD` returns exactly one path — no rule, hook, skill, `settings.json` or `.husky/**` touched. ## Provenance Kickoff `.claude/orchestrator-prompts/token-economy-research-s-a/kickoff.md` (merged to staging in #1200, revised #1201/#1202) · base `origin/staging` · substrate **aif task `089f68fe-bbcd-4542-83a4-df8dfb7c01c6`**, `bridge-profile: Z.AI GLM-5.2 SDK` (Tier-1 executor tier per CLAUDE.md `Task-tier routing`) · host-side measurement + egress + review by the dispatching session (Opus 5) · fidelity **Round 1**. ## Review findings **Factory:** aif closed the task `done` at round 0 with the deliverable **partly uncommitted** (the D12 shape, `/harvest §1` step 2). Not shipped via `--confirm-dirty-residue` (which abandons the modifications) — instead the residue was inspected, verified, and the worker committed it itself during rework round 1 (`13888a764`). **Cold code-review, round 1** (fresh Opus seat, dialogue-blind, re-computed every table and re-ran every `wc -c`) → **REVISE**, 3 blocking + 5 non-blocking: - *B1* — the falsifier claimed a longer session amplifies tool-output relative to always-on; both classes are linear in N, so the ratio is fixed at ~1.82 (moves <0.05% across N=213/517/833). The companion claim that always-on carries no median-vs-mean bias was also false — its 21.2 multiplier is itself median-derived. - *B2* — the dominance thresholds did not bite: at the quoted 160-turn residency tool-output is 626,944 vs 754,884, still 17% smaller; real break-even is 192.7 turns = 90.5% of a median session, unreachable under any arrival distribution. - *B3* — the prose widened the edit-time-injection class to "any `paths:`-bearing rule" and then claimed 15/15 completeness against the narrower enumerated set, omitting `phase-research-coverage.md` (34,151 B — larger than any listed row). - Five non-blocking: cumulative-share column, per-session `÷247` column, est-token total convention, an underivable upper bound, a mislabelled See-also entry. All eight were routed back as an aif `request_changes` round (never self-fixed at the altitude that produced them). Round 2 fixed all eight. **Cold delta seat, round 2** (a *fresh* seat handed only the incremental diff + the round-1 findings, per `cold-seat-economy.md §3` — not a resumed transcript, not a full re-audit) → **GO**. It confirmed B1-B3 and M1-M5 fixed by its own arithmetic, and found six new MINORs, all in explanatory notes with conclusions unaffected. Five were corrected host-side in `e10892cc` rather than spending a third executor round on arithmetic containing no judgment — each independently re-derived first, and each disclosed in that commit's body rather than squashed into the worker's commits: - the est-token method note named the wrong rows (16,585/4 = 4,146.25 rounds *down*; five rows round up, not two — the total was always right, only its justification was wrong); - break-even 192.65 → 192.7 / 90.5%, not 192.6 / 90.4%; - rank-1 threshold `139,856 / 21.2 × 4` = 26,388 B, not 26,364 B; - a stale `(§A3 falsifier)` cross-reference — the falsifier lives in §A4.4; - the self-quoted `wc` block, restated as a **band plus the deciding threshold**. Three rework passes each re-ran `wc` and each quote was stale by the time the edit carrying it was written: recording the size changes the size. A fourth exact quote would be stale too. The band + the 26,388 B rank-1 threshold are stable under further editing, and every figure any pass produced (30,453 / 36,786 / 37,007) lands on the same verdict. **Open MINOR, accepted deliberately:** the fidelity seat noted acceptance item 5 asks for a literal `Coverage:` line where the patch delivers `## A8 — Coverage`; it graded the substance fully present. Not edited — invalidating a fresh `GO` to satisfy a literal-string reading is the worse trade (`cold-seat-economy.md §1`, `#reaudit-on-sha-move`). ## Fidelity verdict FIDELITY: GO Basis: .claude/orchestrator-prompts/token-economy-research-s-a/kickoff.md Round: 1 Audited-SHA: e10892c Evidence: docs/meta-factory/research-patches/2026-08-01-token-economy-s-a-profile.md:1 — first-line scope comment matches `SCOPE_ANNOTATION_RE` (packages/core/principles/10-research-patch-annotation.test.ts:23); the diff adds exactly the binding filename and nothing else. Findings: 2 MINOR — (1) `Coverage:` delivered as an `## A8 — Coverage` heading rather than a literal line, substance fully present; (2) §A5's closing sentence ("single largest disclosure opportunity") edges toward evaluative framing inside an inventory-only section, but stops short of proposing a fix and stays inside the kickoff's carry-the-finding instruction. No missing, no extra, no diverged deliverable; descopes clean (no implementation, no stage-B candidate evaluation, no shipped-axis analysis, one PR / one patch). **Watch-list for any future round on this file** (the fidelity seat's, carried here so a later reviewer inherits it without inheriting a transcript): 1. Filename and first line are load-bearing: the §5 host-verify globs match on `token-economy-s-a` and `markdownlint-cli2` exits 0 on a zero-match glob, so a rename escapes verification silently; the first line must stay `<!-- scope:token-economy -->` with **no space after the colon**. 2. 41 lines of headroom under the 600-line ceiling — growth is the most plausible drift; trim before adding. 3. §A5/§A7 must stay an inventory and one-liners respectively; proposals belong to stage B and the distillation seat. 4. Nothing may be re-measured host-side or inferred beyond the kickoff §2 profile — the drift shape to watch is a future round "tightening" the arrival-position assumption with a plausible derived number instead of leaving it `INCONCLUSIVE`. 5. Row 11's `× 213 × 1.0` is a specified formula, not a modelling choice — normalising it onto 21.2 "for consistency" would collapse the T-TokenA-A demonstration (the digest is rank 3 by cost, last by size). 6. Keep the See-also note reconciling this patch's scope with the arch-v2 S1 figures; without it the two numbers read as a contradiction. ## Parked questions none — the worker parked nothing; the kickoff §3a fork contract was not triggered in either round. ## §1.7 Self-discipline check (REQUIRED if PR touches discipline-bearing files) ### §1.7 Skipped: research-patch only — this PR introduces and extends no discipline rule, touches no `.claude/rules/**`, no `packages/core/principles/**`, no `CLAUDE.md`, no `prior-art-evaluations.md` and no shipped template; it adds one measurement artefact under `docs/meta-factory/research-patches/` whose own folder-level authority is append-only and scope-bound by gap.
9 tasks
artyhoo
added a commit
that referenced
this pull request
Aug 1, 2026
…billing profile (#1204) ## Summary Stage B of the token-economy research umbrella: survey the candidate levers for reducing the expensive-tier bill, each evaluated against the measured billing profile stage A ships (85.5% of weighted spend = context re-submission; median-session residency multiplier 21.2×). Four seed candidates get the full W2 treatment (BFR §3 sweep, T16 problem-class statement, §2.1 cost line with arithmetic, two-axis verdict, cost gate, falsifier); a four-phrasing search sweeps for mechanisms absent from the seed list; six next-stage proposals are ranked by the §2.1 cost line they attack. Zero build — proposals only; the consumer is the Opus distillation seat that merges stages A+B into the single artefact the operator reads. Headline verdicts (provisional, distillation seat adjudicates): **RTK — operator DEFER / shipped REJECT-provisional** (attributable share 1.9-3.5% of total weighted cost; saving 1.7-3.1% after the 89% removal factor — honest arithmetic, not the vendor's per-command number); **progressive-disclosure gap — the largest measured lever** (attacks the resident-context head directly); **native CC features — ADOPT-continue** (configuration + discipline, no new adoption); **engineering plugin — DEFER-with-trigger** (0% on the advertised surface; harvested-verdict probe PENDING). ## Changes - New research-patch `docs/meta-factory/research-patches/2026-08-01-token-economy-s-b-candidates.md` (599 lines; the 600-line gate leaves 1 line of headroom by design — compression paid for every rework addition). - **W1** — own-stack enumeration (progressive disclosure shipped, measured seat/dispatch economy, model-cost routing, arch-v2 umbrella overlap labels) with file:line citations. - **W2** — four candidate blocks, each with the six mandated items; verdicts differ per axis where the axes genuinely differ. - **W3** — four quoted WebSearch phrasings for unnamed mechanisms; new candidates surfaced (CWL structured eviction → WATCHLIST, prompt-prefix-stability audit → ADOPT-as-discipline, structured-output-format audit → DEFER, bounded-session+watch-list → already shipped as `cold-seat-economy.md`); the negative-existence claim («no production-grade Read-result compressor surfaced») runs the explicit 6-item checklist and lands `coverage insufficient` (T14), not «category clean». - **W4** — six ranked next-stage proposals (P1 prefix-stability → P2 disclosure-gap → P3 sub-agent delegation → P4 Bash-output skill → P5 output-format audit → P6 RTK operator-axis trial), ordered by attacked cost line; no winner crowned. - `docs/meta-factory/prior-art-evaluations.md` — one appended SSOT row **#233** (RTK: operator DEFER / shipped REJECT-provisional, with Verdict / Rationale / Trigger to revisit). ## Prior-art consult - [x] All commits carry `Prior-art:` trailers (`skipped — research-patch only` — no capability added: no dependency, no module, no code). - [x] New capability area (RTK / output-filter proxies) → SSOT row #233 appended in the same commit as the evaluating patch, with Verdict / Rationale / Trigger to revisit. - [x] No existing SSOT entry matched RTK (grep verified in-patch); no `Last reviewed` touch due. - [x] Candidate sweeps ran WebSearch ≥3 phrasings + DeepWiki per candidate (quoted verbatim in each W2 block — the round-1 review rejected an unquoted sweep and it was redone quoted). ## Test plan - [x] `bash scripts/host-verify.sh token-economy-research-s-b` → **4/4 PASS on Darwin (host, not container)**, re-run after the merge-forward onto post-#1203 staging. - [x] `bash scripts/run-local-ci-sweep.sh` → PASS (diff-scoped). - [x] Principles 08 + 10 → 12/12 (inside host-verify). - [x] `wc -l` = 599 < 600. - [x] Manual smoke: `git diff --name-only origin/staging...HEAD` returns exactly the two permitted paths. ## Provenance Kickoff `.claude/orchestrator-prompts/token-economy-research-s-b/kickoff.md` (merged to staging #1200, revised #1201/#1202) · base `origin/staging` (merged-forward onto post-#1203) · substrate **aif task `5bf80b65-9de7-4648-a6cb-15f3120dc93e`**, `bridge-profile: Z.AI GLM-5.2 SDK` (Tier-1 executor per CLAUDE.md `Task-tier routing`) · host-side egress + review by the dispatching session (Opus 5) · fidelity **Round 1**. ## Review findings **Cold code-review, round 1** (fresh Opus seat, dialogue-blind; re-fetched external sources, re-computed arithmetic) → **REVISE**, 3 blocking + 8 non-blocking. The blockers: - *B1* — the T16 «wrong SURFACE» evidence claimed Bash output is «~0.03%» of resident context: a denominator category error ~100× off (tokens counted once at entry vs cache-READ raw, which counts every re-read). Corrected to 3.1% (mid-session) / 6.2% (turn-1 entry); the patch's own §3 independently lands 1.9-3.5%. Conclusion weakened, not reversed — and the wrong number had already reached SSOT row #233, amended per that file's stated convention (§3: git history is the audit trail). - *B2* — `CLAUDE_CODE_SIMPLE` does not exist; it was asserted as twice-confirmed (DeepWiki + context7) inside the own-stack section that exists to forbid recall. Retracted in place; the affected channel recorded `INCONCLUSIVE`. - *B3* — candidate 4 dropped required W2 items under a kickoff «reduced block» provision that does not exist (`grep -ic reduced kickoff.md` → 0). Full W2 treatment delivered on rework, queries quoted; only the genuinely-unreachable staging-presence probe stays PENDING. - Non-blocking: an uncited «~95% capacity» compaction threshold (retracted), a substituted checklist item, two mutually exclusive superlatives, a secondary-source attribution, two 404 URLs, a mislabelled arXiv title, three unquantified feature estimates. All routed back as one aif `request_changes` round (rework cost: $8.07 / 588k tokens on the executor tier). **Cold delta seat, round 2** (a *fresh* seat handed only the incremental diff + the round-1 findings, per `cold-seat-economy.md §3`) → confirmed B1-B3 and M1-M8 fixed by its own arithmetic, then **REVISE** on one new MAJOR plus four MINORs the rework itself introduced. All five were fully specified with no judgment left, so they were fixed host-side in `493ed4ae49` (disclosed there, not squashed into the worker's commits) rather than spending a third executor round: - MAJOR — the replacement trigger-sweep row claimed «18 entries» where the quoted grep returns **26**, classifying 12 — a mis-counted population inside the row whose purpose is to stop rows reading as complete. Now: 26 entries, 12/26 classified, remaining 14 fire-checked only, coverage stated per T14. - MINORs — the «100k-token threshold» was the API cookbook's figure attributed to the harness (now `INCONCLUSIVE`); a share-vs-saving label mismatch vs SSOT row #233; Read called the largest tool-result class where Bash leads (now second-largest, Bash named); a self-measurement quote already 34 chars stale (now a band — an exact self-quote goes stale with every edit pass). **Open MINORs, accepted deliberately (fidelity seat, graded MINOR):** candidates 3/4 render the T16 statement without the literal `Evidence:` clause (substance adjacent in both); `Coverage:` delivered as a `## Coverage` section rather than a literal line. Not edited — invalidating a fresh GO to satisfy literal-string readings is the worse trade (`cold-seat-economy.md §1`). ## Fidelity verdict FIDELITY: GO Basis: .claude/orchestrator-prompts/token-economy-research-s-b/kickoff.md Round: 1 Audited-SHA: 7cbb9db Evidence: docs/meta-factory/research-patches/2026-08-01-token-economy-s-b-candidates.md:1 — mandated filename + first-line scope comment; one W2 block per seed candidate (:89/:177/:242/:344) each carrying sub-items 1-6; SSOT row #233 appended at docs/meta-factory/prior-art-evaluations.md:306. Findings: 2 MINOR — (1) candidates 3/4 omit the literal `Evidence:` clause of the verbatim T16 form (substance adjacent at :278/:368); (2) `Coverage:` delivered as a `## Coverage` section, content obligation met. No missing, no extra, no diverged deliverable; descopes clean (no implementation, no crowned winner, no stage-A re-derivation, shipped-axis kept to provisional one-liners; one PR, no new umbrella). **Watch-list for any future round on this file** (the fidelity seat's, carried so a later reviewer inherits substance without a transcript): 1. Filename + first line are load-bearing: the §6 host-verify glob keys on `token-economy-s-b` (disjoint from stage A's glob so a concurrent file cannot mask absence); principle 10 fails without the first-line scope comment. 2. The T16 statement is specified verbatim; two of four candidates already drop the `Evidence:` clause — drift reappears there first, and it is the clause separating a real problem-class match from name-adjacency. 3. Every cost estimate must name its §2.1 row and express reduction as a share of TOTAL weighted cost — vendor per-command percentages read large while the measured head is resident context (85.5%). Candidate 1's arithmetic is where this is most likely to be «simplified» back into a vendor number. 4. W4 ranks FOLLOW-UP STAGES by cost line, never candidates — that migration is the K6 framing bias the seat separation exists to prevent. 5. `PENDING-STAGE-A` / `PENDING — verdict not yet harvested` are contract-grammar, not placeholders; replacing one with an inferred number is the silent-fork failure §4a forbids. 6. 599 of 600 lines — any future addition must remove lines elsewhere. ## Parked questions none triggered the kickoff's park contract; the engineering-plugin harvested-verdict probe is recorded `PENDING` inside the patch (a stated dependency, not a parked fork). ## §1.7 Self-discipline check (REQUIRED if PR touches discipline-bearing files) ### §1.7 Skipped: research-patch + one append-only SSOT row — this PR introduces and extends no discipline rule, touches no `.claude/rules/**`, no `packages/core/principles/**`, no `CLAUDE.md` and no shipped template; the `prior-art-evaluations.md` touch is a single new register row (#233) in that file's own append-only schema, which is the register's normal operation, not a change to the discipline it encodes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Round-2 close-out for the token-economy dispatch inputs: both narrow cold Opus seats returned GO on the revised kickoffs at
d22b4f4014; this PR lands their three non-blocking nits (found by the seats, fixed inline per the operator's no-round-3 ruling) plus the ledger round-2 note.token-economy-research-s-a/kickoff.md— §2.1 share column now one-decimal and sums to 100.0% (was truncated, summed to 98%); W2 row-11 formula states its direction of error (213 assistant turns is an upper bound on user prompts → row 11 overstated).token-economy-research-s-b/kickoff.md— §4 descope now states explicitly that W4's «ranked next-stage proposals» is a different object than the forbidden candidate ranking (the K6 lexical tension the seat flagged).arch-v2-context-pipeline/calibration.md— round-2 note under Rows 2-3: GO/GO, watch-lists verified, shadow deliberately absent (narrow refresh, not a cohort run).§1.7 Self-discipline check
§1.7 Skipped: presentational one-liners from recorded round-2 seat findings + a ledger note; no rule, mechanism, or discipline artefact is created or changed (kickoffs are header-exempt transients per doc-authority-hierarchy.md §2).
Fidelity verdict
FIDELITY: skipped — not a stage PR: three one-line dispatch-input fixes and a ledger note from the interactive session; no aif substrate, no Provenance block, nothing dispatched yet to audit.