docs(research-patch): RTK empirical test — measured 1.8% on our mix, DEFER confirmed (falsifier closed)

[artyhoo]

What

Empirical test of RTK (rtk-ai/rtk) on this project's actual workloads, closing the S-B candidate-1 falsifier with measurement rather than arithmetic. S-B projected 1.7–3.1% of total weighted cost (using the vendor's 89% claim); the falsifier was "a measured Bash-heavy session class on which RTK removes > 5% of total weighted cost → upgrade DEFER to ADOPT-on-operator."

Result — DEFER CONFIRMED (falsifier NOT triggered)

On this project's actual command mix (599 Bash calls across 5 top sessions, 5,149 turns):

• RTK saves 9.4% of Bash-output bytes (char-weighted).
• This translates to ≈ 1.77% of total resident-context weighted cost — below the 5% falsifier threshold and inside S-B's 1.7–3.1% arithmetic range.

The vendor 89% does not reproduce on our mix because:

1. 58% of our Bash bytes is already-piped head/tail output RTK cannot improve (the agent already self-truncates).
2. 71% of our Bash CALLS are compound commands (cd X && Y && Z) that RTK refuses to rewrite.
3. Caveat (c) fires in the opposite direction: on a failing test, RTK's fallback inflates output (+34%) by emitting full Node stack traces that raw vitest trims.

Independent corroboration

The JetBrains benchmark (blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings, 425 trials, Claude Code 2.1.201 + claude-sonnet-5, SkillsBench, paired A/B with Wilcoxon signed-rank) independently found RTK +7.6% more expensive at low reasoning effort (p=0.004), ±0% at high effort — «+13.8% more turns (p=0.03) and +14.3% more cache reads (p=0.008)». This confirms this patch's direction from a controlled method this patch did not use.

Outcome for the distillation seat

• Operator axis: DEFER (unchanged from S-B, now on measured ground).
• Shipped axis: REJECT (provisional) (unchanged — Rust binary violates agnosticism default).
• S-B proposal P4 reinforced: the agent already self-truncates via head/tail/grep; a skill codifying "pipe verbose test-runner output through tail -50 / grep -E '✓|✗|FAIL|Error'" captures the one real saving (verbose test dumps) without the binary. Flagged, not built.
• Headroom (headroomlabs-ai/headroom) emerged as a 2026 competitor with a broader surface (tool output + files + logs + history). WATCHLIST note for the token-economy umbrella; not evaluated here.

Hard boundaries honoured

• No SSOT row added/edited (refactor(audit-ai-docs): DI execSync for warm-path probeR4 tests + testTimeout 120s ceiling #233 on staging unchanged — verdict DEFER consistent with this measurement).
• Zero build (no install.sh, setup.d/, companions.manifest edits).
• RTK activated via wrapper (rtk <subcommand>), NOT rtk init -g (which would mutate host ~/.claude/settings.json — outside this patch's one-file boundary).
• Host-only test (RTK binary + transcripts on host); a container run would prove nothing about the host.

Fidelity verdict

FIDELITY: skipped — this is an empirical measurement patch closing a pre-existing falsifier (S-B candidate 1 §6); there is no kickoff, no deliverable contract, and no design content for a WHAT-conformance seat to audit against — the deliverable IS the measurement (599 Bash calls, 9.4% char-weighted saving, 1.77% total-cost share), which is self-evidencing in §A1–§A2 of the patch rather than a claim a fidelity seat would re-derive. A verdict block here would be decoration, not evidence.

Files

• docs/meta-factory/research-patches/2026-08-02-rtk-empirical-test.md (new, 362 lines, markdownlint-cli2 0 errors)

§1.7 Forward-check applied

This patch introduces no new rule/principle/skill/SSOT row — it measures an existing candidate (RTK, SSOT #233) against an existing falsifier. Checked against the active layer set: (1) code-level R1-R20 N/A — no TS/shell code (zero build); (2) principle-level N/A; (3) capability-commit gate — not a capability commit (one research .md, no tool/hook/skill), verified at docs/meta-factory/prior-art-evaluations.md:306 (row #233) where RTK is already registered; (4) SSOT — RTK is row #233, cited not re-added (grep -niE 'rtk' prior-art-evaluations.md → 1 hit); (5) §1.6 trigger sweep — this closes a falsifier not an open question, no §13.x cascade; (6) doc-authority — Authoritative for: header present, scope narrowly the RTK falsifier. docs/meta-factory/research-patches/2026-08-02-rtk-empirical-test.md:1 carries the <!-- scope:rtk-empirical-test-host --> first-line marker per patch convention.

§1.7 Backward-check applied

Swept the existing artefacts making a load-bearing claim that RTK's effect is unmeasured: (1) S-B candidate 1 (docs/meta-factory/research-patches/2026-08-01-token-economy-s-b-candidates.md on staging, candidate 1 §6) — its falsifier (>5% measured → ADOPT) is now measured-closed at 1.77%, DEFER confirmed; SWEPT-CLEAN, cited not edited (on staging). (2) SSOT row #233 (docs/meta-factory/prior-art-evaluations.md:306 on staging) — verdict DEFER, evidence 1.7–3.1%; this patch's 1.77% is consistent → no verdict change → row NOT edited (hard boundary); its second branch (config-only equivalent P4) reinforced. (3) S-A §A4.3 Bash resident-cost share (18.9%) — cited verbatim at docs/meta-factory/research-patches/2026-08-01-token-economy-s-a-profile.md §A4.3, not re-derived; its uniform-arrival caveat honoured in patch §A5. Exemption mechanism: none needed — one cold-archive research file, gated by no paths: glob. docs/meta-factory/research-patches/2026-08-02-rtk-empirical-test.md:351 (§A5 coverage) states the direction-of-error on the one inferred figure.