feat(eval): behavioral fixtures 5→50 + the eval:proposals runner#145
Merged
Conversation
PART A — the behavioral extraction eval grows from 5 to 50 cases under a 7-tag taxonomy (BehavioralTaxonomy), each case labeled and scored per its expectation: expect:'fact' cases stay on the value-quality axis (scoreFactValue — apples-to-apples with baseline.json's behavioral 40%); expect:'null' cases land on a NEW behavioral null-discipline axis (hard negatives that pass the marker gate + marker-less cases pinning the gate's deliberate safe-direction miss). runEval.ts reports both axes plus a per-taxonomy breakdown (latest.json gains behavioralNull + behavioralByTaxonomy; baseline.json still only via --write-baseline). fixtures.test.ts enforces, in CI: taxonomy well-formedness, gate alignment per tag, 4-gram disjointness vs FACT_SYSTEM_PROMPT, and NEW pairwise 4-gram disjointness across all extract fixtures (no near-duplicates). PART B — the eval:proposals runner deferred from #142 (spec §7 of docs/plans/executor-facts-pilot.md), split per the eval-metric-DOA lesson: - LIVE (npm run eval:proposals; opt-in, spawns the real executor CLI, burns quota, NEVER CI): wraps the behavioral cases as degenerate RunDigests (task = transcript) through buildProposalPrompt -> executorProposalSource -> parseProposals -> admitProposals, scoring the SAME scoreFactValue axis so the number lands beside the 3B baseline; plus the two real-shaped fixture digests. Channel axes: proposals-per-run + grounding-rejection rate. Writes proposalLatest.json (untracked); proposalBaseline.json only with --write-baseline. - CI (proposalFixtures.test.ts): the two digests replayed from src/executor/__fixtures__ through the production mappers + digest accumulator, pinned real-shaped, with parse+admit exercised on canned replies (grounded admitted; ungrounded/boolean/garbage dropped). admission.ts: the grounding check extracted into an exported isGroundedInDigest (used internally, behavior unchanged) so the eval measures grounding with admission's own predicate. Live re-baseline attempt (report only, baseline untouched): DECIDE 22/22, EXTRACT 10/10 — unchanged after the #139 prompt rewording. Behavioral value-quality 74.3% (26/35) on the expanded set; the original 5 cases reproduce the baseline's 40% (2/5) exactly. NEW null-discipline axis: hard negatives 2/10 — marker-bearing one-off instructions/questions make the 3B invent facts (the gate can't save those); marker-less 5/5 gated. Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d6e40623f1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const agent: AgentKind = argv.includes('--agent') && agentArg === 'claude' ? 'claude' : 'codex'; | ||
| const source = executorProposalSource(); | ||
|
|
||
| const factCases = BEHAVIORAL_EXTRACT_CASES.filter((c) => c.expect === 'fact'); |
There was a problem hiding this comment.
When npm run eval:proposals is run after this fixture expansion, this now scores every fact-expecting taxonomy, but the checked-in src/brain/eval/baseline.json still contains the old 5-case behavioral score that the runner prints as the 3B comparison. That makes the executor channel's value-quality percentage a different case mix from the displayed 40% (2/5) baseline, so experiment decisions based on this eval can be misleading; either update the committed baseline for the expanded set or restrict this comparison to the baseline case set.
Useful? React with 👍 / 👎.
Comment on lines
+116
to
+117
| await run(c, (admitted) => | ||
| scoreFactValue(c.valueContract, admitted.length > 0 ? { key: admitted[0].normalizedKey, value: admitted[0].value } : null)); |
There was a problem hiding this comment.
In cases where the executor returns two grounded proposals, admission can keep both, but this metric only checks admitted[0]. If the first proposal is generic or off-topic and the second one satisfies the fixture contract, the row is reported as a value-quality failure even though the proposal channel produced a usable fact for the review surface; score the admitted list with some(...) or track first-choice accuracy separately.
Useful? React with 👍 / 👎.
Review P1 on #145: value contracts were direction-blind. A topic token alone scores the EXACT OPPOSITE value ok — "force pushes to shared branches" satisfies {mustContainOneOf:['force','push']} against the fixture "we never force push to shared branches". An inverted memory recalled to the user is worse than a missed one, and the value-quality axis could not see it. ValueContract gains two conjunctive polarity guards (same whole-word- START matcher; violating either scores the NEW distinct mode 'inverted', never buried in off_topic, per the eval-metric-DOA lesson): - mustAlsoContainOneOf: the value must additionally carry a direction token ('never'/'not'/'green'/...). Multi-word tokens encode word ORDER ('over mysql' matches "postgres over mysql" but not "mysql over postgres"). - mustNotContainAnyOf: the value must not name the disfavored alternative ('jest' when vitest is the rule). Deliberately trades a rare false-inverted on an echoing-but-correct value for never scoring a wrong-direction value ok. 14 fixtures gained guards: the 4 named in review (bool-force-push, beh-green-ci-merge, np-db-postgres, multi-deno-quotes) + 10 more from a full negation-audit of the set (np-python-poetry, np-css-tailwind, beh-rebase-main, beh-comments-why, multi-hints-semicolons, multi-review-deploy, and all 4 supersede cases). fixtures.test.ts now asserts every negation-phrased fact fixture carries a polarity guard (sabotage-checked: removing a guard fails CI). score.test.ts pins the four review-named inverted values scoring 'inverted' and correct negation-carrying values scoring ok. Live calibration finding (report-only, baseline untouched): re-running eval:brain surfaced exactly one guard misfire — the 3B's CORRECT form for "postgres, never suggest mysql" is the comparison echo "prefers postgres over mysql" (deterministic at temp 0), which a bare mustNot ['mysql'] ban false-flags; recalibrated to mustAlsoContainOneOf with the order-encoding phrase token 'over mysql' and regression-pinned. After guards: zero genuinely inverted 3B values on this fixture set — value-quality stays 74.3% (26/35), null-discipline 46.7% (7/15). Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Polarity-guard fix pushed (8dab2ee) — addresses the merge-gate P1 (direction-blind value contracts).
Gate: tsc clean · 872 tests pass (serialized, +6) · eslint 0 errors. 🤖 Generated with Claude Code |
gorajing
added a commit
that referenced
this pull request
Jul 2, 2026
…st-rebuild architecture (#150) The W0-W8 rebuild (#134-#149) reshaped the codebase over two days; the canonical docs described an architecture that partly no longer exists. Reconcile them to the current tree (verified against src/core, src/main, packages/voice + the program log). - HANDOFF.md: rewrote the §2 Architecture (stack, component map, turnRun pipeline, fragile seams) to the Electron-free src/core layout + four platform ports + the Turn/Pump state machine + in-memory memIndex/vectors.jsonl memory (PGlite deleted) + the dark SDK executor + the executor-facts pilot + voice-in-packages; retitled invariants #1/#7; added a W0-W8 program work-log entry; updated the eval scorecard (5->50 fixtures, null-discipline axis, eval:proposals, inverted scorer mode) and its path; flagged MEMORY-ARCHITECTURE.md as pre-W5. - FOUNDING.md: fixed the three now-false locked invariants — turnRun chokepoint is the Turn/Pump machine, files-as-truth cache is the in-memory index + sidecar (not PGlite), point-don't-act now also has the SDK executor's pre-execution gate. - PUBLIC.md: added a competitive-landscape note (Codex Pets — the frame is commoditized; the moat is owned/correctable/portable memory); reinforced the Phase-0 non-founder signed-build magic-moment session as the go/no-go gate and more urgent now; added two founder gates (executor-facts live smoke; SDK-executor live smoke + auth-policy). - docs/ROADMAP.md: reconciled §1 (the engine is further ahead post-rebuild; the bottleneck remains signal/validation, not capability) + the Codex-Pets read; sharpened §8 (pet-first contingency near-dead; job->memory is effectively the only bet). - LESSONS.md: added five hard-won lessons (kept all existing content): * compaction/GC of an event-sourced log must never drop ops carrying cross-entry side effects (a fact's supersedeIds live on the successor's op) — W5/#147. * getAllWindows()[0] orders newest-first; target windows by a named registry — W0/#135. * a token-overlap value-quality metric scores an INVERTED memory as ok; the metric must see inversion (conjunctive/negation contracts) — W3/#145. * after merging a dependency-adding PR, re-sync the shared node_modules (npm install) or local tsc/package break silently while CI (npm ci) stays green. * never land a security feature on a review panel where the security-correctness dimension didn't complete — re-run the missing dimension — W6/#148. Doc-only: no source/config touched. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gorajing
added a commit
that referenced
this pull request
Jul 6, 2026
…gue fixtures, docs absorb the first dogfooding session (#162) * fix(eval): repair the eval runners' post-W7 artifact paths + pin the path family in CI Since the W7 core-split (#149) moved src/brain -> src/core/brain, both live runners still wrote/read their artifacts under the stale 'src/brain/eval' (runEval.ts latest/baseline writes; runProposalEval.ts baseline read + proposalLatest/proposalBaseline writes) and crashed ENOENT at the end of a full run. The directory now lives once in eval/paths.ts (EVAL_DIR + evalArtifactPath), both runners use it, and paths.test.ts pins that EVAL_DIR resolves to the directory the eval files actually live in (plus that the checked-in baseline.json exists there) — sabotage-verified: reverting EVAL_DIR to the stale path fails both tests. Also moves the two stale .gitignore scratch entries (latest.json / proposalLatest.json) to the real output location so runner scratch stays untracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): make deflect-on-vague measurable — 5 vague-initiative DECIDE fixtures (fixture-first) First founder dogfooding session (2026-07-05): 'create something interesting' made the 3B answer 'Let's brainstorm a fun project idea together' — a chat deflection that fails ROADMAP §6's lands-or-clarifies bar (do the task or ask a PERTINENT question). This adds the failure class to the eval WITHOUT touching decide (measure first, fix later): - DecideCase grows alsoAccept (a golden SET for lands-or-clarifies cases) + a 'vague-initiative' tag (the DECIDE-side analogue of BehavioralTaxonomy). - 5 fixtures (vague-interesting/-cool/-fun/-buildme/-impress) accept exactly {run_agent, clarify}; answer FAILS. Expected to fail against the current model — that is the point. - scoreDecision gains the optional alsoAccept set (red-first unit tests: a deflection still scores wrong_command). - fixtures.test.ts pins the class deterministically in CI: acceptance set is exactly {run_agent, clarify}, and every vague case must NOT be caught by the deterministic clarify gate (same loud-re-label discipline as the marker-less pin), so the fixtures keep measuring the MODEL. Live tier stays opt-in/report-only; the deterministic suites stay green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(eval): refresh the shape-stale baseline via --write-baseline (real Ollama, new fixtures included) OLLAMA_AVAILABLE=1 npm run eval:brain -- --write-baseline against local qwen2.5:3b — the sanctioned baseline workflow; baseline.json is never hand-edited. The old snapshot predated the W3 5→50 fixture expansion and the new runner output shape. Old → new headline numbers: - decide: 22/22 (100%) → 26/27 (96.3%; one wrong_command: run-writetest → answer — DECIDE streams at temperature 0.3, a run is a snapshot) - extract: 10/10 (100%) → 10/10 (100%), unchanged - behavioral value-quality: 2/5 (40%, the original 5-case set) → 26/35 (74.3%, the full expanded set — reproduces the #145 report-only number) - NEW keys now captured: behavioralNull 7/15 (46.7%; all 8 misses are hard-negative false_facts) + behavioralByTaxonomy (hard-negative 20% is the weak family; multi-fact/supersede/marker-less 100%) MEASURED TRUTH on the new vague-initiative cases: contrary to expectation, the current 3B lands-or-clarifies on all 5 (and 30/30 in a 3×-repeat probe with and without a memory section) — the fixtures therefore PIN the class against regression; the live 2026-07-05 deflection has not reproduced from paraphrased transcripts, so the fixture-block comment now records that and points the follow-up at promoting a RORO_TRACE capture of a real deflecting turn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: absorb the dogfooding + floating/moat signal — LESSONS, HANDOFF, ROADMAP, VERIFICATION - LESSONS.md: new Dogfooding entry (2026-07-05) naming the four failure classes — deflect-on-vague / feedback-fade / working-illegibility / env-overrides-config — each with its file:line locus and fix status (2–4: the sibling dogfood-blocker slice; deflect-on-vague: fixture-first, with the measured 2026-07-06 twist that the paraphrased fixtures currently PASS, so the real repro needs a RORO_TRACE capture). - HANDOFF.md: §3 work log extended past W8 — the floating redesign (#153–#160, resting purity + truthful receipt), earn-the-moat (#161, codex finalText stitch + verify:memory-steered-real 5/5 vs 0/5), the first dogfooding session, and this signal-reconcile slice; §4 gains the refreshed-baseline numbers (the old bullets stay as the historical snapshot); §8 gains the corrected forward order (dogfood fixes → signing → legible-moment → executor-facts pilot → self-rehearsal → demo → stranger session). - docs/ROADMAP.md: Arc A §6 step 5 (founder dogfooding) marked STARTED 2026-07-05 with the four findings + the fixture-covered status of deflect-on-vague. - docs/VERIFICATION.md: the stale pre-W7 layout references in the cohort trace-review section (lines ~10–27) now point at src/core/… (the vitest command there was unrunnable). Stale paths in LATER sections of the file remain — out of this slice's scope, flagged in the handoff report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(review): address the review panel — doc truthfulness + pin hardening (S-A round 1) Panel findings (all CONFIRMED, 0 false positives across the 5-hat review): - BLOCKER: four docs-of-record sites claimed feedback-fade was 'being fixed in the sibling dogfood-blocker slice' — it is NOT (it belongs to the upcoming legible-moment slice). LESSONS, HANDOFF x2, ROADMAP now say so honestly, and working-illegibility is split truthfully (sleep-mid-run half → dogfood-blockers; the distinct working pose → legible-moment). - LESSONS locus corrected: summoning the ask also pokes the presence clock, not only petting. - alsoAccept untag loophole closed: any DECIDE case widening its acceptance set must declare a failure-class tag (pinned in fixtures.test.ts) — an untagged alsoAccept was silent softening. - evalArtifactPath now __dirname-based (correct from ANY cwd, travels with the dir on a move); new source-grep pin: the runners must route through evalArtifactPath, no quoted hand-spelled eval dirs (the exact post-W7 breakage class), header-comment convention exempted. - Stale src/brain path sweep: cohortTraceReview.ts fixture pointer, COHORT_TRACE_TO_EVAL.md verify command, RUN.md:20, README.md x2, WS1 doc x2, ROADMAP live locus. tsc clean · 1104 passed / 12 skipped · eslint 0 errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: sweep stale PGlite/pgvector memory claims + the temp-0 eval claim (codex S-A round 1) Codex caught materially-false user-facing docs surviving the reconcile: README (6 sites) and RUN.md (4 sites) still described memory as an in-process PGlite+pgvector database at src/memory2 — deleted in W5 (#147); memory is encrypted files-as-truth + a rebuilt-on-launch in-memory vector index at src/core/memory2, embeddings via local nomic-embed-text. Also HANDOFF's eval header said 'temp 0' while decide actually streams at temperature 0.3 (brain/index.ts:190) — the same header the new baseline note's temp-0.3 explanation contradicted. And the crosslaunch proof pointer moved to src/core/orchestrator/. All claims now match the code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: complete the pre-core-split path sweep + fix dangerous embedder advice (codex S-A round 2) - RUN.md's embedder note was DANGEROUS post-W5: it described a fixed vector(N) DB column and told users to move/delete the memory dir when changing embedders — that dir now holds the durable encrypted memories (files-as-truth); vectors.jsonl is the zero-authority derived cache that re-embeds automatically. Rewrote to the true architecture. - README executor pointers (table + tree) → src/core/executor/. - VERIFICATION.md: ALL remaining pre-core-split paths fixed (memory2/executor/orchestrator test commands + module pointers); ROADMAP tracer pointer; PHASE2-TRUST-LOOP's runbook command fixed (adapter.test.ts was deleted in W5 — removed) and PROVEN to resolve (34 tests). - Exhaustive residual sweep across *.md + docs/: zero non-historical pre-split paths remain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: exists-checked path sweep — src/main→core/orchestrator (codex S-A round 3) Round-3 findings + a proactive exists-check sweep: ROADMAP's configStore/memoryContext pointers, the eval fixtures/score header comments, and 12 stale src/main/<file> references across the md corpus (all moved to src/core/orchestrator in W7) — now verified by an automated every-md-path-must-exist check returning zero stale refs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: last stragglers — PHASE2 adapter-successor pointer + eval-source comment paths (codex S-A round 4) PHASE2-TRUST-LOOP prose now names facade.test.ts as the successor to the W5-deleted adapter.test.ts; every src/brain|src/executor comment path inside src/core/brain/eval/* swept to core (grep-verified zero non-core refs remain). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: two path stragglers that escaped the exists-check shapes (codex S-A round 5) PHASE2's relative memory2/profileFacts pointer (no src/ prefix) and VERIFICATION's src/main/memoryHealth* wildcard — both shapes now covered in the sweep; ROADMAP's relative brain/index.ts ref fixed alongside. src/main/ipc*.test.ts verified VALID (ipc tests do live in src/main). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: precise privacy claim — the index is derived/plaintext, the FILES are encrypted (codex S-A round 6) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: embedder-switch remedy is delete-derived-index-only, not auto-re-embed (codex S-A round 7) vectorCache.ts:20 IDENTITY REFUSAL: a mismatched (embedModel, dim) header THROWS; the store does not silently re-embed. Remedy: delete the derived index/ subdir (zero-authority; next launch re-embeds from the durable encrypted files). My round-2 rewrite overstated the automation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: the PHASE2 gate command now runs facade.test.ts, matching its own prose (codex S-A round 8 — cap) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task 11 of the rebuild program: expand the behavioral extraction eval and build the
eval:proposalsrunner deferred from #142.Part A — behavioral fixtures 5 → 50, taxonomy-labeled
noun-preferencebehavioral-habithard-negativemarker-lessmulti-factboolean-collapsevalue:"true"— the guard nulls those, so a collapse scoresmissed_fact(the honest downstream outcome, per the eval-metric-DOA lesson)supersederunEval.tsnow scores two behavioral axes — value-quality (samebehavioralkey/axis asbaseline.json) and a new null-discipline axis — plus a per-taxonomy breakdown (behavioralNull,behavioralByTaxonomyinlatest.json). Baseline discipline unchanged:baseline.jsononly via--write-baseline.CI hygiene (
fixtures.test.ts): taxonomy well-formedness (every tag populated, fact-tags carry contracts, null-tags don't), gate alignment per tag (everything butmarker-lesspassesisPlausiblePreference;marker-lessmust fail it), 4-gram disjointness vsFACT_SYSTEM_PROMPT, and a new pairwise 4-gram disjointness across ALL extract fixtures (no near-duplicates). All pass.Part B — the
eval:proposalsrunner (spec: executor-facts-pilot.md §7)Split per the eval-metric-DOA lesson — protect production in CI, measure the model live:
npm run eval:proposals,-- --agent claudefor the claude CLI): opt-in, spawns the real executor CLI, burns provider quota, never wired into CI. Apples-to-apples tier wraps the behavioral cases as DEGENERATE RunDigests (task = transcript, empty commands/files/messages) through the production pipelinebuildProposalPrompt → executorProposalSource → parseProposals → admitProposals, scored on the SAMEscoreFactValueaxis so the number lands directly besidebaseline.json's 40%. Null cases score post-admission null-discipline (marker-lessexcluded — the 17-marker gate is a 3B-channel artifact; the executor extracting those would be a win, not a failure). Plus the two real-shaped fixture digests (golden[]— a scripted hello.py run teaches nothing durable). Channel axes: proposals-per-run and grounding-rejection rate (computed with admission's own predicate). WritesproposalLatest.json(gitignored);proposalBaseline.jsononly with--write-baseline.proposalFixtures.ts+proposalFixtures.test.ts): the codex v0.139.0 JSONL capture and the documented claude 2.1.x sample replayed through the production mappers +createDigestAccumulatorinto 2 pinned real-shaped digests, with the pure half (parse+admit) exercised on canned replies: grounded admitted, ungrounded dropped, boolean value dropped, garbage →[].admission.ts: grounding check extracted into exportedisGroundedInDigest(used internally; behavior identical — all 17 admission tests pass unchanged).Runner plumbing was smoke-tested quota-free (
RORO_CODEX_BIN=/nonexistent/codex): all 47 asks fail fast intoerrorrows, summaries + JSON writes verified. A real live run is left to the founder (it burns quota by design).Part C — live re-baseline attempt (report only;
baseline.jsonuntouched)Local Ollama was up (
qwen2.5:3b).npm run eval:brain(NO--write-baseline):marker-less5/5 (gate holds),hard-negative2/10Per-taxonomy: noun-preference 7/10 (3
too_thin— terse-but-honest values likenode_version="22"), behavioral-habit 7/12, multi-fact 4/4, boolean-collapse 4/5, supersede 4/4, hard-negative 2/10, marker-less 5/5.The headline finding: the 3B has essentially no null-discipline once a marker lets a turn through — 8/10 hard negatives became invented facts (e.g. the question "which formatter do we use in this project?" became
code_formatter="prettier", lifted from roro's own narration). The gate is currently the ONLY null-discipline in the pipeline; marker-bearing one-off instructions and questions are its blind spot. Founder decides whether/when to re-baseline.Surprise found while building (not fixed here — out of scope)
The claude mapper emits
fileson the STARTEDfile_changeevent andfiles: []on COMPLETED, whiledigest.tscollects files only from COMPLETED events — so claude-channel digests always havefiles: []and the proposal prompt's "FILES YOU CHANGED" section is always empty for claude runs. Pinned honestly inproposalFixtures.test.tswith a comment; worth a follow-up.DO NOT MERGE — for review.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD