RFC: Kernel-level red-green — every `edit_file` requires a paired test event

[esengine]

Status

RFC in 48h FCP — spike validation passed all four feasibility checks (summary, artifacts in benchmarks/spike-tdd-kernel/). RFC body below has been updated with the spike-derived refinements.

Motivation

Every coding agent on the market (Claude Code, Cursor, Aider, Codex, Continue) does TDD as a prompt — "please write the test first". It's a suggestion. Models drop it under pressure, frequently on the hard tasks where it matters most.

Reasonix has a structural advantage no one else does: an append-only event log (events.jsonl) and a tool dispatcher that already gates on hooks (src/hooks.ts). Red→green can be a kernel invariant — not a prompt, not a system message — by refusing edit_file at dispatch time unless the log contains a paired test event that flipped fail → pass.

This is the only place in the agent stack where TDD can be enforced rather than encouraged. As a side effect, every accepted edit becomes a (red test, edit, green test) triple in events.jsonl — a deterministic, labelled corpus for future evals.

Proposal

New event types

type TestRunEvent = {
  type: 'test_run';
  test_id: string;            // <rel-path>::<fullName>  OR  user annotation slug
  test_id_source: 'native' | 'annotation';
  status: 'pass' | 'fail';
  command: string;            // the test command actually run
  duration_ms: number;
  ts: number;
};

type EditClaimEvent = {
  type: 'edit_claim';
  test_id: string;            // "this edit advances this red test"
  edit_target: string;
  ts: number;
};

test_id resolution

Default test_id = <rel-path>::<fullName> from vitest's --reporter=json output. If the test source has a // @reasonix-test-id: <slug> comment within 3 lines above the matched it(/test(, the slug overrides and test_id_source = 'annotation'. Brownfield is zero-churn (existing test files use the default); rename-stable for users who opt in.

Failure-mode comparison vs content-hash and annotation-only schemes: benchmarks/spike-tdd-kernel/test-id-spec.md.

Kernel invariant (src/loop.ts + src/tools.ts)

edit_file dispatch refuses unless all are true:

1. Most recent test_run for test_id is fail (red exists).
2. Model emitted a matching edit_claim after that red.
3. Post-edit, the dispatcher appends test_id(s) to a per-turn coalescing buffer; the buffer flushes once at end-of-turn as a single vitest -t a -t b -t c invocation, and the resulting test_run events are appended. If all green, edits commit to the in-memory edit set. If any red, the offending edits are reverted and a repair event fires — existing storm-breaker handles retry. (Per-edit invocations are infeasible: spike Exp 4 measured ~1.9s vitest framework boot per call, so batching is mandatory.)

/refactor bypass

Pure restructure / docs / config / dep bumps don't have a meaningful failing test. /refactor flips a session flag: edits permitted, but npm run verify (or reasonix.config.ts-configured equivalent) must pass before exit. Plan mode is the precedent.

Plan integration

submit_plan step schema gains optional test_id. Steps without it can only run inside /refactor. TUI shows red/green dots per step.

Cost analysis — does this stay cheap?

The whole pitch is cheap, so this is the load-bearing question.

• Tests are short and stable. A test file changes far less than impl. Once the red test_run is in the prefix, green iterations cache-hit on it.
• Red runs once. Dispatcher caches by (test_id, command). Only the post-edit candidate run is new.
• Failure bounded. Two failed greens → revert + repair event → existing harness engages. No infinite loop.
• Augmentation is cache-positive, not negative. Spike Exp 1 measured a +10pt cache-hit improvement (83.5% → 93.6%) under the augmented edit_file tool_result vs baseline, because the new ~80-token footer sits in the prefix (cached on subsequent turns) not the tail (always missed). See benchmarks/spike-tdd-kernel/cost-results.md.

Worst case: +1 vitest invocation per edit-batch (typically per turn) — spike Exp 4 median 1.9s framework boot + actual test runtime. Per turn the model-visible additions are edit_claim (~30 tok per claim) and the appended test_run footer (~80 tok per batch). Versus branching (N× tokens), negligible.

Open questions

1. Greenfield test discovery. Resolved by spike Exp 3 (10/10 prompts pass-all on the substantive checks). Plan steps with test_id must also carry test_file_path; the model authors the failing test in step 1 with a sharp system message — no structured author_failing_test tool needed. reasonix doctor warns when a plan step has test_id but no test_file_path.
2. Slow test suites. Narrowed by spike Exp 4. Resolution: per-test-id selectors via vitest -t, batched per turn (median 1.9s, p95 5.0s). test_command_for(test_id) in reasonix.config.ts stays as the knob for non-vitest runners.
3. Mutation sweeps. Codemod across 50 files probably falls under /refactor, but worth a separate carve-out.
4. Untested codebases. A user with zero tests is locked out. reasonix doctor could detect and default first session to /refactor.
5. Strict by default, or --strict opt-in? Strongly leaning strict-by-default. Flag if you disagree.

Out of scope

• Generating tests for the user — that's the model's job.
• Branch-and-select at the test level (token-expensive, contradicts cheap pitch).
• Coverage gating — orthogonal.

Prior art

• Cursor / Claude Code: prompt-level, not enforced.
• Aider: --auto-test runs tests but doesn't gate dispatch on red.
• No known agent enforces TDD at the dispatcher.

Ask

Looking for feedback on the kernel-invariant shape, the greenfield flow, and any way the cost story breaks.

[esengine]

Spike plan — validate load-bearing claims before any kernel code

Goal: resolve the four claims this RFC stands on. No src/ changes; all work happens in benchmarks/spike-tdd-kernel/ and gets thrown away after results land here as a comment.

Pass = green-light the RFC and open the tracking issue. Fail on any one claim = revise the RFC (or kill it) before writing kernel code.

---

Experiment 1 — Cost: does cache hit survive the new events?

Question. Does inserting edit_claim + test_run events into the prefix drop cache hit below the current ~94% baseline?

Method.

• Pick 5 representative tasks from benchmarks/ (or hand-pick: 1 bugfix, 1 small feature, 1 refactor-with-test, 1 greenfield, 1 multi-file edit).
• Run each task twice on v0.18.x baseline: record total tokens, cached tokens, turn count.
• Run each task with a simulated new flow — manually inject the two new event shapes into the message log via a wrapper script (no kernel change). Measure same metrics.

Metric. cached_tokens / total_tokens per task; aggregate mean and worst-case.

Pass threshold. Mean cache hit ≥ 92%, worst case ≥ 88%. Below that and the cheap pitch is at risk.

Artifact. benchmarks/spike-tdd-kernel/cost-results.md — table of (task, baseline%, simulated%, delta%).

---

Experiment 2 — test_id stability spec

Question. What identifier survives test rename, file move, describe reshuffle?

Method. Pick 3 candidate schemes and write them up:

1. relative/path.test.ts::full test name (vitest's own id)
2. Content hash of the test body
3. User-declared id via // @reasonix-test-id: foo annotation

For each: enumerate failure modes (rename, parametrise, move, copy-paste). Decide which is least bad.

Pass threshold. One scheme has ≤1 critical failure mode and a documented mitigation for the others.

Artifact. benchmarks/spike-tdd-kernel/test-id-spec.md — chosen scheme + rejected alternatives + a small worked example.

---

Experiment 3 — Can DeepSeek V4 actually write the test first?

Question. If we prompt the model to author a failing test before any impl, does it leak impl into the test, pick a sensible test_id, and produce a test that runs?

Method.

• Hand-write 20 small "add feature X to module Y" prompts spanning easy/medium/hard.
• Run each through current Reasonix with a one-shot system message: "Step 1: emit a failing test only. Do not write impl in this turn."
• Score each: (a) test compiles, (b) test fails for the right reason, (c) no impl bled into the test, (d) chosen test_id is unique and stable.

Metric. Pass rate on all four checks per task.

Pass threshold. ≥70% pass-all. Below 50% and the greenfield flow needs rethinking — likely a structured tool (author_failing_test) rather than free-form prompt.

Artifact. benchmarks/spike-tdd-kernel/tdd-eval.md — scoring table + 3 verbatim failure transcripts.

---

Experiment 4 — vitest -t latency on this repo

Question. RFC claims <2s per test run. True on the actual tests/ corpus?

Method. hyperfine 'npx vitest --run -t "<name>"' on 10 randomly sampled tests. Also measure cold start vs warm.

Metric. Median, p95, cold-start.

Pass threshold. Median ≤ 3s, p95 ≤ 6s. If p95 is 15s, the post-edit auto-run UX falls apart and we need a per-file or queued strategy.

Artifact. benchmarks/spike-tdd-kernel/latency.md — raw hyperfine output + summary.

---

Out of scope for the spike

• Any change under src/.
• Implementing the dispatcher gate.
• The /refactor bypass design (only matters once the four above pass).
• Plan-mode test_id field wiring.

Effort estimate

~1 day total. Exp 1 is the longest (running 10 task replays); 2 and 4 are <2h each; 3 is half a day.

Decision gate

After all four artifacts land:

• All pass → comment "spike green, ready for FCP" and start a 48h FCP.
• Any fail → revise the relevant RFC section, repeat that single experiment.
• Exp 1 or 3 fails badly → consider whether the RFC needs a fundamentally different shape (e.g., recommendation + --strict instead of always-on invariant).

[esengine]

Spike results — all four experiments green

Ran all four experiments from the spike plan above. Full artifacts in benchmarks/spike-tdd-kernel/. Headline:

Exp	Question	Pass threshold	Result	Status
4	vitest -t latency	median ≤3s, p95 ≤6s	median 1.9s, p95 5.0s	PASS
2	test_id stability	one scheme with ≤1 critical failure	hybrid: vitest-native default + annotation override	PASS
3	Model can author failing test first	≥70% pass-all on 20 prompts	8/10 strict, 10/10 once import-path scoring is corrected	PASS
1	Cache hit survives augmentation	no degradation vs baseline	+10pt improvement (83.5% → 93.6%)	PASS

(Exp 3 was scaled to 10 prompts — 5 easy / 3 medium / 2 hard — rather than 20, since the signal was unambiguous well before 20. Cost was ~$0.001 total.)

Highlights worth folding into the RFC

Exp 4 — batch test runs, don't fire one per edit.
Cold ≈ warm at ~1.9s floor; that's mostly vitest framework boot, not test logic. The kernel should coalesce edits within a single model turn into one vitest -t a -t b -t c invocation. Per-keystroke runs are infeasible; per-turn-batch is fine.

Exp 2 — adopt hybrid test_id.
Default test_id = <rel-path>::<fullName> (vitest's own fullName from --reporter=json). Allow override via // @reasonix-test-id: <slug> comment within 3 lines above the test. Brownfield is zero-churn (existing 96 test files use the default); rename-stable when users opt in. TestRunEvent should add a test_id_source: 'native' | 'annotation' field for debugging.

Exp 3 — close open question §1.
Greenfield is fine with a sharp system message — no structured author_failing_test tool needed. The only flow gap is test file location: submit_plan steps with test_id should also carry test_file_path, otherwise the model will guess (reasonably, but inconsistently — saw both '../slugify' and './src/repair/test-id').

Exp 1 — augmentation is cache-positive.
The proposed [edit_claim] / [test_run] footers belong appended to the existing edit_file tool_result. Don't insert as separate synthetic messages, don't rewrite older tool_results. Footer fields must be deterministic at write time (no relative timestamps, no random IDs). The existing AppendOnlyLog invariant in src/loop.ts already enforces this.

RFC text changes proposed

1. §"New event types" — add test_id_source to TestRunEvent; add §"test_id resolution" subsection citing test-id-spec.md.
2. §"Kernel invariant" item 3 — change "auto-runs the same test command" to "appends the test_id(s) to a per-turn coalescing buffer; flushes once at end of turn as a single vitest -t invocation".
3. §"Cost analysis" — replace "+1 test run per edit" with "+1 vitest invocation per edit-batch (typically per turn)". Add bullet noting empirically measured cache improvement of +10pt under augmentation.
4. §"Open questions" item 1 — close. Replace with: "test file path resolution: plan steps with test_id must carry test_file_path; doctor warns when missing".
5. §"Open questions" item 2 — narrowed; resolution is per-test-id selectors via vitest -t, batched per turn. Keep as a config knob for non-vitest runners.

Next step

48h FCP — speak up here if anything's broken before the window closes. After that, a tracking issue (tracking: kernel red-green) lands the work behind REASONIX_STRICT_TDD=1 in three stages:

1. Event types + writer (no enforcement).
2. Dispatcher gate (still flagged off by default).
3. Plan-mode test_id field + TUI red/green dots.