Roadmap proposal: from session evidence to a verifiable Harness evolution loop

[phodal]

Summary

Better Harness already has important pieces of a learning loop, but they are intentionally separate:

• Task Episodes and evidence states define the review unit and distinguish Present, Wired, Exercised, and Outcome-supported evidence.
• Native Learning Candidate Mining can review evidence-bound recurring-correction candidates without mutating Harness assets.
• The Intervention Ledger already models baselines, comparison windows, guardrails, stop/revert conditions, and later results.
• Harness Run Checkpoints are a Draft Git-neutral run-continuity design. They are state anchors for Harness artifacts, not code rollback, transcript storage, Git checkpoints, or native-session resume.

The next step should not be an all-in-one “Coding Session Intelligence” subsystem. It should be one thin, auditable evidence chain:

TaskEpisode + delivery/outcome links
    -> PatternCandidate
    -> human/AI review
    -> HarnessIssue
    -> bounded Harness intervention
    -> later comparable outcome
    -> retain / narrow / revert / reopen

The proposal below turns the external reference map into an incremental Better Harness roadmap.

Coding Session Intelligence: external reference map

This table is the external reference map behind the roadmap. These sources are design inputs, not product requirements: Better Harness should adopt only the parts that fit its existing evidence, privacy, authority, and cross-platform contracts.

Better Harness concern	Core references	What to study	Design that can be reused
1. Bind Sessions to code changes	Entire: Introduction; Git AI: How Git AI Works	How Entire separates Sessions, Checkpoints, and Commits while retaining prompts, tool calls, and file changes; how Git AI uses agent hooks for checkpoints and Git Notes for line-level agent, model, prompt, and Session attribution.	Build a deterministic TaskEpisode -> Diff -> Commit relationship. Prefer typed Hook, checkpoint metadata, commit trailer, or Git Note evidence; never infer the relationship from timestamp proximity or LLM similarity alone.
2. Discover Skills across multiple Sessions	SpecStory/Lore: Assessing	Lore separates latent/theme skills from correlation skills. Candidates retain evidence Sessions, prevalence, cross-vendor coverage, outcome lift, confidence, and a human Keep/Dismiss decision.	Separate repeated procedures from latent working practices. Every candidate should retain source Episodes, counterexamples, coverage, confidence, and missing evidence. Review before generating or installing a Skill.
3. Extract Workflows from noisy traces	TraceCompiler	How it separates accidental ordering from real data dependencies: a hard edge exists only when a later argument contains a value uniquely attributable to an earlier tool output; ambiguous edges remain suspected. Bindings are constants, user inputs, previous outputs, transforms, or residual LLM decisions.	Do not mine only Tool A -> Tool B -> Tool C. Extract both an Action Skeleton and parameter dataflow. Compile only stable, auditable dependencies; leave underdetermined choices to the agent.
4. Turn recurring patterns into durable Issues	LangSmith Engine	How related traces become an Issue with linked traces, diagnosis, priority, proposed fix, evaluation cases, and automatic reopen when the problem returns.	Keep PatternCandidate, report-local Finding, and longitudinal HarnessIssue separate. Issues need Open, Watch, Resolved, Dismissed, and Reopened states with deterministic evidence transitions.
5. Explain why agent work is difficult	DX: Introducing Agent Experience	Per-Session Requirements, Steering, and Scope assessments, each with a qualitative explanation.	Use these as semantic facets for goal clarity, quality of correction, and scope drift. Agent Experience is supplementary evidence; it cannot replace tests, Diffs, Commits, or user acceptance.
6. Connect Sessions to real engineering outcomes	Faros: Harness Engineering; Git AI	Session-to-PR linkage as the prerequisite for metrics such as cost per merged PR, first-pass success, Agent PR survival, churn, and defect escape; explicit attribution shows whether agent-produced code survives review and production use.	Use an outcome ladder: Session completed -> validation passed -> Commit -> PR -> CI -> Merge -> later survival or Revert. Start with Commit, then add optional delivery adapters.
7. Merge Session experience into a Skill draft	Trace2Skill	Independent trajectory-local lesson extraction followed by hierarchical aggregation, deduplication, and conflict resolution, instead of sequentially appending every trace into one document.	Use single-trajectory analysis -> cross-trajectory aggregation -> conflict resolution -> Skill Draft. Never generate SKILL.md directly from one successful Session.
8. Prove that a Skill or Harness change works	SkillOpt; Agentic Harness Engineering; Rethinking the Evaluation of Harness Evolution	SkillOpt accepts bounded Add/Delete/Replace edits only after strict held-out improvement. AHE pairs component, experience, and decision observability with predictions and results. Rethinking Harness Evolution requires matched feedback/inference budgets and simple search baselines to expose overfitting or gains caused only by extra search.	Maintain Discovery, Held-out, and Later Comparable sets. Change one Harness component per experiment. Compare Current vs Candidate and, where relevant, No Skill plus budget-matched retry or sampling baselines. Retain, Narrow, or Revert from evidence.
9. Enforce safety boundaries for generated Skills	Agent Skills in the Wild	Large-scale evidence of prompt injection, data exfiltration, privilege escalation, and supply-chain risks, with higher risk for Skills that bundle executable scripts.	Generated Skills enter Draft/Experimental only. Require secret scanning, prompt-injection review, script static analysis, tool-permission declarations, and external side-effect declarations before any explicitly authorized activation.

P0 reading priorities

Priority	References	Direct value to Better Harness
P0	Entire + Git AI	Define the factual relationship among Session, Checkpoint evidence, Diff, and Commit without conflating Better Harness run checkpoints with Git checkpoints.
P0	SpecStory/Lore	Define multi-Session Skill candidates, confidence signals, provenance, counterexamples, and human promotion.
P0	TraceCompiler	Define how to extract an executable Workflow from Tool Call traces without copying accidental call order.
P0	LangSmith Engine	Define how a temporary recurring pattern becomes a durable, diagnosable, resolvable, and reopenable Issue.
P0	SkillOpt + AHE + Rethinking Harness Evolution	Define how to test whether a Harness change actually improves outcomes without overfitting or unfair search-budget comparisons.

The five primary reference lines map directly to the roadmap:

Entire / Git AI
    -> Session and code-change facts

SpecStory / Lore
    -> multi-Session PatternCandidate

TraceCompiler
    -> Action Skeleton and Workflow extraction

LangSmith Engine
    -> durable HarnessIssue lifecycle

SkillOpt / AHE / Rethinking Harness Evolution
    -> evaluate, retain, narrow, or revert

DX, Faros, Trace2Skill, and Skill Security then extend the loop with agent-experience facets, real engineering outcomes, cross-trajectory asset drafting, and activation governance.

Domain boundaries

The most important design decision is to keep five concepts separate:

Concept	Lifetime	Meaning
TaskEpisode	one task goal	What happened for one goal, target, action, and result
PatternCandidate	one discovery window	A reviewed hypothesis supported by multiple Episodes; not yet a durable problem or a Skill
Finding	one Harness report	A report-local, evidence-backed consequence; it does not own longitudinal state
HarnessIssue	across runs/windows	A durable identity for a recurring problem or opportunity, with Open/Watch/Resolved/Dismissed/Reopened state
InterventionLedgerEntry	one controlled change experiment	What component changed, the baseline, comparison plan, guardrails, and later outcome

A checkpoint remains outside this chain. It may seal references needed to continue a Harness run, but it must not become the identity of a code change, Session, Issue, or intervention.

Proposed roadmap

Increment 1 — Establish the fact layer: Session → code change → Commit

Start with a read-only, privacy-safe link projection for local commits.

Each link should retain:

• explicit evidence: host-provided linkage, an allowlisted commit trailer, checkpoint metadata, or another typed integration signal;
• heuristic evidence: bounded time overlap, repo-relative file overlap, and workspace identity;
• evidence state and confidence;
• the exact Session/TaskEpisode and Commit refs used by the decision.

Heuristic correlation must remain a candidate link, never silently become a fact. Better Harness should not initially write Git Notes, trailers, refs, or shadow branches. A future host integration may opt into an explicit writer, but the core reader should remain Git-neutral and cross-platform.

The first outcome boundary should stop at Commit. PR, CI, Merge, Revert, and survival can be added later through optional delivery adapters.

Why first: without deterministic provenance, later pattern mining cannot tell which behavior produced which code or outcome. This follows the useful part of Entire while avoiding adoption of its Git storage model.

Increment 2 — Expand candidate mining without copying noisy traces

Keep the existing evidence-bound review packet and extend it from the first recurring-correction slice into two candidate families:

1. Procedure candidates — repeated actions that may become a Workflow, Skill, Hook, Rule, or script.
2. Latent-practice candidates — a recurring way of working that correlates with better or worse outcomes.

Every candidate should carry:

• source Episode refs and evidence aliases;
• positive and hard-negative examples;
• prevalence and coverage;
• provider/host distribution;
• observed outcome association, explicitly labelled as correlation rather than causation;
• confidence and missing evidence;
• a stable pattern signature;
• a Review decision: Keep, Watch, Dismiss, or Needs more evidence.

For procedure candidates, do not infer a Workflow from adjacency such as Tool A -> Tool B -> Tool C. Following TraceCompiler, extract both an action skeleton and parameter provenance:

• constant;
• user input;
• uniquely attributable previous output;
• deterministic transform;
• residual agent judgment.

Only uniquely attributable producer-consumer relations become hard dependencies. Ambiguous relations stay suspected and impose no execution ordering.

This extends the current LC-05 direction and aligns with SpecStory/Lore: candidates remain reviewable evidence, not auto-installed assets.

Increment 3 — Add a durable HarnessIssue lifecycle

Today, a PatternCandidate is temporary and a Finding belongs to one report. We need a third object for longitudinal ownership.

A minimal HarnessIssueV1 could contain:

• stable issueId and pattern signature;
• kind, scope, first seen, last seen, and recurrence count;
• linked candidates, Task Episodes, report Findings, and outcome refs;
• diagnosis state and proposed durable owner;
• priority and evidence coverage;
• state: Open, Watch, Resolved, Dismissed, or Reopened;
• resolution evidence and deterministic reopen rule.

Reopen should require a later eligible Episode matching the issue signature after resolution; a title similarity or model judgment alone is insufficient.

The Issue store should be provider-neutral state outside the worktree, with bounded listing and privacy-safe references. It should have its own lifecycle owner and must not widen HarnessCheckpointV1 into an Issue database.

LangSmith Engine is a useful reference for the detect → diagnose → fix → evaluate → reopen lifecycle, but Better Harness should keep mutation, scheduling, and external PR creation explicitly authorized.

Increment 4 — Turn reviewed issues into Draft interventions, then evaluate them

A reviewed Issue may propose the smallest durable owner: Rule, Skill, Hook, Gate, Workflow, script, Eval, or no new asset.

For generated Skills or Workflows:

1. analyze each eligible trajectory independently;
2. aggregate lessons across trajectories;
3. preserve conflicts and counterexamples;
4. produce a Draft/Experimental artifact;
5. run secret, prompt-injection, permission, script, and side-effect checks;
6. require explicit review before installation or activation.

This follows the aggregation direction in Trace2Skill, while respecting the risks reported by Agent Skills in the Wild. Generated executable assets must never be trusted merely because generation succeeded.

Evaluation should reuse and tighten the existing Intervention Ledger:

• Discovery set: Episodes that produced the candidate;
• Held-out set: eligible Episodes not used to produce the change;
• Later comparable set: future real work with a comparable task mix;
• Baselines: Current, No Skill/No intervention, and a budget-matched simple retry or sampling baseline where relevant;
• Change discipline: one Harness component per experiment;
• Decision: Retain, Narrow, Revert, or Needs more evidence.

A candidate is not “effective” because it passed its discovery fixtures. It becomes Outcome-supported only after the held-out and later-comparable contracts are satisfied. SkillOpt is a useful reference for bounded edits and strict held-out improvement, but Better Harness should retain explicit cost, safety, and transfer guardrails.

Increment 5 — Extend outcome linkage beyond Commit

After local Session-to-Commit linkage is stable, add optional outcome adapters in this order:

Session completed
  -> validation passed
  -> Commit
  -> PR
  -> CI
  -> Merge
  -> later survival / churn / revert / defect escape

Every transition needs a typed evidence owner. Missing external integration remains Unobserved; it must not be inferred from branch names, timestamps, or green local tests.

This is where measures such as first-pass success, merged-PR cost, churn, and agent-PR survival become meaningful. They should not block the earlier local loop.

Recommended first vertical slice

Rather than implement all five increments at once, I suggest one end-to-end slice:

1. Link bounded Session/TaskEpisode evidence to one local Commit with explicit versus heuristic evidence.
2. Mine one existing recurring-correction candidate from at least two eligible Episodes.
3. Promote a reviewed candidate to one durable HarnessIssue.
4. Record one explicitly authorized Rule or Skill intervention in the existing ledger.
5. Evaluate one later comparable Episode.
6. Resolve, retain on Watch, revert, or reopen the Issue from evidence.

Success means we can explain every transition with typed refs and reproduce the state change. It does not require hosted telemetry, automatic Skill installation, Git mutation, or automatic PR creation.

Non-goals and guardrails

• No raw transcript lake or cross-provider transcript merge by default.
• No Session-to-Commit claim based only on timestamp proximity.
• No direct PatternCandidate -> SKILL.md generation from one successful Session.
• No automatic installation or activation of generated assets.
• No causal claim from prevalence or outcome correlation alone.
• No reuse of Harness checkpoints as Git checkpoints, code rollback, transcript storage, or native-session resume.
• No automatic external writes, schedules, PRs, or issue mutations without task-local authority.
• No “effective” state without comparable evidence and guardrails.

Questions for the community

1. Should the first Session-to-Commit capability remain read-only, with explicit link writing left to host integrations?
2. Is a separate durable HarnessIssue the right boundary, or should longitudinal state extend an existing contract?
3. Should procedure and latent-practice candidates share one envelope with different evidence requirements, or use separate schemas?
4. What is the minimum promotion gate: two distinct Episodes, two distinct Sessions, cross-provider evidence, or a configurable policy?
5. Which outcome adapter should follow Commit first: PR/CI linkage or later code-survival/revert evidence?
6. For generated assets, should Better Harness stop at a Draft artifact, or also provide an explicitly authorized Experimental install path?

My recommendation is to agree on the domain boundaries and the first vertical slice before writing a broad PRD. Each increment can then get its own reviewable spec and acceptance scenarios.