feat(super): live repo reads — read worktree tracking the integration tip#23
Merged
Conversation
…ktree tracking the integration tip The grounded-answer follow-up: the supervisor's built-but-dark read/search tools (already advertised in its prompt) now operate over a long-lived READ worktree re-pointed at the integration tip, so it reads the CURRENT integrated tree (file CONTENTS, not just diffs) when it reasons, and the answer is repo-structure-aware. - internal/worktree: Checkout(ctx, ref) (hardened git checkout --detach --force, I4) + ListFiles(ctx, maxBytes) (bounded git ls-files). - internal/super: Supervisor.RefreshRead seam + RunContext.Tree + runState.tree; refreshAndPublish re-points the read tree at the tip (single-owner, main goroutine) and folds its file-list into the snapshot at every st.branch change (doSpawn fold, runSpawnWave fold, doIntegrate, doCode). Run-start + doPlan keep the bare publish. The reader still does ZERO live host I/O — it answers off the by-value snapshot, so deadlock-freedom is intact. - cmd/nilcore: buildStack creates the read worktree off base HEAD, wires ReadTools=tools.ReadOnly()/ReadDir/RefreshRead (RefreshRead Checkouts + ListFiles under gitMu), best-effort (create failure => no read tools, logged super_read_skip). renderGrounding renders the trusted file-tree. buildAssembly.cleanup (deferred by build/chat/serve) removes the read worktree and sweeps the run's throwaway branches (task/rebase/integrate/read) — no leak in a long serve session. Adversarial-review fix (was HIGH): buildIntegrateFunc and buildCodeFunc returned their tip branch but `defer wt.Cleanup()` DELETED it before returning, so RefreshRead's Checkout(integrate/<suffix>) always failed and the headline integrated-tip read silently no-op'd (only the spawn paths, which Release+keep their branch, worked). Both now `defer wt.Release()` (keep the branch); the run-end sweep reclaims them. This also fixes a latent issue: the project loop's promote-time Differ diffs the same branch. New hermetic end-to-end test drives the REAL buildIntegrateFunc and asserts the tip branch survives + is readable via Checkout/ListFiles. Honest boundary: the supervisor reads the INTEGRATION TREE, never a subagent's in-progress private worktree. Tests (-race): worktree.Checkout/ListFiles; refreshAndPublish; the file-tree in the grounded answer; integrate-tip-survives. make verify + golangci-lint (0 issues) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RNT56
added a commit
that referenced
this pull request
Jul 5, 2026
… fixes (5 HIGH) (#95) * feat(upgrade): wire the dormant loops — batch 1 (caching, lessons, race evidence, trust fold, bounded results) Four judge-ranked items from the upgrade program, each closing a shipped-but- never-consumed loop: - provider/anthropic: emit the 3 prompt-cache breakpoints (system, last tool, moving last-message marker) the pricer already accounts for; fold Anthropic's disjoint usage tallies into the canonical total-input convention (the dormant under-billing the activation would have exposed). Complete+Stream share the request path; edge cases (empty system, zero tools, uncacheable final block) keep requests valid. - memory: TaskContext merges ScopeProject + ScopeGlobal (lessons) under one budget, newest-first retention; main.go MemoryContext closure uses it — the distilled-lessons pipeline finally reaches a prompt (I7 labels byte-identical). - agent/trust: race escalation threads the failed attempt's fail-class + fenced verifier tail into each candidate's Constraints (frozen contract, existing field — I1); final_verify is class-tagged and trust.Replay folds it alongside race_outcome (skip-not-panic guards; class-less pre-upgrade events replay byte-identical), so per-class trust warms on single-backend deployments. - tools/mcp: read gains {offset,limit} paging + a 48KB cap with a recovery- teaching notice; the mcp tool result is capped at the one seam all three success paths share. Both mirror the webfetch cap precedent. make verify green (119 pkgs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(upgrade): batch 2 — tiered/flake-aware verification, sandbox credential masking, swarm DAG completion, core-loop context lifecycle Four judge-ranked batches: - verify + cmd/verifier: NILCORE_VCACHE default-on (kernelEnabled idiom, =0 hatch) killing the 2x verify tax; FlakeProbe decorator (test-class red + identical content hash => one real-verifier re-run; verify_flaky event; never guesses); TieredVerifier (scoped go vet/test of touched packages short-circuits RED before the full gate; only Full can PASS; word-boundary soundness gate so it wraps only go-test-family verify commands; under-scoping falls through). Order: tiered(flakeprobe(vcache(base))). - sandbox: the namespace backend masks operator credentials before Landlock — vault dir + ~/.ssh/.aws/.gnupg/.netrc/.config/gh/.docker/config.json/.codex/ .claude* (ro tmpfs over dirs, /dev/null over files, fail-closed 126, worktree shadow-exclusion). Closes the in-box vault-decryption I3 hole; parent resolves roots via stripped control env vars. - swarm: Shard.BaseRef + dependency-aware base resolution (dep branch / TipSHA) with fenced claim-status+summary handoff (I7) — DAG edges now buy the dep's CODE, not just ordering; merged-set tracking (no re-merge spam) + conflict requeue on the integrated tip under the requeue Ledger; termination honesty (Outcome.Unmerged folds into Remaining — a dropped green can never exit 0); evidence-carrying focused requeue (red keeps its branch, retry continues from it with red claim ids + verifier detail in the goal). Fixed latent blocker: unique attempt-branch suffixes (retry cuts previously collided with kept branches). - backend/native: bounded shell/registry output (head 2KB + tail 6KB backstop, self-bounded tools exempt); advisor trail records structured edits; MaxOutputTokens (default 16384) + max_tokens salvage turn + json.Valid belt; one-shot budget wrap-up signal; RepoContext seam (fenced repo map, I7); in-run compaction (msgs[0] byte-identical to preserve the cache prefix, pairs never split, tool bodies excluded from summaries) + context-overflow compact-once-retry. Events: truncated_turn, budget_wrapup, loop_compact. make verify green (119 pkgs); swarm -race clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(upgrade): batch 3 — delivery loop (/diff, /apply), evidence-rich gates end to end, orientation wiring - delivery loop: verified chat/run drives KEEP their branch (the front door no longer verifies work then deletes it); the outcome line teaches the verbs; /diff renders a bounded preview and /apply merges through the EXACT structured PromoteToBase gate flow (graapprove envelope intact); WorkState.Branch round-trips a restart; internal/maint caps kept branches. - gate evidence: policy.GateEvidence (diffstat + bounded head-biased excerpt + verify tail with an explicit verdict line + spend), redacted at construction, carried additively on GateAction.Evidence through the existing StructuredApprover seam and mirrored to emit.GatePrompt (AskPrompt precedent). Rendered on every attended surface: console, plain writer, styled REPL, TUI modal, Slack/Telegram (bounded). Legacy flat approvers byte-identical (pinned). - follow-through wiring: the graapprove fall-through now forwards the FULL structured action to a structured human (11 sites via one helper; previously every graded fall-through flattened to Describe()); the swarm promote gate gains the same differ+spend evidence (no verify recorder — swarm verifies per shard, the section stays empty); RepoContext (new bounded repoMap) + CtxWindow (meter's window table) wired at BOTH Native construction sites; swarm --retries flag (default 2) arms Ledger.MaxAttempts — requeue budget was always 0, so focused/conflict retries were inert. - docs: ARCHITECTURE records vcache default-on (kernel-cutover pattern), the flake-probe + tiered-verify composition (I2 arguments), and the namespace credential masking (I3 rationale). make verify green (119 pkgs); tui build green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(changelog): record the upgrade program; fix a staticcheck nit in the graded fall-through test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(upgrade): remediate 25 adversarial-review findings (4 HIGH) + wire the verify decorators into run/serve/chat A 7-lens, 36-agent adversarial review of the upgrade branch (refute-by-default per finding) confirmed 25 real defects — mostly the integration seams between the concurrently-built waves. All fixed, each with a discriminating test. HIGH: - native.go: a mid-tool_use stream EOF (proxy closes SSE without message_stop) recorded an invalid-JSON tool_use into history, so every later request marshal failed and the run died permanently — now sanitized to {} before the turn is recorded (+ routed through the corrective belt). - verify/tiered: the scoped-red fast path shipped false REDs as the verdict and skipped the full verifier (default-on) — its subset guarantee didn't hold for opaque `make verify` recipes, flagged go-test commands, or package-load errors. Made OPT-IN; when on, arms only for full-module `go test ./...` with replicated flags and short-circuits only a provable subset red (loads fall through). - integrate: a green shard that committed nothing (gitignored artifact / already- satisfied dependent) was an ancestor of the tip, so `git merge` said "already up to date", the commit failed, and it was misclassified a CONFLICT — burning rebuild attempts to a false exit-1. mergeOne now treats an already-contained branch as an idempotent Merged success. - swarm/handoff: model-authored claim ids/fields rode UNFENCED as control lines into a dependent shard's goal (I7 break — cross-worker prompt injection). Sanitized (single-line, clipped) and moved inside the guard.Wrap fence. - swarm/passes: a DAG dependent skipped because its dep failed wrote no artifact, was invisible to requeue, and the run reported Done=true with the dependent's planned work missing. Skipped dependents are now tracked, re-run when their dep greens, and a termination-honesty backstop forbids a false green. Also: `finish_reason:"length"`→max_tokens salvage on OpenAI-family; mcp added to selfBoundedTools; cumulative message_delta usage fold; scoped-red output tail- bounded; conflictStale delete only on a fresh green re-run; resume reloads durably-passed-but-unmerged shards; resume suffixes an existing branch; ledger persists the pass's spent attempts; --resume honors an explicit --retries; /apply gates on the merge TARGET (never-auto-approve-main floor sees `main`); non-Done drives don't orphan a kept branch; /diff+/apply get TUI handlers and explicit serve replies; chatNativeBackend gets RepoContext+CtxWindow; sandbox EACCES skips instead of bricking; ~/.git-credentials masked; symlink-resolved workdir-overlap exclusion. WIRING (finding #23): orchestratorVerifier wraps run/serve/chat/resume verifiers with the vcache→flake→tiered chain (previously only build/swarm), so a green run pays one full verify not two and a flaky final verify is re-probed once before the N-worktree race — making the shipped decorators' advertised effects real. make verify (119 pkgs) + tui build + golangci-lint (0) + -race all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(swarm): unresolvedPlanned counts only never-ran nodes (no double-count vs board.Remaining) The Fix #21 termination backstop folded unresolvedPlanned into Outcome.Remaining on top of board.Remaining, but it counted every planned shard not in passed/merged/exhaustedFail — which includes a shard that RAN, verified RED, and still has retry budget. That shard already has a red artifact board.Remaining counts, so a non-converged run (passes/stalled/budget/ctx rail) over-reported Remaining (2 instead of 1 for one red shard). Not an I2 break — over-reporting is the safe direction and Done stays false — but the operator figure was wrong and it contradicted the documented disjoint-counters invariant. Track a ran-set (every planned shard that produced a terminal verdict, green or red — i.e. everything Scan can see) and count as unresolved ONLY planned shards that never ran (a Skipped dependent that never re-ran). Now disjoint from both board.Remaining and unmergedGreens. Regression test pins Remaining==1 for a single red-with-budget shard at the passes rail. Found by the remediation verification pass. make verify (119 pkgs) + swarm -race green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: RNT56 <rnt56.2b@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The grounded-answer follow-up: the supervisor reads file contents, not just diffs
The previous PR grounded the supervisor's answer in a digest (plan + cohort + tip + fenced diff-stats) and left live repo reads as a documented follow-up. This wires it: the supervisor's
read/searchtools — already advertised in its prompt but dark — now operate over a long-lived read worktree re-pointed at the integration tip, so when it reasons it reads the current integrated tree, and the grounded answer is repo-structure-aware too.What changed
internal/worktree—Checkout(ctx, ref)(hardenedgit checkout --detach --force, I4) +ListFiles(ctx, maxBytes)(boundedgit ls-files).internal/super—Supervisor.RefreshReadseam +RunContext.Tree.refreshAndPublishre-points the read tree at the tip and folds its file-list into the snapshot single-owner on the main goroutine at everyst.branchchange. The reader still does zero live host I/O — it answers off the by-value snapshot, so deadlock-freedom is intact.cmd/nilcore—buildStackcreates the read worktree off base HEAD and wiresReadTools/ReadDir/RefreshRead(re-checkout +ListFilesundergitMu), best-effort (create-failure → no read tools, loggedsuper_read_skip).renderGroundingrenders the trusted file-tree.buildAssembly.cleanup(deferred by build/chat/serve) removes the read worktree and sweeps the run's throwaway branches — no leak in a long serve session.Adversarial-review fix (the workflow caught a HIGH-severity bug)
The review found the headline path silently never worked:
buildIntegrateFunc(andbuildCodeFunc) returned their tip branch butdefer wt.Cleanup()deleted that branch before returning, soRefreshRead'sCheckout(integrate/<suffix>)always failed (only the spawn paths, whichRelease+ keep their branch, worked). Both nowdefer wt.Release()(keep the branch; the run-end sweep reclaims it). This also fixes a latent issue — the project loop's promote-timeDifferdiffs that same branch. A new hermetic end-to-end test drives the realbuildIntegrateFuncand asserts the tip branch survives and is readable viaCheckout/ListFiles.Invariants
I4 (Checkout/ListFiles route through the hardened git — a tip branch's hooks can't execute). I2 (reads are advisory; the verifier + integrator still govern). I6 (no new module dep —
tools.ReadOnly). I7 (the file-list is harness-derived git output → trusted grounding; the subagent question/reports stay fenced). Single-ownerrunState+ lock-free reader preserved.Verification
make verify✅ ·go test -race✅ ·golangci-lint run ./...0 issues ✅worktree.Checkout/ListFiles(hermetic git);refreshAndPublish(calls RefreshRead / stores tree / skips empty tip); the file-tree renders in the grounded answer; integrate-tip-survives end-to-end (the regression).🤖 Generated with Claude Code