feat: roadmap tier-run — orchestration depth, resilience, and MCP surface - #11
Merged
Conversation
…ked-error detail
Three fixes from the Countless run: (1) run_project pins the base branch and returns HEAD to it after each task, so a task that strands HEAD via an unhandled error no longer causes later tasks to pile their merges onto the dead branch (13 completed tasks had accumulated on a failed task/T005 branch instead of main). (2) blocker.blocked_reason no longer fires on validation-result phrasing ('invalid api key', 'apiKey missing from response') — only on environment-absence ('required'/'not set'/'provide') — fixing the T018 false 'needs a credential' park on an api-key-issuance task; genuine missing creds still block via absence phrasing and 401. (3) _deferral_reason shows the last MEANINGFUL error line instead of a bare code fence, so parked questions are self-explanatory.
Add an optional reference implementation to the build workflow: point it at an existing project (often another language) and its module/symbol map is extracted READ-ONLY via the tree-sitter symbol graph and prepended to the spec so decomposition and every task reproduce the reference's design idiomatically. - new analyzers/reference_digest.build_reference_digest: bounded, validated, strictly read-only (redirects the symbol-graph cache off the reference tree so the donor directory is never written to). - ProjectOrchestrator.build gains reference_dir; injected into _run_pipeline alongside the existing lesson/warm-start spec context. - exposed on the MCP build tool (reference_dir) and CLI build (--reference). - tests cover extraction, the read-only guarantee, path validation, truncation, and MCP forwarding.
misterdev had a CLI 'report' but no MCP equivalent, so an AI client could start a build but never read its outcome structurally. Add a read-only 'report' tool that returns the latest build report (completed/failed/deferred tasks, unmet-goal gaps, token/cost totals), the audit trail (per-file edits, failed commands, governance escalations), and per-model success/cost stats — misterdev's analog of the donor's get_completion_report/get_completion_findings. Thin wrapper over the existing report_view.collect; no LLM, idempotent, returns an error field for a non-directory path. Tests cover a saved report, an unbuilt project, and the bad-path guard.
Two MCP capability groups that share the same adapter surface. Async job lifecycle (core/execution/jobs.py): build_async/run_async run the existing synchronous orchestrator in a background daemon thread and return a run_id immediately, so a long build no longer blocks the client/event loop. job_status/list_jobs poll; stop_job cancels cooperatively by tripping the existing budget kill-switch (ProjectOrchestrator.request_stop lowers the live client's ceiling so the next model call raises BudgetExceededError and the pipeline degrades to a partial report) — no task-loop surgery. Guards that avoid the donor's known bugs: one running job per project (donor C-010/CLU-003) and every thread exception recorded as an observable 'failed' status (C-007). Plan-approval gate (core/planning/plan_store.py): propose_plan analyzes and persists ranked, UNAPPROVED work proposals; get_plan/approve_plan let a human select a subset; execute_plan builds only the approved items. The codebase is analyzed in-process, so it never enters the client's context. Offline tests cover the registry (status transitions, same-project guard, exception capture, cooperative stop), the plan store (approve/reject/persist, reject-wins-tie), and MCP routing for all nine new tools.
The read-only Phase-1 analysis that motivated the reference-guided build, report, async job lifecycle, and plan-approval ports; records which donor capabilities misterdev already covers and which known donor bugs to avoid.
Close a resource leak this feature introduced (the donor's CLU-004 class, which the gap analysis flagged and this code then reproduced): JobRegistry retained every finished job forever, each pinning an orchestrator/client/report via its stop-hook closure. Retain only the most-recently-FINISHED jobs (cap 50); running jobs are never evicted. Evict by finish time (ended_at), not insertion order — otherwise a long job that started first but finished last would be dropped the instant it completed, discarding the newest result. Also label a stopped run's report 'Stopped by request (partial)': stop trips a $0.00 budget internally, so build()'s report otherwise reads as a real budget exhaustion though the job status is correctly 'stopped'. Tests cover eviction beyond the cap, running-job survival during churn, and the stop label.
…ble requirements Worktree isolation: use a run-unique branch name per task (never a bare task/<id>) so a leftover branch from a prior run cannot collide with the -b create and fail the task; remove the worktree before merge/delete so even merged branches do not leak (git refuses branch -d on a checked-out branch); force-delete any unmerged branch; prune dangling worktrees at wave start. Requirements preflight: any unsatisfied requirement (not just decisions) now carries an Answer line, and a typed answer clears the account gate, so the user can steer past a missing secret/account instead of the run stalling.
…st code, not install speed
Root cause of the parallel-run false failures: a fresh git worktree has no
node_modules, so each gate ran 'pnpm --filter … typecheck' which triggered a full
implicit install (~18s) INSIDE its own 120s timeout while up to max_workers
worktrees hammered the store at once. Under contention that intermittently timed
out or hit a store lock, failing code that is correct on main — the exact
signature seen (same task passing in one worktree, failing in another; done tasks
'failing' when re-attempted).
Fix (Layer 1): prime each worktree's dependencies ONCE at creation, serially, off
the parallel gate path, with a dedicated timeout. Command is auto-detected from the
lockfile (pnpm/npm/yarn/bun) or set via orchestrator.worktree_setup_command ('' to
disable). Gates then run in ~1.5s against present deps, deterministically.
Fix (Layer 2): infra.py classifies a gate failure as an ENVIRONMENT fault (timeout,
missing dependency, locked store, ENOSPC, OOM) vs a code defect; a test gate that
fails on an infra signature self-heals with one re-run even when flaky_reruns is 0,
so misterdev stops blaming — and reflecting on — correct code for a transient.
A gate failing on an environment fault (a timeout, a missing dep, a locked store, ENOSPC, OOM) — the dominant false-failure for typecheck under a fresh worktree — was only self-healed for the test gate via _confirm_flaky. Add a _run_gate helper that runs a gate command and, only on an infra_failure signature, re-primes the worktree deps and re-runs the gate exactly once before trusting the failure; a plain code error (TS2345/AssertionError) is returned as-is. Wire it into all three gate sites (build/typecheck/test) at the same timeouts. Extract the setup-command resolver into a shared worktree_setup_command so creation-time priming and the re-prime agree on one command.
…tries A task could appear in BOTH completed and failed (observed: T002, T062a in progress.json): mark_completed discarded from failed, but mark_failed re-added even after a success — a post-success revert/retry or deferral (which persists through mark_failed) re-poisoned the ledger. Make completed the single terminal state: mark_failed is now a no-op when the task is already completed. Resume still keys on committed content via compute_task_hash/needs_rerun, so a genuine post-success regression re-runs on its changed hash rather than on a stale failed entry. Add a one-shot reconciler in _load that drops any task from failed that is also in completed and persists the healed ledger, so existing poisoned progress.json files self-heal.
…t blamed on the code A partial/broken dependency install (killed download, locked store) leaves a worktree whose toolchain does not resolve, but the failure only surfaces later in the parallel gate where it looks like a code failure. After priming each worktree, run a fast sanity probe (auto-detected for node/pnpm: npx tsc --version when TS is used, else resolve the first declared dependency; overridable via orchestrator.worktree_healthcheck_command, "" disables). On a red probe, re-prime the deps ONCE and re-probe; if it still fails, log clearly that THIS WORKTREE's environment is unhealthy — not the task's code — so downstream gate failures aren't misattributed. Shared worktree_healthcheck_command resolver mirrors worktree_setup_command. Best-effort throughout: never raises, never drops the task.
…atisfied_tasks The wave loop already skips a ready task whose content hash is unchanged since a recorded completion (compute_task_hash + the committed ledger) before it is dispatched, so an already-done task never spawns a worktree or pays a prime/ install just to be re-recognized inside it (unlike already_passed, which only short-circuits after the install). Put that skip behind orchestrator. skip_satisfied_tasks (default true) so it can be forced off, and clarify in the loop that no gate is run on the base branch for the decision — it rests purely on the content hash and the ledger, avoiding serialization/contention. Add tests: a hash-matched completion is skipped without reaching the executor; a stale or absent hash still runs; the flag off forces a re-run.
A wave merge could leave the base branch red (observed: a transient server typecheck failure on main mid-run), and the wave-end integration gate only catches cross-task regressions against a baseline. Add a per-merge gate in _execute_parallel_worktrees: after each successful merge_worktree, run the merged task's owning-target gate (typecheck/test, resolved via the same select_target/ target_commands routing the executor uses) on the base checkout. On a real (non-infra) failure the merge broke the base, so reset --hard HEAD^ removes the --no-ff merge commit and the task is returned unfinished (retried, not recorded done); a transient/infra failure is left in place, not rolled back. The base branch is never left broken at wave end. Guarded by orchestrator. post_merge_healthcheck (default true). Add GitTool.reset_hard and tests: a base-breaking merge is reverted (tree restored) while a clean merge is kept, and the disabled flag keeps the broken merge.
Under sustained ENVIRONMENT faults (gate timeouts, locked stores, OOM), running every wave at full concurrency thrashes — parallel gates contend for the same CPU/store and time out. Track each wave's UN-recovered infra-failure count and re-tune the NEXT wave: over a threshold, halve effective max_workers (floor 1) and multiply gate/setup timeouts by a factor (bounded); recover gradually toward the configured values after clean waves. The decision is a pure function (next_wave_tuning in core/execution/adaptive.py) so backoff/recovery/floors/ ceilings are directly unit-tested; the orchestrator measures the count (_wave_infra_count, completed tasks never count) and applies the tuning by scaling the config the deep gate paths read (_apply_wave_tuning), restoring the configured values at run end. Guarded by orchestrator.adaptive_concurrency (default true); a no-op unless contention actually appears. Tests: the pure policy at every branch/bound, the two helpers, and an end-to-end wave-1-infra -> wave-2-halved-workers check.
Each run rediscovered the same environment facts. Add a tiny, documented per-project ledger (.orchestrator/env_learnings.json) recording durable facts: the effective worktree setup/healthcheck commands and a max_workers learned from P6's backoff (persisted only when the run settled BELOW its base — a real contention-driven reduction; a run that recovered clears the stale reduction so one bad run never pins low forever). Ordering constraints round-trip in the schema as advisory data. On run start (_run_pipeline) the ledger pre-tunes the config, but apply_to_config never overrides an explicit project.yaml value — the user always wins, a learning only fills an unset key. Recorded at run end via _persist_learning (best-effort, like the other learning streams). Closes the cross-run self-improvement loop. Tests: read/write round-trip, corrupt/missing self-heal, explicit-config-wins, reduced-workers persist + clear-on-recovery, and ordering-constraints-persist-but-do-not-tune.
… reinstalling Priming a fresh worktree ran a full pnpm install (~19s) per worktree per wave. On a copy-on-write filesystem the base checkout's node_modules can be cloned into the worktree near-instantly (APFS clonefile — a whole tree in one syscall; Linux cp -al hardlinks). A worktree is the same repo at the same paths, so pnpm's RELATIVE workspace symlinks (apps/server/node_modules/@scope/pkg -> ../../../../packages/pkg) resolve to the WORKTREE's own packages, not the base — no reinstall, no cross-tree bleed. dep_clone.py clones the root node_modules and each workspace package's; the orchestrator verifies with the P3 sanity probe and falls back to install if the clone fails it. clone_supported() probes with a raw clonefile/hardlink so a non-CoW FS (e.g. HFS+, ENOTSUP) is detected up front and never pays a slow full-copy. Guarded by orchestrator.worktree_clone_deps (default true); always safe — a wrong guess costs an install, never a broken build. Empirically validated on an APFS pnpm+tsc workspace: clone 1.3ms, workspace symlink resolves into the worktree, tsc typecheck 0.18s with zero install, and pnpm reports 'Already up to date' (no reinstall). /Volumes/A/countless is HFS+, where clonefile is ENOTSUP, so the feature correctly falls back to install there. Tests behind a skipif when the FS lacks CoW support.
…ing merges cleanly Parallel worktree tasks that both edit a shared file (env.ts, a route registry, a schema) raced on merge — the second either conflicted or silently clobbered the first, and a conflicted 'git merge' left the base branch mid-merge (MERGE_HEAD set), blocking the next merge. Now _execute_parallel_worktrees partitions each ready wave by declared file overlap (files_to_modify/create) into conflict-free sub-waves: tasks sharing a file run in DIFFERENT sub-waves (serially), so a later sub-wave's worktrees are cut from HEAD after the earlier one merged and build on it; disjoint tasks stay parallel. The partitioning is a pure first-fit function (partition_parallel_safe in core/execution/wave_partition.py) with unit tests. merge_worktree now 'git merge --abort's a failed/conflicted merge so the base is never left dirty, and the task is re-queued (returned unfinished) rather than force-merged. Guarded by orchestrator.serialize_conflicting_tasks (default true). Tests: the pure partition at every shape, plus integration checks that a conflicting wave serializes and both complete, and that with serialization off a race conflicts, aborts to a clean base, and re-queues the loser.
…r model -> decompose) A task that failed was retried the same way. The executor now counts only NON-infra (real code) gate failures and climbs a ladder as they accumulate: attempt 1 normal; after N code failures widen context (re-anchor on the full target files + verbatim spec/acceptance), then route generation to a configured stronger model, then (instead of failing outright) request a split into named sub-steps and park the task deferred so the convergence loop re-decomposes it. Infra failures (infra_failure) self-heal and NEVER advance the ladder — the counter is gated by should_count_failure. The rung choice is a pure function (choose_rung in core/execution/escalation.py), independently unit-tested per rung and for the infra-never-advances guarantee. Config-driven and bounded (escalation_enabled + cumulative thresholds + escalation_model). Also route a deferred executor result to report.deferred_tasks (not failed) in _execute_tasks, so a decomposition/walk-away park isn't miscounted as a terminal failure or trip the consecutive-failure abort. Tests: pure ladder + should_count_failure, the executor rung/spec-block/substeps/decompose helpers, and deferred routing.
Make a run's failures signal, not noise. A new operational taxonomy (core/execution/failure_taxonomy.py) classifies every terminal non-success into infra / blocked-external / merge-conflict / acceptance-unmet / genuine-code-failure / deferred-needs-input, reusing the existing pure signal detectors (infra_failure, blocked_reason) so a fault is labelled the same way it was handled. classify_failure and build_run_summary are pure and unit-tested (routing per category, signal-wins-over-status, aggregation, top-obstacle tie-break). At run end _write_run_summary (from _persist_learning, on normal and budget-halt paths alike) aggregates counts + a failure breakdown + wall-clock + the top recurring obstacle with an exemplar, prints a one-glance console line, and writes .orchestrator/run_summary.json for P7/P10 to consume. The wave loop's error path now stashes the failure text on the task so a merge-conflict/worktree error is classifiable (it records no ExecutionResult). Named test_run_summary.py to avoid colliding with the unrelated core.learning.failure_taxonomy tests.
Validate a project is ready for an unattended run BEFORE spending budget. New 'misterdev doctor' subcommand checks: git repo, clean working tree, on the base branch (not a stranded task/*/doctor/* branch or detached HEAD), no leftover task/* branches or dangling worktrees, the configured models resolve (reuses the model preflight health_check), a throwaway worktree primes and its toolchain resolves (reuses _worktree_setup_command/_worktree_healthcheck_command), and REQUIREMENTS.md inputs are answered. Prints a pass/warn/fail checklist with an actionable fix per issue and exits non-zero on a HARD blocker (dirty tree, stranded branch, unroutable model); warnings inform but never block. The check routing and aggregation live in a pure module (core/execution/doctor.py) — the orchestrator gathers the environment facts and run_doctor routes them through the pure checks. Tests: per-category routing, aggregation + exit codes (all-pass=0, warnings-only=0, any-fail=1), and run_doctor end-to-end on a temp git repo (ready / dirty-tree-fail / unresolvable-model-fail).
Pure refactor, no behavior change. agent.py was a ~3450-line god-module; move the cohesive parallel-execution unit (_execute_parallel, _execute_parallel_worktrees, _worktree_setup_command/_worktree_healthcheck_command/_worktree_healthcheck, _prime_worktree_by_clone, _post_merge_healthcheck, _partition_disjoint, _task_file_set) into core/execution/parallel.py as ParallelExecutionMixin, which ProjectOrchestrator now inherits. Every method uses only self + module-level helpers, so method resolution and all call sites (including tests) are unchanged. Drop the now-unused imports (concurrent.futures, partition_parallel_safe, _WorktreeProjectView) from agent.py and repoint the test patch targets that reached into agent's concurrent.futures to the new module. Full suite green (2270 passed); agent.py down ~400 lines.
Pure refactor, no behavior change. Move the cohesive integration-gate unit (_integration_gate + its count/identity/targets variants, _suite_failures/ _suite_failing_ids/_failing_ids_from_output parsers, _wave_commits, _target_regressed, and the post-build _maybe_rollback_regression) into core/execution/integration_gate.py as IntegrationGateMixin, which ProjectOrchestrator now inherits alongside ParallelExecutionMixin. Every method uses only self + the executor passed in, so call sites (including _execute_tasks and _run_pipeline) and tests are unchanged. Full suite green (2270 passed); agent.py down another ~330 lines (now ~2710, from ~3450 before P13).
…y self-unblock Dogfooding countless (via 'run --tasks') exposed three defects behind the 20/90 completion: 1. run_project (the run --tasks path) never wrote run_summary.json or recorded env learnings — P7/P11 were wired only into the build pipeline, so the exact command used on countless was undiagnosable and fed no cross-run memory. Extract _emit_run_summary (shared with the build path) and call it + _record_env_learnings from run_project's close-out, classifying its failed/deferred tasks through the same taxonomy. 2. The escalation ladder's DECOMPOSE rung was unreachable at default config: the in-loop check needs code_failures >= escalation_decompose_after (3), but the loop breaks at attempt >= max_task_attempts (3) the moment it reaches 3 — so a keystone like T007 (a real TS2322 that failed every attempt, on the strongest model, since the selector already climbs tiers) dead-ended at a vague park instead of decomposing. Fire decompose at attempt EXHAUSTION when the ladder climbed to it, so the too-large task is split into named sub-steps (the build pipeline re-decomposes them). 3. A detected stall set 'try a different approach' but did NOT advance the ladder, so a staller (T005) spun until attempts ran out and never escalated. Count a stall as a code failure so it climbs toward widen/decompose. Tests: run_project writes a classified run_summary.json; a keystone that fails real gates decomposes (deferred + named sub-steps) at exhaustion. Full suite 2272 passed.
…ource block Dogfooding countless surfaced the real freeze: T007 (the server keystone, gated by @cloudflare/vitest-pool-workers) failed typecheck/test but its output carried a 401, so blocker.py's broad \b401\b/\b403\b rules parked it as 'authentication failed — needs your credential'. The user's own QUESTIONS.md answer said it plainly: not a missing requirement, the server package failed typecheck/test. A mis-parked keystone froze the entire server subtree (~68 cascade parks). A bare 401/403 emitted INSIDE a test run is code to fix (a test asserting on a 401/403 response, or local miniflare/workers-pool noise), not a credential block. Suppress ONLY those two broad HTTP-status auth rules under test context (vitest/jest/miniflare/expect()/.toBe/describe/it/Test Files/.test./.spec.); the SPECIFIC rules — an explicit wrangler login, a named missing key/token — still fire, because those name a real external resource an HTTP status alone does not. So T007 is now treated as the code task it is (retried/laddered/decomposed) instead of waiting on a human. Full suite 2274 passed.
…smatch diagnostics
…ode_context/reference_sites)
…ion-gated, opt-out)
…ry, not a reprompt)
…g the stuck candidate
…er can't leak past the deadline
…s (O(S) scans -> file-local)
… on trimmed tasks
…no indefinite hang)
…in the parallel wave)
…ntly defaulting when the strongest tier is empty
…hape (no later crash)
…nterrupted on reload)
…e via a reporter)
…ault (persistence live)
…ro production callers)
… (done/total/phase)
… ' preserved; drop dead ModelSelector.is_ready
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
66 commits across the full orchestration stack. Groups by concern:
Tier 0–5 execution ladder
Critical / high correctness fixes
Async job persistence (A1–A4)
Performance
Security
Infrastructure / resilience
MCP surface
misterdev doctorpreflight commandreporttool)Test plan
misterdev doctorexits 0 on a healthy checkout