v0.30.0

Added

• Three observability/capacity fixes for a stuck-orchestrator incident nobody could diagnose live. Tonight the orchestrator process sat at 113% CPU for 80 minutes with no way to see why: the image had no profiler and Postgres had no per-query stats, and separately pg_stat_user_tables showed chunks_journals taking 196 sequential scans reading 6.1M tuples against only 387 live rows because its only vector index (ivfflat, added in migration-era code back in June) trains its k-means clusters on whatever rows exist at CREATE TIME -- nothing, on a brand-new table -- and the planner never trusted it. Fix one: the orchestrator image now ships py-spy as a standalone uv tool install, never a roboco dependency, so docker compose exec orchestrator py-spy dump --pid 1 dumps every thread's stack without a restart; it needs SYS_PTRACE, added via cap_add on the orchestrator service in all three compose files. Fix two: migration 095_pg_stat_statements enables pg_stat_statements inside an autocommit_block (mirroring migration 090's enum-add pattern), and the postgres service in all three compose files now carries command: ["postgres", "-c", "shared_preload_libraries=pg_stat_statements"] so the extension actually collects something -- without the GUC, CREATE EXTENSION succeeds but the stats view stays empty; the postgres container restarts once on the next bring-up to pick up the flag. The migration's CREATE EXTENSION is still wrapped in a try/except, narrowed to sqlalchemy.exc.DBAPIError and only two SQLSTATEs verified against a live Postgres (not the file-access class 58P01 a control-file-missing error might suggest -- Postgres actually codes that path 0A000 feature_not_supported): 0A000 (extension not available, e.g. an image without pgvector's contrib packages) and 42501 (insufficient privilege); it also checks the error names pg_stat_statements specifically, so a typo'd extension name in this file's own SQL re-raises instead of quietly no-oping forever. Anything else re-raises too. Fix three: VectorStore.initialize() (the single ensure-schema point behind every chunks_* table) now provisions CREATE INDEX CONCURRENTLY ..._embedding_hnsw_idx USING hnsw (embedding vector_cosine_ops) instead of the old ivfflat index, matching the <=> cosine operator that search/hybrid_search already order by: HNSW builds incrementally as rows are added instead of training once on an empty table, initialize() repairs a previously-failed CONCURRENTLY build (indisvalid = false) by dropping and recreating the index instead of leaving it dead forever, and the old ivfflat index -- useless in production for the reason above -- is dropped once the HNSW index is valid.

• New gateway verb cancel_leaf(task_id, reason) gives a Cell/Main PM the mechanical path to close a zero-diff leaf (task de9379cc, 2026-09-05). A fix leaf whose findings a sibling had already resolved on the parent branch left no legitimate diff behind, but nothing in the gateway could close it: i_am_done refused (NO_COMMITS gate), complete refused (no open PR), unblock just re-spawned the developer, and 8 from-scratch verifications by fe-dev-1, fe-dev-2, fe-pm, and main-pm all agreed nothing remained to fix before the PM's own escalate/unblock loop tripped the oscillation breaker and forced the CEO to cancel it by hand via POST /api/tasks/{id}/cancel. cancel_leaf reuses TaskService.cancel, refuses up front if the target has any children of its own so its cascade never reaches a real descendant, requires the caller own the leaf's parent coordination task (a Main PM may instead act on any root's descendant), and fails CLOSED on a zero-diff check reusing the same GitService.is_behind_base ahead-count _maybe_waive_pr_creation uses - an open PR, any commit ahead of the leaf's base, or an unanswerable git error all refuse rather than guess, the opposite of that check's fail-OPEN since this one authorizes discarding a task outright. reason is recorded as both the PM's journal:decision and the task's cancellation note; the returned next hint points the PM at re-submit_up/submit_root on the parent. The oscillation breaker's own remediation text (_oscillation_unblock_guard / _maybe_trip_oscillation_breaker) now names cancel_leaf so the next PM stuck in the same loop finds the exit without escalating to the CEO. A cancelled leaf was already excluded from the sequence bar's candidate set for every cancel path; a new test pins that for this one too.

• ROBOCO_ROLE splits the single python -m roboco.cli process into three (api / dispatcher / indexer), so the fleet stops pinning one core. Measured live on the production NAS (2026-09-05): the whole backend ran as ONE process, FastAPI plus the Orchestrator's roughly twenty background loops (dispatcher, sweeper, self-heal, CI-watch, dep-update, release manager, board programs, vault, telegram, ...) plus the RAG/embedding pipeline, and the GIL pinned it near one core (113% CPU on a 12-core box), starving every gateway verb (120s timeouts, 60 evidence-leg timeouts per 30 minutes). The new role setting (env ROBOCO_ROLE, default all) gates a single seam in AgentOrchestrator.start(): api serves stateless FastAPI request/response routes, but never runs the once-per-fleet startup reconciliation or launches a single background loop; dispatcher is the fleet owner, running that reconciliation plus every loop; indexer runs only the KB/embedding worker (roboco/services/optimal_brain/indexer_worker.py, landing on a parallel effort). all keeps today's single-process behavior byte-for-byte, including running the indexer worker in-process, so make quickstart and the test suite are unaffected. AgentOrchestrator._instances/_waiting_records (the live fleet) are per-process dicts, so docker/nginx.conf gained a dispatcher upstream and routes every fleet-touching path there instead of to the orchestrator (whose copy of that state is always empty in split-role mode): /api/orchestrator/* (status/spawn/stop/waiting-agent), /api/secretary/live/* and /api/prompter/live/* (the on-demand Secretary/Intake chat spawn + SSE relay, which also needs to run where the reconciliation/reaper loops that must see their containers run); everything else stays on the orchestrator. OptimalService's auto-index-on-startup + periodic RAG update, the FastAPI app's own RAG-index reconcile pass, and register_default_handlers's fleet-side-effect event handlers (waiting-record resolution, auditor spawn) are role-gated the same way (all/indexer own the KB work; dispatcher/all own the fleet-side-effect handlers; api owns neither); the WebSocket bridge is the mirror image, running only where /ws/ actually terminates (api/all, never dispatcher, which has no live WS connections to forward events to). The event-bus consumer name is now orchestrator-{role}-{hostname} (previously orchestrator-{host}:{port}, which collided across the api/dispatcher containers since both set ROBOCO_HOST=0.0.0.0/ROBOCO_PORT=8000, putting two processes under one Redis consumer-group identity). The dispatcher's own task/notification queries go over HTTP to its OWN uvicorn: internal_api_url resolves to loopback for the dispatcher role by construction, never through the orchestrator (api-role) service. All three compose files gained dispatcher and indexer services sharing the orchestrator's image, environment and volumes via YAML anchors, depends_on a new orchestrator healthcheck (python -c "import urllib.request; ..." against /health, since curl is purged from the runtime image), and ROBOCO_ROLE=api set on the orchestrator itself.

• CEO-gated GET /api/dashboard/portfolio — the first per-project metrics view of the governed fleet (task 1e0b4c7b, PR #966, the assembled portfolio branch; the route leaf was delivered by sibling task b3960175, PR #965). The CEO governs 5 products but every prior dashboard endpoint was per-team or org-wide. The new endpoint returns one PortfolioProjectMetrics row per project — project_id/project_slug/project_name, active_task_count, median_lead_time_hours, rework_rate, open_findings_count, monthly_budget_burn_usd — sorted most-active-first; a project with no activity still appears, zeroed. All aggregation is server-side (task 1e0b4c7b's MetricsService.get_portfolio_metrics): active = statuses minus {backlog, completed, cancelled}; median lead time + rework rate are read from ONE windowed completed-tasks pass (median, not mean — agent lead times are heavy-tailed); open findings are SQL-grouped from the revision-findings ledger rather than the capped ReviewFindingsRepository.list_open_findings dashboard fetch; the calendar-month burn mirrors TaskService.project_month_spend_usd's pricing (open spawn sessions priced live from token counts) in one batched query instead of N per-project queries. The route carries the dashboard router's FIRST explicit per-route role gate (non-CEO panel token → 403), on top of the router-wide panel-token dependency. Integration tests cover the seeded aggregation math (month-boundary-safe) and the CEO-only 403. See docs/backend/api/dashboard-portfolio.md.

• Branch/parent-topology validation moves from the terminal verbs to open_pr and the re-parent path (task df9803b2). Previously a task's branch/PR-base mismatch against its real parent was only caught at complete/submit_root — the CEO_ONLY head-branch routing in cell_pm_complete — after all work and review had already been spent (the incident class behind task 5612b225/PR #856, which bounced 3+ times over two days, plus 6866e888/6b9e19aa/7de89c6e). find_topology_issue (roboco/services/gateway/merge_chain.py), reusing the existing resolve_parent_branch rather than re-deriving a base from the branch-name string, detects two bad shapes: a parented task whose recorded PR base is the project's integration/head branch instead of its parent's own branch (parented_base_is_head), and a parentless root whose branch name encodes a nested hierarchy — the signature of a task detached from a coordination ancestor during a restructure, never a genuinely standalone root (parentless_in_coordination_tree). It is wired into two earlier catch points: open_pr's preflight (roboco/services/gateway/choreographer/_impl.py) refuses BEFORE create_pr runs, naming the expected base branch and the concrete repair; and the parent_task_id re-parent write-through in TaskService.update() (roboco/services/task.py, via the new TaskService.recorded_pr_base() comparison input) records the same mismatch via a topology_issue marker rather than blocking the PATCH, since a re-parent PATCH is often itself the fix for a prior mismatch. A follow-up round (QA finding F-1256daf6) closed a gap where a detach-style PATCH ({parent_task_id: null}) bypassed the recheck entirely — that field is popped out of the generic update loop and applied via a direct setattr in _apply_null_clears, never touching TaskService.update()'s field-update loop. TaskService.recheck_topology_after_reparent is now public and called both from update()'s existing set-path and from a new _recheck_topology_after_detach helper in roboco/api/utils/tasks.py, invoked in the PATCH route right after _apply_null_clears. The existing terminal-verb CEO_ONLY checks are unchanged and remain the final gate.

Changed

• AgentOrchestrator (roboco/runtime/orchestrator.py, formerly a single ~19,700-line class) is mechanically decomposed into per-family mixins under the new roboco/runtime/engines/ package, with no behavior change. Every non-core method's body moved byte-for-byte verbatim into the mixin matching its family, dispatch (dispatch_routing, dispatch_claim, dispatch_prompts, dispatch_work, dispatch_breaker), spawn (spawn_config, spawn_launch, spawn_exit), the sweeper loop and the standalone startup-reconciliation/health/board-program loops (sweeps, reconcile), the rate-limit probe loop, the on-demand intake/secretary session subsystem (interactive_sessions), a small set of cross-family helpers (_shared), and the remaining default-off autonomy-engine/board-program poll loops (ci_watch, dep_update, env_sync, release_manager, strategy, telegram_poll, vault, video_render, x_mentions), AgentOrchestrator now composes all of them and keeps only its core lifecycle (__new__/__init__/start/stop) and module-level constants/functions, every one of which still resolves at roboco.runtime.orchestrator.<name> for any caller or test that patches it there. A new TYPE_CHECKING-only AgentOrchestratorSelf Protocol (roboco/runtime/engines/_types.py) gives every mixin's methods typed access to the attributes and methods sibling mixins provide, so mypy passes without touching a single moved method signature.

• The agent gateway (all 102 endpoints across roboco/api/routes/v1/do.py and flow_*.py) is now WAF-scanned for real, without pulling the docker mesh off the global whitelist or exposing it to rate limiting. The mesh (and only the mesh) previously skipped the signature WAF, rate limiting, and IP-ban checks entirely on every route via _INTERNAL_NETWORKS's membership in guard-core's global whitelist, which is EXCLUSIVE and also shared by the panel container, the SDK sidecar, nginx, and both bridges, so the mesh could not simply be removed from it. mesh_scanned (roboco/security.py) applies a route-level require_ip allowlist to every gateway endpoint instead: guard-core's IpSecurityCheck skips the global whitelist for a route with route_config.ip_whitelist set and stamps is_whitelisted=False, so SuspiciousActivityCheck (the WAF) now actually runs against agent traffic there. bypass(["ip_ban", "rate_limit"]) on the same routes keeps the mesh scanned but never rate-limited and never banned on the gateway -- a 429 on a verb is a wedge vector, not a security win, on an already HMAC-authenticated peer, and the route-level rate_limit bypass was confirmed to also skip guard-core's GLOBAL 120/60 bucket for that route, not just its own per-route decorators. Every other route (panel, /api/optimal/*, /health, ...) stays exactly as exempt as before. _WAF_FREETEXT_BODY_FIELDS (the exclusion set shielding agent-authored free text from the signature WAF) turned out to match key names at ANY depth, not top-level only as its comment claimed -- it skips the whole subtree under an excluded container key -- so 22 more field/container names (resolved_findings, note, commit, finding_id, evidence, items, reply_body, posts, process_change, ...) were added after auditing every request model reachable from the gateway, closing false positives reproduced on i_am_done/submit_up/submit_root's resolved_findings[].note, waive_finding.note, and every Board propose_*_hunt/*_fill/*_fixes's items[].evidence. A parametrized regression test builds a request body from every v1 gateway endpoint's pydantic model (enumerated from app.routes, so a future endpoint is covered automatically), fills every free-text-shaped field with a shell/SQL/XSS/path-traversal/diff/URL corpus string, and asserts none of the 102 endpoints get WAF-blocked. Ban safety is separate and load-bearing: IPBanManager.ban_ip is patched at import time so a mesh-overlapping IP or CIDR range is blocked per request on a WAF hit but never recorded as banned, since a ban overrides the whitelist on every route and would otherwise wedge an agent container fleet-wide from a single false positive; the refusal now also catches a CIDR-form ban request that overlaps the mesh (ipaddress.overlaps), not just an exact address match. _INTERNAL_NETWORKS, trusted_proxies, and the route-level allowlist now derive from one shared _MESH_NETWORKS constant instead of three hand-typed copies. roboco/mcp/flow_server.py and do_server.py now recognize a guard block's plain-text response (400/403/429, not JSON) and surface a gateway_blocked envelope telling the agent to move shell/SQL/HTML-looking content into a free-text verb field and retry, instead of the generic transport_error that read as the orchestrator being down.

• Adopted fastapi-guard 8.0.0 / guard-core 4.0.1. Deploy note: guard-core 4.0.0 hashes the endpoint path segment of its Redis rate-limit keys, so every in-flight rate-limit counter resets once on deploy (one-time, not a recurring reset). guard-core 4.0.1 makes IPBanManager.ban_ip return a bool that every ban call site now reads (the WAF threshold ban and rate-limit auto-ban, the identity-violation escalation, and the behavioral usage_monitor(action="ban") action), so roboco/security.py's mesh-safe wrapper now propagates it (False on a mesh refusal) instead of returning None; without that, every ban RoboCo actually applies would have been logged and reported to telemetry as tracked rather than banned (the identity-escalation path that catches an external attacker at the exclusive whitelist, and every usage_monitor route left on its default action="ban"), and on the WAF threshold path, which no client class reaches in the deployed topology but a test now exercises with an emptied whitelist, the banning response would have been 400 "Suspicious activity detected" instead of 403 "IP has been banned" while the IP was in fact banned. 4.0.1 also fixes a ReDoS validator stall (up to 40s per catastrophic custom pattern arriving through config, decorators, or dynamic rules, present since 3.13.0); RoboCo's only exposure is enable_dynamic_rules=True.

• Embedding/RAG-index writes now flow through a Redis stream to a dedicated indexer worker instead of running inline in whichever process happened to schedule them. Measured on the production NAS: ollama (qwen3-embedding:0.6b, CPU) runs at 200% CPU, one /api/embed call takes 40 to 55 seconds, and "Parallel embedding starting" fired roughly 4 times a minute inside the single backend process (journal entries, doc writes, i_am_idle), stealing CPU and event-loop time from the API/dispatcher rather than adding request latency, since indexing was already backgrounded. Producers (roboco/services/journal.py, roboco/services/task.py) now call enqueue_index_request(kind, payload) (new roboco/services/optimal_brain/indexer_worker.py) instead of awaiting OptimalService.index_journal_entry / index_documentation / unindex_journal_entry / record_learning directly; the request publishes onto the roboco:stream:index Redis stream (reusing StreamEventBus, new EventType.INDEX_REQUESTED, category "index") and consumer group indexer consumes it via run_indexer(), which also owns the periodic RAG re-index sweep (OptimalService.ensure_periodic_update_running, new); run_indexer() runs in the standalone ROBOCO_ROLE=indexer process, or as an in-process task under ROBOCO_ROLE=all, both wired by the process-roles change landing alongside this one. StreamEventBus gained a generic MAX_DELIVERY_COUNT (5) reclaim cap, but delivery count alone is never enough: _recover_stream dead-letters a message only once it is BOTH past that cap AND idle at least as long as the reclaim loop's own idle threshold, so a handler still actively working a message on its 6th delivery is never dead-lettered out from under it. The backlog check also measures the real thing: _check_backlog reads the indexer consumer group's lag from XINFO GROUPS (falling back to XPENDING's pending count on Redis <7, where lag is absent) instead of XLEN, which also counts already-acked history that nothing ever XTRIMs/XDELs, so a fully-drained stream no longer looks permanently over-backlog; past indexer_max_backlog (new setting, default 500) lag, the oldest UNDELIVERED journal_entry/documentation requests (read from strictly after the group's last-delivered-id) are dead-lettered with reason backlog_shed; journal_unindex is never shed, since a skipped de-index leaves stale or private content searchable. enqueue_index_request falls back to running the index inline (a tracked background task, own dead-letter handling for journal_entry via the existing rag_index_failures table) whenever the stream is disabled (indexer_stream_enabled, new setting, default true) or Redis is unreachable, resolving the OptimalService through an optional optimal_resolver the producer passes in (journal.py's own lazily-cached instance) rather than always re-resolving the global singleton, so a single-process ROBOCO_ROLE=all deployment with no separate consumer never loses indexing. Producers await the enqueue call itself directly (a single Redis XADD, or the inline fallback's own backgrounded task) rather than scheduling the whole enqueue as a background task, so a journal entry's index request can no longer race behind a same-entry unindex request issued right after it. roboco/api/routes/optimal.py's explicit CEO-triggered POST /kb/index/docs and rag_index_failures.py's startup-only reclaim_due retry path are both left calling OptimalService directly, on request and off the request path respectively. ollama itself is a single CPU-bound server shared by indexing AND by query embeds (KB search, institutional memory), and the same night's incident also had KB searches timing out at 30 seconds because an indexing batch held every ollama concurrency slot; run_indexer(dedicated_embedder=True) (the standalone indexer process only) seeds this process's shared-embedder singleton (set_shared_embedder, new, in roboco/services/optimal_brain/shared_embedder.py) with its own OllamaEmbedder built at indexer_embed_concurrency / indexer_embed_batch_size (new settings, default 1 / 8) instead of the shared default of max_concurrent=4 / batch 32 (OllamaEmbedder gained a batch_size constructor field that embed_chunks/aembed_chunks now use), so a query embed in that process always finds a free slot; under ROBOCO_ROLE=all, dedicated_embedder stays False and the shared embedder is never touched, since that same process also serves interactive query embeds through it and seeding it there would throttle every one of those too.

• Migrated the eight MCP servers to the mcp 2.x SDK. The 2026-08-31 lock refresh (GLM 5.3 upgrade, commit 5a6d25805) bumped mcp 1.29.0 to 2.1.1 under pyproject's unpinned "mcp" dependency, and mcp 2.x removes mcp.server.fastmcp outright (FastMCP is now MCPServer in mcp.server.mcpserver), which broke all 30 e2e-smoke lifecycle tests at import and produced 46 mypy errors across every roboco/mcp/* server. Nothing was missing from the venv — the API moved. All servers (flow, do, optimal, docs, search, intake, secretary, git_readonly) and the two mcp unit tests now import MCPServer; json_response=True was dropped from the three factory constructors (docs, optimal, search) because mcp 2.x moves transport params off the constructor to run()/sse_app()/streamable_http_app(), and it was inert for these stdio-only servers anyway. call_tool()'s direct-call return changed from the 1.x (content_blocks, structured_dict) pair to a CallToolResult, so test_optimal_role_scope reads .structured_content instead of unpacking the tuple. submit_directive's payload annotation gained | None = None to match its own runtime tolerance (payload or {}) — the mismatch was always there but mcp 1.x's untyped @mcp.tool() decorator hid it from mypy; mcp 2.x types the decorator, surfacing it. mcp 2.x also stops reading MCP_* env vars and pydantic-settings files (none were relied on) and swaps its HTTP stack to httpx2 (lock-level only; no code imports httpx via mcp).

• Split-role deployment hardening.

  • "split the backend into api, dispatcher, and indexer processes"
  • "split the orchestrator into engine mixins"
  • "never pull the dispatcher and indexer images"
  • "pin two mixin attribute types mypy inferred as non-Optional"
  • "start nginx after the dispatcher it proxies to"
  • "resolve the dispatcher per request so nginx starts without it"
  • "keep the dispatcher's loopback API URL out of agent MCP configs"
  • "health-check the indexer role and wait for both split roles at bootstrap"
  • "run the boot-time RAG reconcile in the indexer role"
  • "drop the indexer healthcheck, the worker role serves no HTTP"

• Claiming and dispatch fairness.

  • "commit the claim before branch work and keep row locks off git I/O"
  • "one in-flight claim per agent and release parked claims"
  • "hand a developer its bounced work before new pending work"
  • "hand out work oldest root first"
  • "hand out work oldest first"
  • "assign the PR reviewer at gate entry and stop overflow double-dispatch"
  • "clear PM ownership at PR-gate entry; oldest-first offer order; skip blocked in give_me_work"
  • "remove dead blocked-skip from give_me_work; correct stale submit_up docstring"
  • "wait for a parent's branch instead of auto-blocking the child"
  • "log the provisioning wait at debug and cover status=claimed"
  • "hand root-run install outputs back to the agent user"
  • "stop holding task row locks across the post-merge chown walk"
  • "keep the MCP gateway reachable under spawn bursts and stale clones"
  • "cancel_leaf gives PMs a path to close a zero-diff leaf"
  • "move embedding and indexing to an indexer worker fed by a Redis stream"
  • "ship py-spy, pg_stat_statements, and HNSW indexes for the KB"
  • "record_learning best-effort skip on embedder timeout"

• Uptime-aware staleness.

  • "task_time verb reports uptime-adjusted task times"
  • "discount fleet downtime in every staleness check"
  • "derive fleet active time from dispatcher heartbeats"

• Gateway and transport fixes.

  • "WAF-scan agent gateway traffic from the docker mesh"
  • "gate /ws handshakes with guard, arm rate-limit auto-ban, drop guard-agent"
  • "guard_ws fails open when SecurityMiddleware never mounted"
  • "send empty JSON body on bodyless POSTs tripping content_type_filter (415)"
  • "park Anthropic agents on an expired host login and cap crash retries per task"
  • "Sonnet 5 promo runs through 2026-09-13, not 08-31"

Fixed

• The boot-time RAG index reconcile ran nowhere in a split-role deploy (2026-09-06). The FastAPI lifespan scheduled it for ROBOCO_ROLE all and indexer, but the indexer role never runs that lifespan (roboco/bootstrap.py runs only run_indexer), so after the split no process re-indexed unindexed approved playbooks, reclaimed dead-lettered index writes, or backfilled zero-chunk journals/learnings at startup. The three passes now live in roboco/services/optimal_brain/indexer_worker.py as reconcile_rag_indexes(optimal): run_indexer schedules them in the background under the indexer role (and cancels a still-running pass on stop), while the lifespan schedules them only for all.

• The indexer service's HTTP healthcheck could never pass, since the indexer role never runs uvicorn (2026-09-06). Commit bfdd3c44 added the orchestrator's /health probe to the indexer service in both compose files on the claim that "every role runs uvicorn, only the lifespan work is role-gated" - false for indexer: roboco/bootstrap.py's _run_indexer_role only awaits run_indexer(dedicated_embedder=True) and returns before _run_api_server is ever created. On the NAS roboco-indexer sat Up 2 hours (unhealthy) (FailingStreak 811, connection refused) while its logs showed normal embedding work; nothing depends_on the indexer with a health condition and scripts/bootstrap.sh waits on running state, not health, so the bad probe was misleading, never blocking. The healthcheck: block is dropped from the indexer service in both compose files, replaced with a comment naming why one can never apply (no HTTP endpoint; process death is covered by restart: unless-stopped).

• A loopback ROBOCO_API_URL pinned on the dispatcher leaked into every spawned agent container's MCP config, breaking every gateway verb fleet-wide for hours (2026-09-06). _generate_mcp_config writes settings.api_url straight into every agent's ROBOCO_API_URL/ROBOCO_ORCHESTRATOR_URL, but both compose files pinned the dispatcher's ROBOCO_API_URL to http://127.0.0.1:8000 so its own internal_api_url (used for the dispatcher's task/notification queries) would resolve to its own uvicorn -- the same env var doubled as the agent-facing URL handed to every spawned container, so every MCP server inside every agent container was told the API lives at 127.0.0.1:8000, which is nothing inside that container, and every gateway verb crashed with httpx.ConnectError. internal_api_url (roboco/config.py) now derives the loopback URL from role == "dispatcher" directly instead of from api_url, so both compose files drop the ROBOCO_API_URL: http://127.0.0.1:8000 line from the dispatcher service entirely -- api_url is now only ever the agent-facing URL handed to spawned containers' MCP servers, never the dispatcher's own address.

• i_will_work_on / i_will_plan start the task after the branch is provisioned, not before. The claim durability boundary moved branch creation out of the verb runner's composed savepoint, but start stayed inside it, and the claimed -> in_progress branch gate rejected every real (branchless) claim with "Cannot start work: no branch assigned" (the E2E smoke went red on slave; the real-DB lifecycle test never saw it because its fixture pre-seeds a branch). The intents now compose (claim, set_plan); the choreographer provisions the claim, then calls start, so the durable ladder is claim fields, then branch, then in_progress. A branchless real-DB test and a call-order unit test pin it.

• The developer/PM claim (POST /api/tasks/{id}/claim → TaskService.claim_task_for_agent → claim, and the verb runner's composed i_will_work_on / i_will_plan) now commits the claim fields before branch auto-creation, commits AGAIN right after the branch lands, and never runs branch creation, upstream-base inherit, or work-session setup inside a savepoint on either path. The prior fix (PR #951's boundary, ported onto this claim) committed the claim fields early but left two gaps a critic caught on real Postgres: a failure AFTER a successful branch create but before the caller's trailing commit (_create_work_session_if_needed, _inherit_upstream_base, session.refresh) rolled back the uncommitted branch_name while the earlier CLAIMED commit stood, landing status=CLAIMED, branch_name=NULL with the branch already pushed to origin (non-idempotent on retry); and the verb runner's _do_claim ran the WHOLE claim (including _ensure_branch_for_task's network git) inside _run_composed_actions's begin_nested() savepoint, holding the task row's FOR UPDATE lock and the agent row's implicit update-lock for up to the verb timeout (120-900s) on i_will_work_on / i_will_plan and every MegaTask root-subtask / self-heal claim that routes through them. TaskService._finalize_claim is now split: _apply_claim_fields (status transition plus claimant fields, commits durably unless nested) and _provision_claim (branch creation, then a SECOND durable commit, then upstream-base inherit plus work session plus proactive context, never nested on either path). claim() takes provision: bool = True; the verb runner's _do_claim calls claim(..., provision=False) inside its savepoint (fields only), and Choreographer._claim_plan_start_run calls the new TaskService.provision_claim(task_id, agent_id) once that savepoint closes and before building the success envelope; provision_claim itself commits the savepoint's result durably first (so a choreographer built against a mocked TaskService, every gateway unit test, never needs to mock a raw session call it did not already have to), then re-reads the task/agent fresh and runs _provision_claim. A provisioning failure reverts the claim/plan/start fields durably (the same revert-and-commit _provision_claim always ran on a branch failure) and surfaces the same "verb runner failed" rejection a savepoint failure already returns. The pre-claim field snapshot _provision_claim's revert needs crosses this boundary via TaskService._pending_claim_snapshots, a per-instance (per-request) dict keyed by task id, populated by _apply_claim_fields when nested and popped by provision_claim. That snapshot also captures plan and started_at: the composed claim/set_plan/start sequence commits both before _provision_claim ever runs, and without them in the snapshot a branch-failure revert put the task back to PENDING while leaving the failed attempt's plan and started_at in place, corrupting cycle-time and showing a plan on a pending task. Net failure semantics, by point of failure: before _apply_claim_fields flushes, nothing happened, safe to retry; between that flush and its commit, durable only if non-nested (nested rolls back with the enclosing savepoint on any later raise); between the claim commit and a branch-creation failure, reverted to the pre-claim snapshot durably (fields, plan, and started_at all restored), claim is retryable; between a successful branch create and its own follow-up commit, no failure point exists there; after the branch commit but before work-session/upstream-base/refresh completes, CLAIMED plus branch_name stand, NOT reverted. Two different things retry-heal from there: _ensure_branch_for_task trust-but-verifies a pre-set branch_name, so a later provision_claim re-run only redoes the failed remainder; a missing work session specifically heals on the agent's own next verb call instead, since _resume_from_claimed (the i_will_work_on / i_will_plan re-entry for a task already CLAIMED and assigned to the calling agent) calls TaskService.ensure_work_session right after start(), a no-op once the session exists, so it is safe on every re-entry. claim_pr_review was never on this boundary: its verb body dispatches TaskService.pr_review_claim directly, bypassing the verb runner and _finalize_claim entirely, so an earlier draft of this note naming it here was wrong.

• The Cost-Tiered button in Settings → AI Providers stopped promising a seed the backend no longer performs (PR #983). The ModeButton description, the confirm dialog and the success toast in panel/src/components/settings/ai-routing-card.tsx all claimed "Seed developer:low → Haiku (see below)", but _COST_TIERED_SEED (roboco/services/llm.py) has been an empty tuple since the 2026-07-24 retirement ("haiku retired from delivery-lifecycle roles", #680) — _apply_cost_tiered seeds nothing, so every click toasted an action that did not happen. The copy now states what actually happens: the description reads "Pin cheaper models per role and complexity in the Complexity overrides table below. No rows are seeded automatically.", the confirm dialog says the same, and the success toast derives its message from the applyMode response snapshot — "Cost-tiered mode applied. No rows seeded." or N complexity override row(s) active., counting ROLE-scope rows whose scope_value contains a colon (the compound role:complexity convention). The sweep also fixed the same button's labelHint, which still advertised a re-runnable one-time additive seed, and replaced the hardcoded active={false} with active={currentMode === 'cost_tiered'} so the mode's live state highlights like every other mode button (though derive_mode does not return cost_tiered today, so the badge stays dark in practice; the mode stays wired for a future above-floor re-seed). roboco/services/llm.py is untouched — the seed retirement is a deliberate product decision, not a regression to re-fix. Interaction tests in ai-routing-card.test.tsx pin the corrected copy (the card renders no "Haiku" anywhere), the response-derived toast, a zero-rows case, an apply-failure error toast, and the active-state highlight. See docs/map/panel.md for the companion note.

• The dispatcher no longer stacks a second claim onto the same not-yet-active agent, and the stale-claim reaper now releases a claim an agent parked while working elsewhere. Live production, 2026-09-05: at 22:07:16 to 22:08:16 UTC the dispatcher routed three pending backend tasks (c15af9ff, c0d0f4f4, 60c3feb2) to the same idle be-dev-1 within 60 seconds; each claim runs _finalize_claim (sets agent.current_task_id, then creates the task's branch, 53s or more on that host), so the second claim blocked 42s on the agents-row lock and the third queued behind it, both eventually dying on the 60s lock_timeout while the dispatcher's own 30s client gave up and re-routed, growing the pile; separately, be-dev-1 ended up holding four claimed tasks (7daca5ff claimed 21:17, 5d95b385 21:41, c15af9ff 22:07, c0d0f4f4 22:07) whose heartbeat never advanced past the claim for up to an hour while its container worked a different task and be-dev-2 idled, because the reaper's live-container skip only checks whether the agent has any live container, not whether it is working THIS task. Fix: a new in-memory registry, AgentOrchestrator._claims_in_flight (agent slug mapped to a task_id and monotonic deadline), is set the moment a claim is issued and cleared once it has a definite outcome (success, after the follow-up spawn, or an immediate server rejection); a client-side timeout or exception leaves the entry parked until dispatch_claim_inflight_ttl_seconds (new setting, default 600) elapses, since the server may still be finishing the claim. _select_agent_for_cell, the single chokepoint for dev/qa/doc/pm/pr_reviewer selection, now treats an in-flight agent exactly like an active one and returns None (never candidates[0]) when every candidate is busy or in flight, so the task waits for the next scan instead of stacking a claim; _resolve_routing_target's dev branch now distinguishes that wait from a genuinely unroutable team, since falling back to main-pm for a code task hits the MAIN_PM_NO_CODE claim-reject loop. The reaper's new _maybe_release_parked_claim releases a claimed row whose heartbeat never advanced past its own claim, stale past stale_claim_reap_seconds of fleet-active time, once the same agent's other claimed or in_progress row proves it is alive and busy elsewhere; it never fires on a genuinely advancing heartbeat, an agent's only claim (left to the existing dead-run logic), or while _claims_in_flight still holds an unexpired entry for that agent.

• A child dev's spawn no longer auto-blocks on its parent's branch the instant that parent's branch creation runs long, since _apply_claim_fields now commits the claim (status, assigned_to, claimed_at) before _provision_claim creates the branch and commits again, so a slow git network op under load can leave a healthy, mid-claim parent branchless for minutes rather than a sub-second gap. _check_parent_branch_ready (roboco/runtime/engines/dispatch_claim.py) still fast-retries a mid-claim parent (status claimed/in_progress with assigned_to set) 3x250ms for the sub-second commit gap, but once that retry is exhausted it now compares the parent's claimed_at age, measured in fleet-active time via _active_age so a paused fleet never eats into the budget, against the new parent_branch_provisioning_grace_seconds setting (default 900, matching flow_verb_slow_timeout_seconds, the same budget a slow claim/branch verb already gets server-side); inside the grace the child is skipped for this dispatch tick with a debug log instead of auto-blocked, and the next tick retries. A parent past the grace, or one that was never mid-claim to begin with (pending/unassigned, or terminal - no branch is ever coming), auto-blocks exactly as before.

• Post-merge panel hotfixes (Fixed).

  • "dedupe the ShieldCheck import left by back-to-back root merges"
  • "task detail grew to 10 tabs after the root merges; widen the grid and the tab-count test"