perf: optimize high-concurrency Docker orchestration

[numnx]

What changed

• Replaced full provider-transcript polling and message rewrites with byte-cursor delta reads, bounded output buffers, and tail-only SQLite synchronization.
• Parallelized independent virtual-worker dispatches while preserving durable dispatch leases, exclusive repair attention, atomic provider claims, and lease heartbeats during resource-pressure pauses.
• Added adaptive local-provider admission that reserves interactive reply capacity and freezes background expansion under CPU or memory pressure.
• Reused managed images, immutable provider-tool volumes, browser volumes, Git bundles, and runtime ownership state across warm launches; automatic registry checks now use a six-hour freshness window.
• Shared Docker inventory and bounded reconciliation/cleanup work to eliminate repeated control-plane probes and unbounded filesystem or removal bursts.
• Fused containerized Git artifact export into one warm helper invocation and added three-way materialization so stale workspace bases no longer create avoidable patch conflicts.
• Made concurrent feature-branch patch and LOCAL merge publication atomic: a writer that loses the Git ref race rebuilds on the new tip and retries, preserving both histories without a coarse orchestration lock.
• Moved large historical backfills, read-index builds, pruning, and vacuum work out of the startup critical path into persisted, bounded, idle-only maintenance.
• Updated Playwright coverage for the reworked settings page, kept the production settings rate limit unchanged, and raised only the explicit E2E fixture limit.
• Removed the obsolete serial virtual-worker planning layer and documented the high-concurrency architecture in canonical and published docs.

Why

Under concurrent provider and CI workloads, the runtime amplified host pressure through repeated Docker inventory/inspection commands, full transcript rereads and reparses, recurring SQLite rewrites, per-launch cache verification, serial dispatch, repeated short-lived Git helpers, and startup-time database maintenance. Concurrent CI-fix and task-merge publishers could also race on the same feature ref. These changes make warm launch work proportional to new data, keep independent work parallel, stop background maintenance from competing with active providers, and prevent a late writer from discarding already-published task history.

User and developer impact

• Managed provider tools remain immutable versioned Docker-volume caches; they are not installed per invocation.
• Interactive replies retain a launch reservation during CPU-heavy CI bursts unless memory is critically low.
• A missing or externally corrupted cached artifact is detected at launch, repaired through a singleflight path, and retried once.
• Existing large databases open without waiting for full data backfills or non-unique index construction.
• Admission pauses remain durable without producing false stalled-sprint failures, and repetitive pressure diagnostics are coalesced.
• Overlapping task merges and CI repairs retain both results while remaining parallel.

Validation

• pnpm run ci
  • quality guardrails passed
  • dependency audit passed with no known high-severity vulnerabilities
  • strict TypeScript passed
  • backend: 5,676 passed, 7 skipped; 84.16% line coverage
  • dashboard: 2,228 passed, 1 skipped
  • server and dashboard production builds passed
• pnpm run test:e2e: 67/67 Playwright tests passed in 3.6 minutes
• Docker large-DAG pentest: 129/129 tasks completed and merged in 456.533 seconds; validation passed with no stall or dependency-merge violations
• Concurrent Git publication regression suite: 112 passed, 7 skipped
• Rapid orchestration suite: 275 passed, 7 skipped
• Final strict typecheck, production build, and git diff --check passed after the publication hardening

Risk and rollback

The main risk is scheduling/admission behavior under host-specific pressure and one-time deferred migration/index convergence. SQLite remains the atomic claim authority, unique correctness indexes stay synchronous, maintenance is skipped while providers run, and destructive Docker recovery requires a fresh inventory. Feature-ref publication uses bounded compare-and-swap retries rather than a global branch lock. Revert this PR to restore the previous orchestration behavior.