2026.08.07-rc

One stack can no longer take the whole host down with it.

On 2026-08-24 a single sdk-proxy container retried a permanently failing Anthropic call — HTTP 400 "You're out of extra usage" — for 40 hours at roughly 700 connections per second. TIME_WAIT climbed to 20,775 sockets against a 16,384-port ephemeral pool. Because Docker Desktop on macOS routes container egress through the host's own sockets, that runaway took outbound networking down for every process on the machine.

Running several Hivemind stacks on one box is the goal, not the bug. The bug was that a single stack had no upper bound on its consumption of a host-wide, shared, finite resource. This release adds that bound — four of them, independently.

Fixes

Resource discipline (#451)

Four independent limits, each of which alone would have prevented the outage.

Error classification. Providers::ErrorClassifier and its mirror sdk-proxy/error-classifier.js produce one {retryable, reason} verdict from one place, so no caller has to string-match an error message to decide whether retrying could possibly help. A quota 400 is permanent. EADDRNOTAVAIL and EMFILE are permanent, because retrying local port exhaustion only deepens it. Anything unrecognised defaults to permanent — the opposite posture is precisely what turned one exhausted credential into a host-wide outage. A spec asserts the Ruby and JavaScript sides keep agreeing on which reasons open a circuit.

Per-credential circuit breaker. After three consecutive permanent failures the circuit opens and no outbound socket is opened at all: the check runs before socket allocation and before the concurrency slot is taken, so an open circuit costs nothing. State lives in Redis on the Rails side so every Puma and Sidekiq process shares one verdict, and in-process on the proxy. It fails open on Redis errors, so a Redis blip can never block a healthy provider. Credentials are keyed only by a 12-character SHA256 fingerprint and are never logged or stored raw. One exhausted credential does not silence another; one exhausted provider does not silence another.

Bounded concurrency in sdk-proxy. Every /v1/chat call now passes through a semaphore (SDK_PROXY_MAX_CONCURRENCY, default 4) with a bounded wait queue; overflow is shed with 429 rather than spawning another Claude Code subprocess. Each subprocess opens its own connections with its own internal retry ladder, so this ceiling is the stack's declared share of the host pool, documented in the new MULTI-STACK.md.

Bounded job retries. ApplicationJob now discards PermanentProviderError and caps TransientProviderError at three attempts. Nothing inherits Sidekiq's default 25 retries any more.

The failure is now loud. A red banner names the provider, the reason, and the fix, with a "Retry now" button that clears the circuit on both sides. One [ALARM] log line fires when a circuit opens, alongside a provider_circuit_opened audit entry. GET /api/v1/system/provider_health returns 503 when degraded and distinguishes "the provider refused us" from "we cannot bind a socket".

Mid-stream errors are no longer swallowed. Anthropic::SdkProxyClient ignored the SSE error frame, so a provider failure part-way through a stream looked like a successful empty response and the caller retried it forever. It now carries the proxy's structured verdict, while still keeping any partial reply that already streamed.

In-proxy retries are classified first. The transient-error retry loop added in #448 now classifies before it retries, so it never re-dials a failure that will fail identically every time.

Operator tooling (#451)

• MULTI-STACK.md — the shared-resource arithmetic, host tuning, the per-stack budget table and the rule sum(MAX_CONCURRENCY) x 10 < pool/4, observability, an isolation matrix and pre-flight checklist, a recovery runbook, and the remaining known churn sources.
• scripts/hivemind-host-tuning.plist — a LaunchDaemon for persistent sysctls (portrange.first=16384, tcp.msl=1000), since sysctls applied by hand do not survive a reboot.
• scripts/port-pressure-canary.sh — the single TIME_WAIT number that would have caught this on day one, with --json and --watch modes and warn/critical exit codes.
• scripts/multi-stack-fault-drill.sh — the two-stack fault drill. It needs real Docker, real sockets and a real host to prove the part that actually broke, so it ships as an operator drill rather than a CI job.

Compose

• Added a restart policy to all services (ea5da4a).

CI / Chores

• New sdk-proxy-test job running the proxy's 34 node --test cases on Node 22, so the resource-discipline guarantees are enforced on every pull request (#451).
• Removed a jest devDependency from sdk-proxy that could never run.

Verification

Verified against the live provider, not only stubs: with an invalid key pointed at api.anthropic.com, attempts 1–3 returned real 401s and opened the circuit, and attempts 4–5 were refused locally with total_acquired stuck at 3 — no fourth socket. Under fault injection, 200 calls against a permanently failing credential produce exactly 3 network attempts, and a 500-request stampede peaks at exactly 4 concurrent. 34 sdk-proxy tests and 61 new RSpec examples; full suite 5172 examples with no new failures.

Upgrade notes

Two operator actions are still required on any host running multiple stacks:

1. Install the sysctl LaunchDaemon from scripts/hivemind-host-tuning.plist — see MULTI-STACK.md §2.
2. Review the per-stack budget in MULTI-STACK.md §3 before adding another stack to a box.

Full Changelog: 2026.08.06-rc...2026.08.07-rc