feat(routing): add provider cooldown failover and network recovery

[yury-procoders]

Summary

Adds provider-aware failure classification and deterministic rerouting for the tier model pool. The dispatcher now distinguishes transient provider pressure, permanent provider failures, model-local failures, and correlated connectivity loss, then persists the resulting cooldown/circuit/probe state across resume.

This is a stacked PR and must merge after:

1. feat(zai): a headless GLM worker backend #5 — z.ai backend
2. feat(pool): deterministic weighted tier model pools #6 — tier/model pool
3. this PR — rate-limit and failure rerouting

The implementation branch intentionally contains both prerequisites so the three-provider integration can be tested as one executable state machine. Please review/merge in the order above.

Three-provider failure scenarios covered

The default pool can involve Codex, Claude, and z.ai. The same rules apply to other explicitly configured backends.

Scenario	Effect on the failed provider	Effect on the rest of the pool	Liveness guarantee
First rate-limit/overload/timeout	Short bounded retry of the same assignment	No provider is excluded yet	Foreground delay is capped at 60 seconds
Second transient failure	Persist backend cooldown and advance the frozen ring	Healthy providers remain eligible	No busy wait; an absolute next_retry_at is returned
Long provider reset window	Cooldown keeps the provider out until its real reset timestamp	Work reroutes to another healthy provider	A multi-hour cooldown cannot block the process/thread
Cooldown expires	One real job receives a half-open probe lease	Competing jobs remain blocked from probing that provider	Exactly one probe; lease expiry requires liveness reconciliation
Auth failure	Permanent circuit opens for that backend	Other providers may continue	No automatic auth retry loop
Out of credits	Permanent circuit opens; bounded ring advances	Explicit healthy fallback may run after ring exhaustion	Open circuits and active cooldowns cannot be bypassed by fallback
Model unavailable	Excludes only the exact backend/model pair	Another model on the same backend or another provider can run	Does not unnecessarily disable the whole provider
One true connection/DNS/TLS/reset failure	Records causal no_response evidence	Other providers still run	One provider cannot declare the internet down
Two distinct providers fail with no_response inside 60 seconds	Opens a run-wide network pause	All provider launches pause	After expiry, exactly one real-job recovery probe is leased
A provider succeeds in the same 60-second window	Success vetoes stale outage evidence	Pool remains operational	Prevents a false global pause during partial provider trouble
z.ai error 1234 / provider-reported network failure	Treated as provider-local evidence, not proof of internet loss	Other providers remain eligible	Cannot trigger the correlated offline pause
Internet returns after a global pause	Global recovery probe has priority over a simultaneous expired provider cooldown	Successful probe clears the run pause; provider cooldown remains independently tracked	Prevents the halt / no next_retry_at deadlock
Retry budget exhausted or state is replayed after a crash	Atomic, attempt-id-based ledger rejects duplicate, unknown, malformed, and future charges	No extra provider launch is minted	Resume is idempotent and fails closed on poisoned state

Temporary internet loss

A transient local outage is deliberately handled more conservatively than a single provider outage. The run pauses only after two distinct providers produce real no-response failures in the same batch and 60-second correlation window. Provider-formatted errors do not count. A completed success within that window is a live veto. When the pause expires, one real job—not a synthetic request—is leased as the recovery probe. Probe failure renews the pause; probe success clears it. This prevents fan-out storms while ensuring the run cannot remain indefinitely halted without a retry timestamp or a leased recovery attempt.

Cooldown and fallback interaction

Cooldowns are backend-wide by design: subscription/provider quota signals generally do not prove that a second model on the same account has independent capacity. This may be conservative, but it avoids multiplying requests against a shared quota. Model-unavailable remains exact-model scoped. Explicit fallback assignments are rechecked against the target backend's circuit and cooldown; an expired cooldown receives a single half-open lease. If global network recovery and provider recovery become eligible simultaneously, the global network probe wins and the provider cooldown probe stays idle.

Implementation

• Adds strict UTC timestamp parsing and canonical persisted cooldowns.
• Adds normalized failure classification for Codex, Claude, and z.ai, including exact official z.ai message variants.
• Adds bounded retry/reroute policy with a run-level retry ledger.
• Adds frozen-ring selection, circuit breakers, half-open leases, correlated network evidence, and crash-safe transition validation.
• Wires result collection, status/dashboard rendering, resume behavior, schemas, routing docs, runbooks, spec, audits, plan, and execution manifest.
• Keeps non-pool jobs on their existing backend; only foreground waiting is capped so a long reset cannot hang the orchestrator.

Verification

• All 36 Python --selftest entrypoints passed on the final commit.
• Pool state machine: 124 ok, 0 fail.
• Dashboard: 75 checked, 0 failed.
• z.ai wire smoke: 8 ok, 0 fail.
• z.ai worker stub: 39 ok, 0 fail.
• Advisor-worker and path-transport shell integration suites passed.
• Shell syntax, execution manifest validation, and git diff --check passed.
• Negative controls were exercised: mutations to exact z.ai codeless classification, retry cap, pool-state invariants, and dead-link validation were detected by their checks.
• Independent final review: approved with no remaining Critical/Important findings.

Non-goals

• No quota-aware load balancing or adaptive weights.
• No resizing/reordering of a frozen pool during a run.
• No automatic recovery from auth failures without explicit reconciliation.
• No use of provider marketing/status text as proof of local internet loss.