feat(#249): warm the backup stack's XvB donation state on failover#726
Merged
Conversation
A two-host failover pair (same wallet, workers listing both hosts in pools[]) cold-started the backup's XvB donation controller when the fleet failed over to it: the closed-loop split restarted from the feedforward estimate and re-ramped for hours, over/under-shooting the credited tier. Persist the controller's commanded donation fraction so a plain restart resumes the warmed split, and add a one-way, backup-pulls-from-primary standby channel: the backup points xvb.standby.source at the primary dashboard's new read-only /api/xvb-standby endpoint, periodically pulls the primary's controller state, holds it as standby (inspectable in /api/state), and adopts it the first time it actually donates at failover. Seed precedence: this host's own persisted commanded fraction (a restart) beats the primary's standby (a first failover), which beats the cold feedforward estimate. Inert unless configured; an idle backup has no workers so it never donates while the primary is authoritative. An .onion source rides Tor, a LAN source dials direct — no new clearnet egress. The source URL is a capability secret (it can carry the primary's dashboard basic-auth), so pithead renders it into the owner-only .env and masks it in the control-channel config, mirroring the Healthchecks ping URL. Tier-1 unit coverage for persistence, warm-resume seed precedence, cold-start fallback, the standby puller, and the endpoint. Live 2-host failover validation is tier-4 and deferred. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate IP Three security fixes on the #249 warm-standby puller: - CRITICAL clearnet leak: _proxies() only sent .onion sources over Tor; every other host (public IP or hostname) fell through to a direct dial, leaking the backup's real IP and the backup↔primary relationship (#160 threat class). Now mirrors egress.py's carve-out: onion or any non-provably-private source rides Tor; only a private/loopback/link-local IP literal dials direct. Classification shared via egress._xvb_standby_route so the puller and the Security panel can't drift. - HIGH secret in logs: the startup line printed the full source URL (which can embed basic-auth userinfo). Now logs enabled-state only, like healthchecks.py never logging the ping_url. - MEDIUM blind panel: added the xvb_standby route to compute_egress_posture and compute_topology (tor / local / inactive, never clearnet) so the panel tells the truth about where the pull goes. Corrects the now-false "no new clearnet egress either way" claim in the docstrings and CHANGELOG. Tests cover onion/private/public/hostname routing, log secrecy, and the egress classification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Closes #249. On a two-host failover pair — same wallet, workers listing both hosts in
pools[](primary first, backup as failover) — the backup's XvB donation controller used to cold-start when the fleet failed over to it: the closed-loop split restarted from the feedforward estimate and re-ramped for hours, over- or under-shooting the credited tier until it reconverged. This warms it up.The warm-resume mechanism
AlgoService.donation_fraction(the closed-loop integrator — the fraction of each cycle it commands to XvB, distinct from the routed fraction the dashboard already shows) lived only in memory and re-seeded cold on every restart. It's now persisted to the existingkv_store(a newcommanded_fractionXvB field) each cycle, so a plain process restart resumes the warmed split.GET /api/xvb-standby(behind the same Caddy auth + loopback bind as/api/state) carrying{commanded_fraction, avg_1h, avg_24h, mode, donation_level, ts}. The backup setsxvb.standby.sourceto that URL; a newXvbStandbyPullerservice periodically pulls it and holds the result as standby (StateManager.set_xvb_standby, persisted; inspectable in/api/stateasxvb_standby).AlgoService._seed_donation_fractionseeds with precedence: own persisted commanded fraction (a restart / an already-authoritative stack) → primary standby (a first failover) → cold feedforward (fresh install). An idle backup never steers, so its own fraction stays 0.0 and the standby is what fires at handover — resuming warm, not from zero.Safety / privacy
.onionsource rides the bridge Tor SOCKS, a LAN source is a private-network hop..envand masks it in the control-channel config — mirroring the Healthchecks ping URL exactly (CONTROL_SECRET_PATHS+control_service.SECRET_PATHSkept in step).Design note
The credited 1h/24h averages are XvB's per-wallet numbers; because both stacks share the donor id, the backup's own XvB fetch already returns them warm — so the transported state that actually matters is the commanded fraction. The averages are carried in the standby blob for inspectability.
Tests
Tier-1 unit only (the honest lowest tier for controller/state/parse logic):
tests/service/test_storage_service.py::TestXvbWarmStandby— commanded fraction persists across a restart; standby set/get roundtrip + survives restart; unset → None; bad/unserializable payload degrades safely.tests/service/test_algo_service.py::TestWarmResume— seed prefers own persisted fraction; adopts standby when no own state; clamps to the VIP reserve; cold-start uses feedforward when no state;get_decisionseeds from standby on a simulated failover.tests/service/test_xvb_standby.py—parse_standbyvalid/non-dict/bad-numeric/defaults; puller disabled/inert; fetch stores; non-200/network-error/bad-payload keep the last standby; onion→Tor vs LAN→direct routing; one run-loop iteration.tests/web/test_server.py::TestXvbStandbyApi— endpoint payload, reflects controller state, sanitized 500.make lintclean (onlylint-protoskipped — needs Docker, no proto touched).make test-dashboard: 1490 passed, 96.7% total.make test-patch-coverage: 98% on changed lines.make test-stack: pass (exit 0) — validates the new.envrendering + secret masking.Deferred (owed)
🤖 Generated with Claude Code