Skip to content

feat(#249): warm the backup stack's XvB donation state on failover#726

Merged
VijitSingh97 merged 4 commits into
develop-v1.11from
feat/249-xvb-warmup
Jul 21, 2026
Merged

feat(#249): warm the backup stack's XvB donation state on failover#726
VijitSingh97 merged 4 commits into
develop-v1.11from
feat/249-xvb-warmup

Conversation

@VijitSingh97

Copy link
Copy Markdown
Collaborator

What this does

Closes #249. On a two-host failover pair — same wallet, workers listing both hosts in pools[] (primary first, backup as failover) — the backup's XvB donation controller used to cold-start when the fleet failed over to it: the closed-loop split restarted from the feedforward estimate and re-ramped for hours, over- or under-shooting the credited tier until it reconverged. This warms it up.

The warm-resume mechanism

  1. Persist the commanded fraction. AlgoService.donation_fraction (the closed-loop integrator — the fraction of each cycle it commands to XvB, distinct from the routed fraction the dashboard already shows) lived only in memory and re-seeded cold on every restart. It's now persisted to the existing kv_store (a new commanded_fraction XvB field) each cycle, so a plain process restart resumes the warmed split.
  2. Backup pulls the primary's state. The primary exposes a small read-only GET /api/xvb-standby (behind the same Caddy auth + loopback bind as /api/state) carrying {commanded_fraction, avg_1h, avg_24h, mode, donation_level, ts}. The backup sets xvb.standby.source to that URL; a new XvbStandbyPuller service periodically pulls it and holds the result as standby (StateManager.set_xvb_standby, persisted; inspectable in /api/state as xvb_standby).
  3. Adopt at failover. AlgoService._seed_donation_fraction seeds with precedence: own persisted commanded fraction (a restart / an already-authoritative stack) → primary standby (a first failover) → cold feedforward (fresh install). An idle backup never steers, so its own fraction stays 0.0 and the standby is what fires at handover — resuming warm, not from zero.

Safety / privacy

  • Never donates while the primary is authoritative: the backup has no workers then, so its controller sees zero shares and stays on P2Pool regardless. Standby state is held separately and only folded into the live controller when it first actually donates.
  • One-way (backup pulls from primary), inert unless configured (blank source = off; a single stack pays nothing).
  • No new clearnet egress: an .onion source rides the bridge Tor SOCKS, a LAN source is a private-network hop.
  • The source URL is a capability secret (it can carry the primary's dashboard basic-auth as userinfo), so pithead renders it into the owner-only .env and masks it in the control-channel config — mirroring the Healthchecks ping URL exactly (CONTROL_SECRET_PATHS + control_service.SECRET_PATHS kept in step).

Design note

The credited 1h/24h averages are XvB's per-wallet numbers; because both stacks share the donor id, the backup's own XvB fetch already returns them warm — so the transported state that actually matters is the commanded fraction. The averages are carried in the standby blob for inspectability.

Tests

Tier-1 unit only (the honest lowest tier for controller/state/parse logic):

  • tests/service/test_storage_service.py::TestXvbWarmStandby — commanded fraction persists across a restart; standby set/get roundtrip + survives restart; unset → None; bad/unserializable payload degrades safely.
  • tests/service/test_algo_service.py::TestWarmResume — seed prefers own persisted fraction; adopts standby when no own state; clamps to the VIP reserve; cold-start uses feedforward when no state; get_decision seeds from standby on a simulated failover.
  • tests/service/test_xvb_standby.pyparse_standby valid/non-dict/bad-numeric/defaults; puller disabled/inert; fetch stores; non-200/network-error/bad-payload keep the last standby; onion→Tor vs LAN→direct routing; one run-loop iteration.
  • tests/web/test_server.py::TestXvbStandbyApi — endpoint payload, reflects controller state, sanitized 500.

make lint clean (only lint-proto skipped — needs Docker, no proto touched). make test-dashboard: 1490 passed, 96.7% total. make test-patch-coverage: 98% on changed lines. make test-stack: pass (exit 0) — validates the new .env rendering + secret masking.

Deferred (owed)

  • Tier-4 live 2-host failover validation cannot run in this environment (needs the two-host + borrowed-rig harness). The warm-resume logic, standby pull, endpoint, and config rendering are implemented and tier-1/2 tested; an end-to-end failover on the bench (primary + backup, simulated primary outage, confirm the backup resumes warm and never double-donates) is owed before release.

🤖 Generated with Claude Code

VijitSingh97 and others added 2 commits July 21, 2026 10:43
A two-host failover pair (same wallet, workers listing both hosts in
pools[]) cold-started the backup's XvB donation controller when the fleet
failed over to it: the closed-loop split restarted from the feedforward
estimate and re-ramped for hours, over/under-shooting the credited tier.

Persist the controller's commanded donation fraction so a plain restart
resumes the warmed split, and add a one-way, backup-pulls-from-primary
standby channel: the backup points xvb.standby.source at the primary
dashboard's new read-only /api/xvb-standby endpoint, periodically pulls the
primary's controller state, holds it as standby (inspectable in /api/state),
and adopts it the first time it actually donates at failover.

Seed precedence: this host's own persisted commanded fraction (a restart)
beats the primary's standby (a first failover), which beats the cold
feedforward estimate. Inert unless configured; an idle backup has no
workers so it never donates while the primary is authoritative. An .onion
source rides Tor, a LAN source dials direct — no new clearnet egress.

The source URL is a capability secret (it can carry the primary's dashboard
basic-auth), so pithead renders it into the owner-only .env and masks it in
the control-channel config, mirroring the Healthchecks ping URL.

Tier-1 unit coverage for persistence, warm-resume seed precedence,
cold-start fallback, the standby puller, and the endpoint. Live 2-host
failover validation is tier-4 and deferred.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate IP

Three security fixes on the #249 warm-standby puller:

- CRITICAL clearnet leak: _proxies() only sent .onion sources over Tor;
  every other host (public IP or hostname) fell through to a direct dial,
  leaking the backup's real IP and the backup↔primary relationship (#160
  threat class). Now mirrors egress.py's carve-out: onion or any
  non-provably-private source rides Tor; only a private/loopback/link-local
  IP literal dials direct. Classification shared via egress._xvb_standby_route
  so the puller and the Security panel can't drift.
- HIGH secret in logs: the startup line printed the full source URL (which
  can embed basic-auth userinfo). Now logs enabled-state only, like
  healthchecks.py never logging the ping_url.
- MEDIUM blind panel: added the xvb_standby route to compute_egress_posture
  and compute_topology (tor / local / inactive, never clearnet) so the panel
  tells the truth about where the pull goes.

Corrects the now-false "no new clearnet egress either way" claim in the
docstrings and CHANGELOG. Tests cover onion/private/public/hostname routing,
log secrecy, and the egress classification.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant