RFC 0002: Active/passive Fleet HA for real-time control

[ankitgoswami]

Reviewable diff: +419/-0 across 1 files (excludes generated, test, and story files).

Summary

This is an RFC-only PR. Please review it as an architecture and scope decision for the future HA implementation, not as an implementation diff.

The most useful feedback is on whether the proposed contract is correct, simple enough to deploy, and explicit about failure behavior. Nits on exact installer flags, Patroni timing values, file names, and low-level implementation APIs can wait for implementation PRs unless they expose a flaw in the RFC's contract.

Suggested review path

RFC section	What to check
Decision summary	Are the top-level choices clear enough to accept, reject, or revise?
HA contract	Is the v1 scope right: real-time control and live HA alerting are in, telemetry/Grafana/history are best effort?
Architecture	Does the separation between DB primary election, Fleet activeness, and endpoint routing avoid split-brain and single points of failure?
Stable endpoint	Does the on-prem keepalived/VRRP VIP contract cover same-subnet failover, VIP ownership loss, and endpoint fencing without making the VIP the correctness authority?
Database layer	Is Patroni + quorum, multi-host DB routing, and auto-degrade vs strict-sync framed at the right level?
Fleet activeness	Do the lease identity, epoch, expiry, renewal, and active-runtime fencing rules prevent dual active dispatch?
Health and operator status	Are load-balancer-safe health checks, authenticated HA diagnostics, and live HA alerts separated cleanly?
Deployment model	Is the pre-config preflight vs post-start validation split deployable for first-time on-prem setup?
Failure behavior / Validation gates	Are the expected single-failure cases and support gates complete enough to protect the HA claim?
Alternatives considered / Unresolved questions	Are the rejected alternatives and deferred questions the right ones for an RFC-stage decision?

Reviewer focus

Decision area	Review question
Single-active Fleet	Is avoiding active/active the right simplification for v1, given scheduler, command execution, MQTT intake, ControlStreams, and curtailment reconciliation?
Fleet-owned DB lease	Does decoupling app activeness from Patroni primary placement make app failover cleaner without introducing hidden correctness gaps?
Patroni + three-member quorum	Is this the simplest substrate that still avoids a DB split-brain / two-primary failure mode?
Multi-host DB routing	Is relying on pgx/libpq writable-host selection preferable to adding HAProxy for DB writer routing?
On-prem VIP endpoint	Does keepalived/VRRP cover the supported same-subnet on-prem case, including local VIP loss, without requiring an existing load balancer?
Default auto-degrade mode	Is continuing real-time control after standby loss acceptable when HA status clearly reports reduced durability and failover readiness?
Critical state boundary	Are all state types needed for real-time control included, especially MQTT source runtime state and pending edge retry/dedup state?
Alerting boundary	Does the RFC make live HA alerting independent of Grafana/dashboard/history availability?
Deployment simplicity	Can the proposed install flow remain understandable for both on-prem Docker hosts and later cloud packaging?

Out of scope for this PR

Topic	Why it is not the main review target here
Exact Patroni timing values	The RFC requires lab validation before support; final values should come from failover measurements.
Exact installer flags/templates	The RFC defines install phases and contracts; command surfaces belong in implementation PRs.
Full telemetry/Grafana HA	The v1 HA contract is real-time control plus live HA alerting, not full history/observability continuity.
Passive read-only product APIs	Passive is intentionally health/status-only in v1 to avoid a broad read-only allowlist audit.
Cloud substrate selection	Cloud must satisfy the same HA contract, but the concrete cloud packaging can be decided when that implementation starts.

Areas of the code involved

Area / package / file	What changed	Why it matters for review
docs/rfcs/0002-active-passive-fleet-ha.md	Adds RFC 0002 for active/passive Fleet HA.	This is the architecture contract reviewers should approve or reshape before implementation starts.

Key technical decisions & trade-offs

• Single-active Fleet over active/active Fleet.
• Fleet-owned active lease over “Fleet follows DB primary.”
• pgx/libpq-style multi-host DB routing over HAProxy as the default DB writer path.
• Bundled keepalived/VRRP VIP for the supported same-subnet on-prem endpoint.
• Auto-degrade + HA status reporting as the default durability mode; strict sync remains opt-in.
• Real-time control HA and live HA alerting in v1; full telemetry/Grafana/history HA deferred.

Testing & validation

No runtime tests were run; this is a documentation-only RFC.

The RFC defines the validation gates required before HA can be marked supported: active/passive fencing, DB failover, quorum behavior, VIP movement and VIP-loss fencing, ControlStream reconnect behavior, MQTT intake recovery, critical-write durability while replication is healthy, degraded-mode HA status/alerts, and authenticated HA diagnostics.

---

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (19f70355a9ca3ccda247a0494c0ffbc7de7c5d3a...cbd969ad27d816c43c7df3f506417b2afda92e74, exact PR three-dot diff)
• Model: gpt-5.5

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: HIGH

Findings

[HIGH] Authoritative review diff appears incomplete

• Category: Other
• Location: docs/rfcs/0002-active-passive-fleet-ha.md:1
• Description: .git/codex-review.diff contains only the new HA RFC document, but the stated commit range also reports changes across multiple client files and tests. Because the provided review artifact is the authoritative scope yet does not match the stated PR range, the actual latest changes cannot be reviewed safely from this diff.
• Impact: Security, correctness, or reliability regressions in omitted changed files could be missed, including frontend behavior changes around scanner/modals and shared UI components.
• Recommendation: Regenerate .git/codex-review.diff from 19f70355a9ca3ccda247a0494c0ffbc7de7c5d3a...cbd969ad27d816c43c7df3f506417b2afda92e74 and rerun the review against the complete artifact.

Notes

Within the visible authoritative diff, the only reviewed content is docs/rfcs/0002-active-passive-fleet-ha.md. I did not identify a concrete runtime vulnerability in that document alone, but the incomplete review artifact prevents a reliable assessment of the full stated PR diff.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[Copilot]

Pull request overview

This PR adds RFC 0002, proposing an active/passive high-availability installation mode for Proto Fleet focused on preserving real-time control correctness and fast recovery (single-active control plane, Patroni-backed DB HA, and a Fleet-owned active lease), while explicitly allowing degraded behavior for history/observability subsystems.

Changes:

• Defines the HA contract boundary (critical control vs real-time runtime vs best-effort history vs local artifacts) and the supported 3-host topology (2 app/DB + 1 witness).
• Specifies the separation of concerns for HA correctness: Patroni DB primary election, Fleet DB-backed active lease for single-active runtime, and endpoint routing via /health/active (VIP/keepalived on-prem or cloud LB).
• Outlines runtime gating, health surfaces, failure behavior, and validation gates/phased rollout for eventual implementation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da0b851eb1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[mcharles-square]

This approach looks pretty good! As discussed, would like to ensure that we handle stopping jobs cleanly when we fail over