feat(spurctld): get Raft HA working for K8s and bare-metal

[shiv-tyagi]

Raft HA for spurctld

This gets Raft-based HA working on the K8s cluster and bare metal. Changes in #84 served as a foundation. The Raft log is now the sole persistence path — even single-node deployments run a 1-member Raft cluster, eliminating the dual WAL/redb code path entirely.

Raft core (new: raft.rs, raft_server.rs, raft_internal.proto)

• Disk-backed storage for vote, log entries, and snapshots under state_dir/raft/
• gRPC transport for Raft peer RPCs on dedicated port 6821 (separate from client API on 6817)
• Single-member bootstrap with add_learner retry so bare-metal nodes can start simultaneously
• Fail-fast when node_id cannot be determined from hostname or config

Always-on Raft (main.rs, raft.rs, cluster.rs)

• Raft runs in all modes: single-node auto-synthesizes a local peer list and self-elects instantly.
• SpurStore takes the applier (ClusterManager) at construction — Raft recovery replays entries into live state correctly on startup
• Snapshot data is restored into ClusterManager on startup, fixing a latent data-loss bug where snapshot metadata was loaded but the actual state was not deserialized
• ClusterManager::new() creates empty state; all recovery flows through Raft log replay + snapshot restore
• Removed: FileWalStore, SnapshotStore (redb), open_snapshot_with_retry, maybe_snapshot, take_snapshot, dual-path propose(), WalEntry/WalStore/WalStoreError

State replication (cluster.rs)

• propose() always calls raft.client_write() — no local WAL branch
• Unified apply path renamed apply_wal_op_internal → apply_operation (called by Raft's apply_to_state_machine on commit)
• Snapshot includes full cluster state (reservations, steps, licenses, aliases) not just jobs/nodes
• Deadlock fixes: mutation methods drop locks before calling propose()

Leader gating (server.rs, scheduler_loop.rs)

• RaftHandle is non-optional throughout — no if let Some(raft) branching
• All gRPC RPCs forward to leader when hit on a follower via LeaderProxy
• x-spur-leader header hints the client API address (not the Raft port)
• Scheduler, health checker, time-limit enforcer, power manager only run on leader

Deleted: spur-state crate

• Entire crate removed (FileWalStore + SnapshotStore/redb) — Raft storage in raft.rs is the only persistence layer
• redb removed from workspace dependencies
• WalOperation enum kept in spur-core/src/wal.rs as the Raft log entry payload type
• t53_state test module removed (coverage replaced by Raft storage unit tests)

K8s manifests

• spurctld.yaml: 3 replicas, PVCs for Raft state, dual ports
• configmap.yaml: peers via StatefulSet DNS
• pdb.yaml: maxUnavailable: 1 for quorum safety
• CI patches replicas to 1 for single-node tests

K8s operator bug fixes (job_controller.rs, agent.rs)

• Fixed double job submission caused by finalizer 422 retry seeing stale status
• Operator refreshes controller gRPC connection on address changes
• Agent namespace-resolve backoff increased to 15s
• Scheduler retries dispatch 3 times on transient agent failures

Tests

• Unit tests for apply logic, snapshot round-trip, Raft storage, hostname parsing (all use single-node Raft via #[tokio::test])
• K8s integration tests 1-12: SpurJob lifecycle, multi-node jobs, cancellation, failure detection, cross-namespace, Raft cluster formation, leader election, log replication, failover recovery
• Net: -963 lines (eliminated dual persistence path)

Issues resolved

• Snapshot scope mismatch: redb only saved jobs+nodes; single-node users silently lost reservations/steps/licenses on restart
• Startup ordering: Raft replayed committed entries before applier was set, silently dropping them
• Stale redb keys: SnapshotStore::save() never deleted removed jobs/nodes (redb has no clear())
• Snapshot data not restored on startup: metadata was loaded but state machine was empty until Raft replayed post-snapshot entries

Config

[controller]
peers = ["host1:6821", "host2:6821", "host3:6821"]
# node_id = 1           # auto-detected from hostname ordinal if unset
# raft_listen_addr = "[::]:6821"

[shiv-tyagi]

Cluster Tests failure seems to be pre existing from yesterday.

K8s integration tests pass which include raft HA integration tests.

[powderluv]

lmk if the cluster CI is flaky and I can investigate it. Thanks for the changes. We can land once we get the CI green.

[shiv-tyagi]

Yes, there is a last minute thing which I want to fix before merging. I will let you know when it is ready. Thanks.

[shiv-tyagi]

@powderluv Cluster Tests are still failing, for a reason I suspect not related to this work.

K8s intergration tests PASS, which means we are good to merge. HA with failover handling is covered in those tests. Please hit merge whenever you are ready.

[shiv-tyagi]

I rebased on main and the CI is passing now. Can we merge this @powderluv?