HydraCache 0.66.0

HydraCache 0.66.0 - Cluster Corner-Case Hardening

Status: in progress. This file records the candidate scope; it is not a shipped-release claim.

Scope

0.66 lifts the existing Raft/Sled/control-plane proofs to real daemon, HTTP transport, mixed-version,
operator, fuzz, and resource-budget lanes. The implementation branch was forked from the 0.65 branch
at 2926551.

The 0.65 client/RESP execution boundary remains unchanged: values, TTL state, idempotency state, and
locks are node-local to one daemon. Consequently, 0.66 control-plane nemesis/history tests do not claim
cross-daemon client-value linearizability.

The release also does not claim a live backup/restore engine, encrypted live restore, ownership
routing, a distributed client backend, or lease reads. The plan carries executable boundary guards so
those absent capabilities cannot be inferred from an accepted admin request or helper-level tests.

Implementation status

• W0 exposes the existing typed Sled compaction path through an authenticated, off-by-default
  control and proves restart recovery at the exact applied boundary.
• W1 records real snapshot send/install outcomes and proves compacted-lagger catch-up plus
  sender/receiver interruption with real daemons.
• W2, W3, and W7 exercise only committed control-plane metadata. The process nemesis uses stable
  invoke/complete IDs, public membership epochs, deterministic replay, shrinking, and frozen seeds.
• W4 makes the backup authority boundary executable. /admin/backup currently accepts a configured
  request but returns durable_artifact_created: false and restore_point_available: false.
  BackupDataset.values remains caller supplied; no live backup or restore claim is made.
• W5 injects bounded block/failure modes into real Sled save/install/commit boundaries. The live
  slow-disk leg remains a separately capability-gated Chaos Mesh/operator receipt.
• W6 runs explicit previous/current binaries per node, records provenance, and proves snapshot
  catch-up plus rolling replacement during membership change. Development fallback is pinned to
  2926551; release evidence requires a full-history v0.65.0 tag and does not accept that fallback.
• W8 compares committed metadata with an independent reference model in both runtime and daemon
  adapters. W9 adds the fifth cargo-fuzz target, committed replay corpus, and real HTTP socket bound.
• W10 separates real process scheduling pauses from the local monotonic TTL/lock clock contract and
  fences the public authoritative metadata projection until local apply is current and current-term
  Raft traffic is fresh; it does not introduce or claim lease reads.
• W11 checks scale churn, voter removal, ghost-voter rejection, and public epoch observations in the
  model and an opt-in CNI-enforced kind lane.
• W12 generalizes the resource artifact, enforces one snapshot HTTP request per sender/peer across
  clones, cancels obsolete leader requests on role/term loss, and uses a bounded loopback-only
  decoded-snapshot response delay to keep a real reservation in flight. Its machine-readable
  disclosure identifies 200 ms event-checkpoints from /admin/raft/compaction: legacy
  tracked_connections and held_snapshot_messages are retained sampled observations, not
  continuous maxima. Observed cluster request/task current stays within 2, the daemon-local
  monotonic sender-task HWM stays within 1 for the one-lagger scenario, and current work finishes
  at 0. Linux evidence bounds current RSS/FD residuals and baseline-to-peak live-process VmHWM;
  HWM need not fall after quiescence. Portable evidence cannot satisfy the Linux /proc proof.
• W13 registers every W0-W13 canary and the fast, daemon-process, operator-kind, and fuzz receipts.
  The operator receipt rejects empty pod/resource output and binds the controller's own runtime
  marker to a unique run nonce, live Linux PID, and exact candidate-binary inode before hashing its
  declared artifacts. Exact clean-candidate receipts remain release evidence, not implementation
  status.

Release-time blockers

• Mixed-version ship proof must build the previous daemon from the full-history annotated v0.65.0
  tag (peeling to edf0fd1) and record that provenance on the exact candidate; the pinned 2926551
  development fallback is not valid release evidence.
• Operator kind/Chaos, bounded cargo-fuzz, and Linux process/resource receipts must be produced by
  their registered release lanes on the exact candidate commit.
• This file remains in progress until --require-ship accepts a clean candidate and every receipt
  matches that commit and its registry digest.

Ship status is determined only by
release-evidence --release 0.66 --receipts-dir target/release-evidence/receipts --require-ship on the exact
clean candidate commit.