Mirror runbook: batch path is live; surge-free rolls; the writer has no volumeClaimTemplates ai-meta#343/#344 closed out. Records what proved the transport fix (the path=batch counter moving 0 -> 460, after 0 of 80,264 records ever), the repair going from partial/1-of-39 to repaired/64-of-64, and parity 14/14 agree. Adds two operational traps that cost an outage: a maxSurge=25% roll on a 1-replica pool rounds to +1 pod at 3Gi and cannot schedule at 99% node memory; and the writer StatefulSet mounts its PVCs by NAME with no volumeClaimTemplates, so deleting them leaves the pod unschedulable forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mirror runbook: the timeout is an un-batched fan-out; tier records cannot be corrected in place ai-meta#344 and #345. Two things an operator needs before reaching for the obvious fix in either direction. The mirror timeouts are not a slow network: the relay appends one record at a time (batch path 0 of 80,264) against an fsync-per-append writer at ~118ms each. Raising APPEND_TIMEOUT buys time for a loop that should not exist. Arming the batch path needs chunking first, because the request cap stays at 1 MiB by design. And a record already in the tier cannot be corrected: dedup ignores rather than replaces, and the event log has no replace/upsert/delete op. A re-mirror to fix content is a no-op that reports success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mirror runbook: a repair closed the count gap and opened a content gap ai-meta#343. Three findings from the 2026-09-12 session, all reachable from the mirror runbook because that is where an operator looks when a repair reports success and parity still disagrees. 1. The #342 repair re-mirrored the PERSISTED row, whose result carries a reference, while the live path mirrors the in-memory row, whose result is inlined. Repairing again rewrote the same reference, so it could never converge. Every event still differing was one the repair had re-mirrored: 3 of 39 and 3 of 26, zero outside. Fixed in server#430; repair now reports 'hydrated'. 2. not_comparable is not a divergence. Three of a fixed 40 were unreadable, not divergent, and a boolean could not say so. The sweep now publishes accounted_for so the denominator is stated rather than subtracted. 3. A mirror send failure now carries its cause as a label. Do not change APPEND_TIMEOUT unless kind="timeout" is what moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wiki(#155): the async event-log mirror, and correct the stale "primary not activated" claims Adds Runbook-Async-Event-Log-Mirror — the operational surface for the mirror that went asynchronous on prod 2026-08-19, and which this wiki did not mention at all. Leads with the one-flag rollback and with the pairing that makes it dangerous: async on with the tolerance at 0 makes the comparator judge a healthy tier on its own liveness. Records the parts that are easy to get wrong rather than only the settings. That the window must be MEASURED and then set above the observed p99 — prod came back p99 699ms, so the window is 30s. That it sits deliberately below SETTLE_SECS so the background sampler still compares every execution. That the two lag controls are a discrimination rather than a clean-parity check, because a comparator taught to ignore everything passes the naive version perfectly. And that reserve rather than send is load-bearing, because the first implementation lost 32 of 60 records while every counter still read correctly. Corrects two current-state claims that had gone stale: * Home said the reference tiers are O-of-n per op and "not primary-serve". Still true for KV/object/vector; false for the event log since the runtime cache landed and since the tier went primary on 2026-08-13. * Roadmap said the primary cutover "stays a later step" and that primary is "recognised but not activated". Both annotated as superseded rather than deleted, since they remain accurate as the plan of record. Sessions-Log is left untouched — it is history and correct as written. Home gains a dated current-state banner so a reader meets the live state before the older planning prose. Also records that Option 2's batch flag is still OFF on prod, batch=0 on both replicas, so the substrate is landed and measurable but inert. Tracked at noetl/ai-meta#284.