v0.23.1 — relay rollout readiness and faster reconnects

Relay and edge hardening on top of the scale-out work in v0.20.0–v0.23.0. Everything in this release is relay/edge-side.

No upgrade action required. No schema change in either store, no wire-protocol change, no re-enrollment. Existing databases, tokens and enrollments carry over untouched, and relays and agents can be rolled in either order. The usual relay-before-agents ordering is still the safer habit, but nothing here depends on it.

Rolling relays

• Kubernetes rollout readiness (#557, #552, #553, #555, #556). /readyz and /livez on both ops listeners: the relay's readiness follows its pool row — 503 until the first heartbeat upsert, 503 again from SIGTERM — and the edge's follows its listeners. Neither probe touches Postgres, so a relay whose database is down keeps serving every tunnel it already holds. The ops listener now binds when its address is set explicitly or a toggle is on, so a manifest that sets PIPER_RELAY_OPS_ADDR=:9090 for the kubelet gets probes with no extra flag.
• Edge drains on SIGTERM (#534). Readiness flips, listeners close, live connections are carried up to 20 s, then exit cuts the rest. Compose edge gets stop_grace_period: 30s. Raw :7000 forwards never end on their own and always take the full grace — intended; HTTP through them works meanwhile.
• Startup schema apply is serialized under a Postgres advisory lock (#555). Several relays starting at once against an empty database used to race in CREATE ... IF NOT EXISTS and the loser died on a catalog duplicate key — 4 of 5 runs in the new concurrent-Open test before the lock, 0 after. This is the failure a rolling deploy onto a fresh database would hit.
• PIPER_RELAY_DATA_DIR removed (#553) — the vestigial mkdir, the image VOLUME, and the systemd Environment= line. Nothing read it; readOnlyRootFilesystem now works. If you set it in your own env file you can drop it.
• Runbook gains a ### Kubernetes subsection with the manifest knobs (rolling strategy, probes, grace periods, runAsUser/fsGroup/sysctl for non-root, Secret defaultMode: 0400, zone placement, PgBouncer caveat), and "What still drops" is updated for the edge drain.

Reconnect behaviour

• A half-open held session is replaced instead of refusing the redial (#538). After an unclean agent stop, both relays kept the dead sessions until yamux's keepalive reaped them (~30–40 s) and every redial in that window was refused as duplicate session. A post-auth duplicate is the same agent by proof, so the relay now pings the held session first: a live session still refuses the dial, a session that does not answer within 5 s is closed and unregistered so the redial lands.
• The slot-1 boot race is cheap (#548). Slot 1 dials the instant slot 0's handshake returns, but the relay acks before it writes the owner row, so the edge could place slot 1 on the relay that already holds slot 0 — and the agent then slept the full 1-minute duplicate backoff with only one session. The duplicate wait is now a ladder: 2 s, ×4, capped at the existing minute. A lost race costs 2 s; a genuine one-relay pool still settles at the cap. Slot 0 keeps its constant 5 s. Caught by the new cluster e2e, and confirmed against the production relays — one boot lost the race by 16 ms.
• A bad preface gets a named rejection (#543). An agent with an empty PIPER_BASE_DOMAIN saw awaiting relay handshake ack: EOF and redialled forever; it now gets the ordinary undifferentiated rejection and can say so. Only a preface that arrived whole but is unusable is answered — a silent client still times out with nothing written, and the edge path stays a silent drop.

Scale

• One owner re-assert per beat (#540). reassertOwnership ran one SetOwner per held base every heartbeat; it is now a single statement for the whole set, still announcing only on insert.
• A hostname NOTIFY re-derives only the agents it concerns (#540). Every piper_hostnames payload names one exact hostname or custom domain, so the fan-out is resolved to the holders that payload actually touches rather than swept across every held base. A sync-apps of M apps costs M small syncs per relay instead of M×N.

Testing

The relay e2e suite now runs the real topology (#537): piper-edge on the public ports in front of two piper-relay processes in two zones sharing one database, with PROXY protocol on, and piperd dialing the edge without ever learning a relay's address. New TestRelayCluster covers two owners in two zones → SIGTERM the relay holding slot 0 → visitor still served through the survivor → restart → back to two owners. Building the four binaries once per test process took TestRelayLoopback from 50 s to 1.4 s.

Known issues shipped open

• The relay and edge images still run as root (#554). A :nonroot variant was built and dropped in review: the compose layout bind-mounts certbot's tree and the GitHub App key as root 0600/0700, which uid 65532 cannot read, so the hosted relays would crash-loop on the next bump. On Kubernetes, use the runAsUser/fsGroup recipe now in the runbook.
• The slot-1 race is made cheap, not closed. Ordering the relay's ack after SetOwner would fix it at the source; noted in #548 as a possible follow-up.
• One behaviour change worth recording: before #540, any hostname NOTIFY anywhere in the cluster incidentally re-ran CustomDomains for every held agent, sweeping expired pending custom-domain claims out of every router. An expired-but-uncontested claim now stays routed until a NOTIFY names that agent or the listener reconnects. That matches the documented lazy expiry, and the contested case still propagates.