v0.23.1 — relay rollout readiness and faster reconnects

Relay and edge hardening on top of the scale-out work in v0.20.0–v0.23.0. Everything in this release is relay/edge-side.

No upgrade action required. No schema change in either store, no wire-protocol change, no re-enrollment. Existing databases, tokens and enrollments carry over untouched, and relays and agents can be rolled in either order. The usual relay-before-agents ordering is still the safer habit, but nothing here depends on it.

Rolling relays

• Kubernetes rollout readiness (#557, #552, #553, #555, #556). /readyz and /livez on both ops listeners: the relay's readiness follows its pool row — 503 until the first heartbeat upsert, 503 again from SIGTERM — and the edge's follows its listeners. Neither probe touches Postgres, so a relay whose database is down keeps serving every tunnel it already holds. The ops listener now binds when its address is set explicitly or a toggle is on, so a manifest that sets PIPER_RELAY_OPS_ADDR=:9090 for the kubelet gets probes with no extra flag.
• Edge drains on SIGTERM (#534). Readiness flips, listeners close, live connections are carried up to 20 s, then exit cuts the rest. Compose edge gets stop_grace_period: 30s. Raw :7000 forwards never end on their own and always take the full grace — intended; HTTP through them works meanwhile.
• Startup schema apply is serialized under a Postgres advisory lock (#555). Several relays starting at once against an empty database used to race in CREATE ... IF NOT EXISTS and the loser died on a catalog duplicate key — 4 of 5 runs in the new concurrent-Open test before the lock, 0 after. This is the failure a rolling deploy onto a fresh database would hit.
• PIPER_RELAY_DATA_DIR removed (#553) — the vestigial mkdir, the image VOLUME, and the systemd Environment= line. Nothing read it; readOnlyRootFilesystem now works. If you set it in your own env file you can drop it.
• Runbook gains a ### Kubernetes subsection with the manifest knobs (rolling strategy, probes, grace periods, runAsUser/fsGroup/sysctl for non-root, Secret defaultMode: 0400, zone placement, PgBouncer caveat), and "What still drops" is updated for the edge drain.

Reconnect behaviour

• A half-open held session is replaced instead of refusing the redial (#538). After an unclean agent stop, both relays kept the dead sessions until yamux's keepalive reaped them (~30–40 s) and every redial in that window was refused as duplicate session. A post-auth duplicate is the same agent by proof, so the relay now pings the held session first: a live session still refuses the dial, a session that does not answer within 5 s is closed and unregistered so the redial lands.
• The slot-1 boot race is cheap (#548). Slot 1 dials the instant slot 0's handshake returns, but the relay acks before it writes the owner row, so the edge could place slot 1 on the relay that already holds slot 0 — and the agent then slept the full 1-minute duplicate backoff with only one session. The duplicate wait is now a ladder: 2 s, ×4, capped at the existing minute. A lost race costs 2 s; a genuine one-relay pool still settles at the cap. Slot 0 keeps its constant 5 s. Caught by the new cluster e2e, and confirmed against the production relays — one boot lost the race by 16 ms.
• A bad preface gets a named rejection (#543). An agent with an empty PIPER_BASE_DOMAIN saw awaiting relay handshake ack: EOF and redialled forever; it now gets the ordinary undifferentiated rejection and can say so. Only a preface that arrived whole but is unusable is answered — a silent client still times out with nothing written, and the edge path stays a silent drop.

Scale

• One owner re-assert per beat (#540). reassertOwnership ran one SetOwner per held base every heartbeat; it is now a single statement for the whole set, still announcing only on insert.
• A hostname NOTIFY re-derives only the agents it concerns (#540). Every piper_hostnames payload names one exact hostname or custom domain, so the fan-out is resolved to the holders that payload actually touches rather than swept across every held base. A sync-apps of M apps costs M small syncs per relay instead of M×N.

Testing

The relay e2e suite now runs the real topology (#537): piper-edge on the public ports in front of two piper-relay processes in two zones sharing one database, with PROXY protocol on, and piperd dialing the edge without ever learning a relay's address. New TestRelayCluster covers two owners in two zones → SIGTERM the relay holding slot 0 → visitor still served through the survivor → restart → back to two owners. Building the four binaries once per test process took TestRelayLoopback from 50 s to 1.4 s.

Known issues shipped open

• The relay and edge images still run as root (#554). A :nonroot variant was built and dropped in review: the compose layout bind-mounts certbot's tree and the GitHub App key as root 0600/0700, which uid 65532 cannot read, so the hosted relays would crash-loop on the next bump. On Kubernetes, use the runAsUser/fsGroup recipe now in the runbook.
• The slot-1 race is made cheap, not closed. Ordering the relay's ack after SetOwner would fix it at the source; noted in #548 as a possible follow-up.
• One behaviour change worth recording: before #540, any hostname NOTIFY anywhere in the cluster incidentally re-ran CustomDomains for every held agent, sweeping expired pending custom-domain claims out of every router. An expired-but-uncontested claim now stays routed until a NOTIFY names that agent or the listener reconnects. That matches the documented lazy expiry, and the contested case still propagates.

v0.23.0 — zone-aware placement for the second tunnel session

Upgrade step (relay operators)

relay_instances gains a nullable zone column. The schema is applied with CREATE TABLE IF NOT EXISTS, which skips an existing table, so on a live database run one of these before the new relays and edge start, or every heartbeat fails on the missing column:

ALTER TABLE relay_instances ADD COLUMN zone TEXT;

or, as with the last two releases, drop the two pool tables and let the binaries recreate them:

DROP TABLE agent_owners, relay_instances;

Nothing on the wire changed. Agents need no update and no re-enrollment; roll relays and the edge in any order.

What's new

Zone-aware placement for the second tunnel session (#531, fifth child of the scale-out epic #524). A relay may now label its failure zone with the optional PIPER_RELAY_ZONE environment variable. When an agent already holds a session on a relay in zone X, piper-edge prefers a relay outside X for the second session, ahead of load, so a zone outage does not take both. Zone is a preference, never a filter: if every candidate is in X, placement falls back to fewest sessions as before.

• Unset means unknown and never counts as a clash. Pools that never set the variable are placed exactly as in v0.22.0.
• Set it on every relay or on none. A zoneless relay is always eligible as "a different zone", so a mixed pool can put both sessions in one physical zone without the edge knowing.
• Runbook: docs/runbooks/relay-deploy.md covers what to set and where Kubernetes (topology.kubernetes.io/zone node label) and ECS (task metadata endpoint) expose it. A single-host compose deployment leaves it unset.

Design: docs/superpowers/specs/2026-09-05-zone-aware-placement-design.md.

Known issues shipped open

• #538 — a relay keeps pinging a session the agent has moved elsewhere for up to ~10 s after a duplicate dial.
• #537 — the e2e harness does not yet run on the supported topology (edge in front of two relays).
• #535 — compose recreates relay replicas together; there is no rolling relay restart on a single host.

v0.22.0 — two tunnel sessions per agent

Two tunnel sessions per agent: a relay restart is no longer an outage

The fourth child of the scale-out epic (#524). Every agent now holds two tunnel sessions on two different relays, so a relay going away is never an agent's only link and the edge keeps routing through the survivor. This completes the zero-drop rollout story that #523's graceful drain started.

What's new

• Two sessions per agent (#530, PR #539) — piperd runs two dial loops. The second starts only after the first has connected, so the edge already knows which relay holds the agent and places the second dial on a different one. If a relay drains or dies, the other session keeps serving while the lost slot redials. Not a knob: a supported deployment has at least two relays behind piper-edge.
• Handshake split into a routing preface and a credential frame. The agent sends its base domain first, then its token. piper-edge peeks only the preface on :7000 and places the dial on the relay with the fewest sessions among those that do not already own that agent, falling back to the whole pool when every relay does (so a dial with a made-up base learns nothing). The relay refuses a second session for a base it already holds, but only after the token checked out, so a bad token gets the same undifferentiated rejection as before.
• Ownership is a set. agent_owners is keyed on (agent_name, instance_id). The edge picks among an agent's live owners (non-draining first, fewest sessions, earliest started) for :443/:80 and retries once on the next owner if a dial fails. Relays re-record every base they hold on each heartbeat, so a stale row never outlives its relay.
• Every relay serves every route. Hostnames and custom domains are derived from Postgres when a session registers and re-derived on piper_hostnames NOTIFY, so both of an agent's relays route the same hostnames regardless of which session the agent registered them over.
• piper status names a duplicate refusal. A slot refused as a duplicate reports the reason instead of a blank "retrying".

Upgrading

This release changes the relay schema in place and the agent↔relay wire protocol (pre-1.x, no migrations). Agents on v0.21.0 or older cannot connect to a v0.22.0 relay, so roll the relay pool first and upgrade agents the same day.

1. Drop the two pool tables before starting the new binaries — mandatory:

   DROP TABLE agent_owners, relay_instances;

   Both are ephemeral (heartbeats and ownership; refilled within seconds). agent_owners changed its primary key, and both piper-relay and piper-edge apply the same schema.sql with CREATE TABLE IF NOT EXISTS, so whichever new process starts first re-creates them. Unlike v0.21.0, skipping the drop fails silently: the old table keeps one row per agent, the edge never sees an agent's second session, and the pool runs single-owner.
2. Relays and edge together, then agents. Old relays cannot read the new preface and an old edge forwards it blind, so bring both up on v0.22.0 in one step; the pool is unreachable to agents until the agents follow:

   docker compose up -d --no-deps --scale relay=2 relay
   docker compose up -d --no-deps edge

   Compose recreates both relay replicas together (#535), so every agent's sessions drop once here anyway.
3. Upgrade every agent. Each piperd logs slot 0: connected and slot 1: connected; agent_owners shows two rows per agent once both are up. A one-relay pool logs slot 1: every relay the edge can offer already holds <base> once and retries every minute: the deployment is mis-sized and both sides say so.

Verified on the hosted relay on 2026-09-05.

Runbook: docs/runbooks/relay-deploy.md — "How it routes", "At least two relays", "What still drops", "Upgrade across a schema change".

Known limits

• After an unclean agent stop (crash, power loss) both relays keep the dead sessions until the yamux keepalive reaps them (about 30–40 s); the first slot retries every 5 s in that window, so the box's apps are unreachable for about 40 s. A clean stop or upgrade reconnects at once. #538 tracks a relay-side ping that would cut the window to about 10 s.
• With in-place replacement (no surge), the second step of a rolling relay restart finds each agent on one session for up to a minute. Kubernetes' and ECS' surge defaults avoid this.
• piper-edge itself still stops abruptly on SIGTERM (#534). Zone-aware placement of the second session is #531; the e2e harness still runs one relay (#537).

Spec: docs/superpowers/specs/2026-09-05-two-sessions-per-agent-design.md. Prior art: docs/superpowers/specs/2026-09-05-tunnel-placement-prior-art.md.

v0.21.0 — login state in Postgres, graceful relay drain

Relay scale-out, round two: login state in Postgres, graceful drain, honest edge metrics

Three children of the scale-out epic (#524) land together. api.<apex> is no longer pinned to one relay, a relay restart no longer yanks its tunnels mid-request, and the edge's connection counters now add up.

What's new

• Login-flow state in Postgres (#522, PR #529) — dashboard web login, brokered CLI login, the GitHub device flow and the login rate limit each live in a shared table (login_web_states, login_cli_handles, login_device_flows, login_rate). Any relay can answer any step, so piper-edge now round-robins api.<apex> across live relays in start order. The device flow has no background poller any more: the CLI poll makes the upstream call, so a relay dying mid-login strands nothing. The rate limit is one budget across the pool (30/min per client), not one per process. No secret is at rest in a login table. Wire shapes of /v1/login/* are unchanged.
• Graceful drain on SIGTERM (#523, PR #533) — a stopping relay marks its pool row draining, which the edge reads from the heartbeat: no new tunnel placement and no api.<apex> traffic go there, while hostnames it owns keep routing to it. It refuses new :7000 connections (the listener stays open so the edge does not evict it), closes each agent session the moment it has no open streams, force-closes the rest after 20 s, then leaves the pool and parks in-flight webhooks for the new owner. Worst case 60 s from signal to exit. Agents are untouched; they still see one reconnect. Zero-drop rollouts come with #530 (two sessions per agent).
• Edge metrics decompose (#526, PR #528) — new piper_edge_conns_dropped_total{listener} counts connections that close without a usable SNI or Host (background scanners), and a dial to a dead owner on :443/:80 now counts as unrouted. accepted = routed + unrouted + dropped holds on every listener.

Upgrading

This release changes the relay schema in place (pre-1.x, no migrations). Agents need nothing; the tunnel wire protocol is unchanged.

1. Drop the two pool tables before starting the new binaries — mandatory:

   DROP TABLE agent_owners, relay_instances;

   Both are ephemeral (heartbeats and ownership; refilled within seconds). relay_instances gained a draining column, and both piper-relay and piper-edge apply the same schema.sql with CREATE TABLE IF NOT EXISTS, so whichever new process starts first re-creates them. Skipping the drop breaks every new relay's heartbeat. The four login_* tables are new and need nothing.
2. Roll relays first, edge last. An upgraded edge round-robins api.<apex>; a relay still on v0.20.0 holds login state in memory, so logins would fail until every relay is on v0.21.0. Stagger relay restarts on this first roll so two relays don't race the same CREATE TABLE.
3. Give relays a 60 s stop timeout. Compose: stop_grace_period: 60s on the relay service (the repo's deploy/compose/relay/docker-compose.yml has it). ECS stopTimeout, Kubernetes terminationGracePeriodSeconds. systemd's default 90 s already covers it.
4. Relays, then the edge. Compose recreates both relay replicas together (COMPOSE_PARALLEL_LIMIT=1 does not serialise them, see #535), so every agent redials once:

   docker compose up -d --no-deps --scale relay=2 relay
   docker compose up -d --no-deps edge

Expect this first roll to drop tunnels once anyway: the relays being stopped are v0.20.0 and do not drain. From here on a relay restart drains. Verified on the hosted relay on 2026-09-05.

Runbook: docs/runbooks/relay-deploy.md — "Upgrade across a schema change", "Scale out" and "Single host with compose".

Known limits

• A drained relay's agents still reconnect once (~1 s backoff). Redundant sessions for zero-drop rollouts are #530; zone-aware placement is #531.
• piper-edge itself stops abruptly on SIGTERM; readiness flip and splice grace are #534.
• Only one SIGTERM is honoured; a second during the drain is ignored (the orchestrator's SIGKILL at the stop timeout still applies).

Specs: docs/superpowers/specs/2026-09-04-login-state-postgres-design.md, docs/superpowers/specs/2026-09-05-relay-graceful-drain-design.md, prior art docs/superpowers/specs/2026-09-05-tunnel-placement-prior-art.md.

piper v0.20.0

piper-edge: scale the relay out behind one address

piper-relay can now run as N processes behind a single public address. A new binary, piper-edge (ghcr.io/piperbox/piper-edge), is a stateless L4 entrypoint that routes every connection to the relay that owns the agent's tunnel (#521, epic #524).

What's new

• piper-edge — peeks SNI on :443 and Host on :80, resolves the owning relay from an in-memory copy of the pool, and splices bytes to it with a PROXY v2 header so the relay still sees the real client IP. :7000 goes to the least-loaded relay; api.<apex> pins to the earliest-started relay (until #522). A relay whose dial fails is evicted (one retry on :7000 only). If Postgres is unreachable the edge keeps serving from memory.
• Relay pool membership — each piper-relay heartbeats into relay_instances (5s; live for 15s) with the addresses it advertises (PIPER_RELAY_ADVERTISE_HOST, default first non-loopback IPv4) and leaves the pool on SIGTERM.
• Tunnel ownership — agent_owners records which relay terminates which agent. Register upserts, unregister clears only its own row and only when no newer session is held, and every heartbeat re-asserts ownership for held agents with no live owner.
• Cluster-aware control API — a piper command that lands on a relay not holding the agent is forwarded unchanged to the owner over :8080 (one hop, never two). GET /agents and the delete guard count live owners on any relay and fail closed on a lookup error.
• NOTIFY-woken webhooks — parked GitHub events wake the owning relay through piper_events instead of waiting for the sweep.
• Runbook: new Scale out section in docs/runbooks/relay-deploy.md with a compose example (edge on the host network, relays on the bridge with PIPER_RELAY_PROXY_PROTOCOL=1).

Upgrading

• Relay schema: two new tables (relay_instances, agent_owners) created by CREATE TABLE IF NOT EXISTS on start. Nothing existing changes; no re-enroll, no data migration. Agents and the wire protocol are untouched, so relay-before-agents ordering does not apply — old agents and this relay interoperate both ways.
• Single relay, no edge: everything works as before. Two behaviour changes: SIGTERM now always drains (leaves the pool, ≤5s), and a host with no non-loopback IPv4 must set PIPER_RELAY_ADVERTISE_HOST or the relay refuses to start.
• Behind an edge: set PIPER_RELAY_PROXY_PROTOCOL=1 on every relay, PIPER_RELAY_ADVERTISE_HOST to an address the edge can dial, and give the edge PIPER_EDGE_DB_URL + PIPER_EDGE_APEX. Postgres must be reachable from the edge.

Known limits

• api.<apex> (device flow, web login, rate limiter) is pinned to one relay until login-flow state moves to Postgres (#522).
• A relay restart still drops its tunnels; agents redial from 1s backoff. Graceful drain is #523.

v0.19.0 — relay on Postgres, direct serve mode

Promoted from v0.19.0-rc.1 as a same-commit re-tag: identical binaries, now validated in production — the hosted relay at public.getpiper.dev has been running this build on Postgres since 2026-09-03, with its data carried over by the copier (~2 s of downtime, every agent re-registered with its existing token).

The headline change is infrastructure: piper-relay stores its state in Postgres instead of a SQLite file.

Manual upgrade steps — read before upgrading

Relay operators (hosted or self-hosted)

1. Provision Postgres and set PIPER_RELAY_DB_URL. The relay refuses to start without it (PIPER_RELAY_DB_URL is required (postgres://user:password@host/dbname)). Tables are created on first start. PIPER_RELAY_DATA_DIR remains the mount point for the TLS pair and the GitHub App key; relay.db is no longer read or written. Full procedure: docs/runbooks/relay-deploy.md → 2. Configure.
2. Existing data: copy it with the one-off cmd/relay-sqlite-to-pg (go run ./cmd/relay-sqlite-to-pg -sqlite relay.db -pg "$PIPER_RELAY_DB_URL") before starting the new binary — tokens, accounts, GitHub App links, custom domains, and repo bindings survive; no re-enrollment. The copier is temporary and is removed after the hosted cutover (#516). Or start empty and let boxes re-login. Running relay and Postgres together on one public-IP host? The runbook's Single host with compose section is the exact procedure used for the hosted relay's cutover, including the host-networking, certbot-mount, and nightly-dump details.
3. systemd: the shipped piper-relay.service now carries After=postgresql.service and StartLimitIntervalSec=0, so a relay that boots before its database keeps retrying instead of tripping systemd's start-rate limit. Install the new unit file, not just the binary.
4. Relay before agents, as always. The agent↔relay wire protocol is unchanged, so boxes on v0.18.0 keep working against a v0.19.0 relay.

Agent (piperd) operators

domain_config gained a serve column (direct-serve mode, #508). Schema is applied with CREATE TABLE IF NOT EXISTS, so an existing piper.db does not pick it up on its own. Either apply it in place before restarting —

sqlite3 /var/lib/piper/piper.db "ALTER TABLE domain_config ADD COLUMN serve TEXT NOT NULL DEFAULT 'relay';"

— or start with a fresh DB and redeploy. Boxes without a custom domain configured are unaffected either way.

What's new

• Relay on Postgres (#515, #517): one shared database for every relay process; SQLite removed from the relay. SQLite's global write lock is replaced by row-level locking on the enroll/app/domain cap checks and on the parked-webhook drain, each covered by a concurrency test against real Postgres. Foreign keys are now enforced. Fixes a latent tie-break in org-target GitHub App install routing that would have picked the wrong org under Postgres NULL ordering.
• Direct serve mode (#505/#508, #506/#511): a box-wide custom domain can be served by the box's own :443 (DNS → box), with the relay kept for login, webhooks, and control; per-app custom domains get DNS-01 certs through the same token source, no relay splice.
• Direct HTTPS on a never-enrolled box (#509).
• PROXY protocol v2 on the relay's public listeners, opt-in via PIPER_RELAY_PROXY_PROTOCOL=1 (#485/#500), so rate limiting and logs key on the real client IP behind a trusted L4 proxy.
• TUI: list and switch to relay boxes (#504).
• Relay logs failed terminate-path TLS handshakes, rate-gated (#501). Token and installation listings order by insertion, not lexical RFC3339Nano (#503). Tilde-free asset names for prerelease debs (#502). Repo hygiene: issue forms, PR template, SECURITY.md, CONTRIBUTING.md (#512, #514).

Still one replica

Postgres removes the database as a reason the relay can't scale out, but two per-process states remain — tunnel sessions and in-flight login flows — so run exactly one relay until those follow-ups land. The runbook's container section spells this out.

Known issues shipped open

• PIPER_RELAY_DATA_DIR is created but no longer read by the relay itself (certs and the App key are addressed by their own env vars); it stays as the documented mount point for now.
• cmd/relay-sqlite-to-pg ships in this release and will be deleted once the hosted relay has moved (#516).

v0.19.0-rc.1 — relay on Postgres, direct serve mode

Release candidate. The headline change is infrastructure: piper-relay now stores its state in Postgres instead of a SQLite file. This RC exists so the hosted relay can be cut over and validated on real hardware before v0.19.0 is promoted from the same commit.

Manual upgrade steps — read before upgrading

Relay operators (hosted or self-hosted)

1. Provision Postgres and set PIPER_RELAY_DB_URL. The relay refuses to start without it (PIPER_RELAY_DB_URL is required (postgres://user:password@host/dbname)). Tables are created on first start. PIPER_RELAY_DATA_DIR remains the mount point for the TLS pair and the GitHub App key; relay.db is no longer read or written. Full procedure: docs/runbooks/relay-deploy.md → 2. Configure.
2. Existing data: copy it with the one-off cmd/relay-sqlite-to-pg (go run ./cmd/relay-sqlite-to-pg -sqlite relay.db -pg "$PIPER_RELAY_DB_URL") before starting the new binary — tokens, accounts, GitHub App links, custom domains, and repo bindings survive; no re-enrollment. The copier is temporary and is removed after the hosted cutover (#516). Or start empty and let boxes re-login.
3. systemd: the shipped piper-relay.service now carries After=postgresql.service and StartLimitIntervalSec=0, so a relay that boots before its database keeps retrying instead of tripping systemd's start-rate limit. Install the new unit file, not just the binary.
4. Relay before agents, as always. The agent↔relay wire protocol is unchanged, so boxes on v0.18.0 keep working against a v0.19.0-rc.1 relay.

Agent (piperd) operators

domain_config gained a serve column (direct-serve mode, #508). Schema is applied with CREATE TABLE IF NOT EXISTS, so an existing piper.db does not pick it up on its own. Either apply it in place before restarting —

sqlite3 /var/lib/piper/piper.db "ALTER TABLE domain_config ADD COLUMN serve TEXT NOT NULL DEFAULT 'relay';"

— or start with a fresh DB and redeploy. Boxes without a custom domain configured are unaffected either way.

What's new

• Relay on Postgres (#515, #517): one shared database for every relay process; SQLite removed from the relay. SQLite's global write lock is replaced by row-level locking on the enroll/app/domain cap checks and on the parked-webhook drain, each covered by a concurrency test against real Postgres. Foreign keys are now enforced. Fixes a latent tie-break in org-target GitHub App install routing that would have picked the wrong org under Postgres NULL ordering.
• Direct serve mode (#505/#508, #506/#511): a box-wide custom domain can be served by the box's own :443 (DNS → box), with the relay kept for login, webhooks, and control; per-app custom domains get DNS-01 certs through the same token source, no relay splice.
• Direct HTTPS on a never-enrolled box (#509).
• PROXY protocol v2 on the relay's public listeners, opt-in via PIPER_RELAY_PROXY_PROTOCOL=1 (#485/#500), so rate limiting and logs key on the real client IP behind a trusted L4 proxy.
• TUI: list and switch to relay boxes (#504).
• Relay logs failed terminate-path TLS handshakes, rate-gated (#501). Token and installation listings order by insertion, not lexical RFC3339Nano (#503). Tilde-free asset names for prerelease debs (#502). Repo hygiene: issue forms, PR template, SECURITY.md, CONTRIBUTING.md (#512, #514).

Still one replica

Postgres removes the database as a reason the relay can't scale out, but two per-process states remain — tunnel sessions and in-flight login flows — so run exactly one relay until those follow-ups land. The runbook's container section spells this out.

Known issues shipped open

• PIPER_RELAY_DATA_DIR is created but no longer read by the relay itself (certs and the App key are addressed by their own env vars); it stays as the documented mount point for now.
• cmd/relay-sqlite-to-pg ships in this release and will be deleted once the hosted relay has moved (#516).

v0.18.0 — relay as a container image; honest login and upgrade advice

⚠ Upgrading requires retiring the agent database

app_env gains an updated_at column (#495). Piper has no migrations, and CREATE TABLE IF NOT EXISTS is a no-op against a table that already exists — so an upgraded box hits no such column: updated_at on the first env read or write.

Upgrade the relay first, then each agent. GET /v1/github/status now returns the account's username, which the new piper login reuse path reads.

The relay's schema is unchanged — a relay upgrades in place (install the binary, restart) and keeps its database. Only agents are affected.

On each agent box — option A: migrate in place (recommended)

The change is additive, so one ALTER TABLE plus a backfill gets the existing database to the shape the new code expects. Nothing is lost — apps, env vars, deployments and running containers all survive, and the only interruption is the daemon restart the binary upgrade needs anyway. Verified on a live box serving traffic.

sudo systemctl stop piperd
sudo cp -a /var/lib/piper/piper.db /var/lib/piper/piper.db.pre-0.18

DB=/var/lib/piper/piper.db
has() { sudo sqlite3 "$DB" "PRAGMA table_info($1);" | cut -d'|' -f2 | grep -qx "$2"; }
has app_env updated_at   || sudo sqlite3 "$DB" "ALTER TABLE app_env ADD COLUMN updated_at TEXT NOT NULL DEFAULT '';"
has deployments hostname || sudo sqlite3 "$DB" "ALTER TABLE deployments ADD COLUMN hostname TEXT NOT NULL DEFAULT '';"
sudo sqlite3 "$DB" "UPDATE app_env SET updated_at = COALESCE((SELECT a.created_at FROM apps a WHERE a.name=app_env.app), strftime('%Y-%m-%dT%H:%M:%SZ','now')) WHERE updated_at='';"

The second ALTER is only needed coming from v0.16.0 or earlier (deployments.hostname, #479); the guard skips it otherwise.

Three things that matter:

• The backfill is not optional. An empty updated_at fails time.Parse(RFC3339Nano) and the client treats that as fatal, so piper env ls would error on every migrated row. apps.created_at is already RFC3339Nano and is an honest floor — an env var cannot predate its app.
• Use the T…Z form. sqlite3's more natural datetime('now') yields 2026-08-04 09:15:22, which does not parse.
• Check file ownership afterwards. /var/lib/piper is DynamicUser state owned by nobody:nogroup; sqlite3 run as root creates -wal/-shm owned by root. A clean exit removes them, but a lingering root-owned WAL is exactly what would stop the daemon writing.

Verify with sudo sqlite3 -header -column "$DB" "SELECT app,key,updated_at FROM app_env;" — every row must carry a timestamp — then upgrade and start.

The migration is backward-compatible: the old binary's insert names only (app, key, value), so it keeps working against the migrated table and you cannot get stranded halfway.

Option B: retire the database

If you would rather not touch SQL, retiring only piper.db also works. The box keeps its identity, so there is no re-enrollment, no new box id, and no stale row on the relay — but you lose every app definition and env var, and you must clean up stranded containers by hand:

sudo systemctl stop piperd
sudo sh -c 'mkdir -p /var/lib/piper/pre-0.18 && mv /var/lib/piper/piper.db /var/lib/piper/pre-0.18/'
# install 0.18.0 (apt upgrade, or the installer), then:
piper login    # converges — prints "already enrolled as <box>"

box-id, relay.json, acme_account.key, appdomains/ and caddy/ stay put. domain_config re-derives itself from relay.json on first start. Verified on a Raspberry Pi 4 upgrading from 0.16.0.

What you do lose: app definitions and env vars, which lived in the discarded database. Recover them from the retired file before re-adding — it is a normal SQLite database and still readable:

sudo python3 -c "
import sqlite3
c=sqlite3.connect('file:/var/lib/piper/pre-0.18/piper.db?mode=ro',uri=True)
for r in c.execute('SELECT name,port,repo,branch,root_dir FROM apps ORDER BY name'): print(r)
for r in c.execute('SELECT app,key,value FROM app_env ORDER BY app,key'): print(r)
"

Stop your running containers before re-adding the apps. Discarding the database does not stop anything Docker is running, so the previous deployments survive as orphans the agent will never reconcile or retire — and they hold the host ports a fresh deploy of the same app wants. docker ps will show them.

The relay now ships as a container image

ghcr.io/piperbox/piper-relay:0.18.0 — one multi-arch image (linux/amd64 + linux/arm64), distroless, built from the release binary. This is the relay's packaging format, deliberately not apt or brew: its deployment targets are container platforms. Version tags publish on every tag including RCs; :latest moves only on finals. See the runbook's "Run as a container" section for volume/cert mounts, the single-replica-only constraint, and the L4-only fronting rule. (#486, closes #483)

piper login is idempotent and honest (#492)

• A second piper login no longer charges a browser round-trip. Both login paths try the stored credential first and settle it with one cheap authed request; only an expired, revoked, or disabled credential falls through to the full device flow. --relogin is the escape hatch for switching GitHub accounts. (#473)
• The apply-wait -race failure is fixed, the deadline message no longer tells you piperd did not come back about a daemon that answered every poll, and the enroll POST gets a budget that survives a Pi on a slow uplink. (#467)
• Device-flow hardening: a disabled account is refused at all three doors rather than handed an inert credential, and the in-flight flows map is now bounded by expiry instead of growing for the life of the process. (#81, which stays open for its remaining FakeVerifier ask)
• GitHub App installs reconcile: an App link that was left stranded now heals. (#470, #474)

piper agent status stops giving wrong upgrade advice (#499)

It compared the running daemon against whatever piperd resolved to in the caller's PATH, so a box with a current system service and a stale ~/.local/bin copy was told to restart into a downgrade. It now compares against the systemd unit's ExecStart, skips the comparison entirely when systemd won't report one, and prints restart piperd to apply only on positive evidence that the disk build is newer. Git-describe and dirty builds are never ordered. A shadowing copy on your PATH gets its own explicit warning. (Closes #472)

Fixes

• Relay reloads its wildcard TLS certificate without a restart. A renewed cert on disk is now picked up through a GetCertificate callback that re-stats and re-parses only on change; a failed reload keeps serving the last good pair. A serving the last good certificate line during a normal renewal is expected, not an incident. (#497, fixes #484)
• Custom-domain DNS target is the box's base domain, not the dial address. A box whose relay dial address isn't the relay's public name advertised a nonsense record and its issuance gate could never flip. The mirror-image bug is also closed: a domain resolving to loopback no longer "points at" a loopback dial address. Docs updated to match. (#493, #498, fixes #434)
• Monorepo pushes only rebuild the app they touch. A push that changes nothing under an app's root_dir is skipped. A push whose file list GitHub truncated still rebuilds — a needless build beats a swallowed change. PR previews still rebuild on any change. (#490, closes #319)
• Tunnel handshakes can no longer be pinned by a stalled peer, in either direction. Dial's handshake write is bounded (#457, fixes #436), Serve's success-ack write is bounded (#487, fixes #455), and both agent callers make the handshake context-interruptible so shutdown doesn't wait out the deadlines (#488, fixes #481).
• Relay heals a stranded target_login after a GitHub org rename, on the token-mint miss path only. (#464, fixes #433)
• installation_repositories webhooks link the installation, so re-saving an App's repository selection works; a failed link now names the installer instead of failing silently. (#489, fixes #471)
• piper env ls shows how long ago each variable was last set. (#495, fixes #442)
• Tests no longer pop a real browser tab at GitHub device login; PIPER_NO_BROWSER=1 suppresses the opener anywhere, which a headless box or SSH session wants anyway. (#491)
• Data race on the test log buffer in TestServeTunnelLogsRejectedHandshake. (#463, fixes #439)
• The installer one-liner is now curl -fsSL https://get.piperbox.dev/install.sh | sh. (#480)

Known issues shipped open

• #494 — token and installation lists sort RFC3339Nano timestamps lexically, so rows can come back mis-ordered.
• #485 — the relay does not speak PROXY protocol, so client IPs are lost behind an L4 ingress.
• #81 — FakeVerifier is neither one-shot nor pruned; it is reachable in production under PIPER_RELAY_FAKE_APPROVE=1 for the loopback e2e.
• If you installed the v0.18.0-rc.1 debs by hand, nfpm rendered that version as 0.18.0.rc.1, which outranks 0.18.0 under dpkg's comparison rules. apt upgrade will not move you to the final; use sudo apt install --allow-downgrades piperd=0.18.0 piper=0.18.0. The apt repo is unaffected — RCs never publish there.

v0.18.0-rc.1 — relay as a container image; honest login and upgrade advice

Release candidate. This tag exists to prove the new container-image publishing path against the real registry before v0.18.0 takes the latest pointer. It does not move latest, does not publish to apt, and does not update the Homebrew tap — install it explicitly.

⚠ Upgrade requires a fresh agent database

app_env gains an updated_at column (#495). Piper has no migrations, and CREATE TABLE IF NOT EXISTS is a no-op against a table that already exists — so an in-place upgrade hits no such column: updated_at on the first env read or write.

Every agent box must be given a fresh DB and re-enrolled. Apps and their env vars must be re-added afterwards. This is the pre-1.x policy (see CLAUDE.md); it is stated here because it is a real break, not assumed.

The relay's schema is unchanged — a relay keeps its database.

Order of operations: upgrade the relay before the agents. GET /v1/github/status now returns the account's username, which the new piper login reuse path reads.

The relay now ships as a container image

ghcr.io/piperbox/piper-relay:0.18.0-rc.1 — one multi-arch image (linux/amd64 + linux/arm64), distroless, built from the release binary. This is the relay's packaging format, deliberately not apt or brew: its deployment targets are container platforms. Version tags publish on every tag including RCs so an RC can be smoke-tested from the registry; :latest moves only on finals. See the runbook's "Run as a container" section for volume/cert mounts, the single-replica-only constraint, and the L4-only fronting rule. (#486, closes #483)

piper login is idempotent and honest (#492)

• A second piper login no longer charges a browser round-trip. Both login paths try the stored credential first and settle it with one cheap authed request; only an expired, revoked, or disabled credential falls through to the full device flow. --relogin is the escape hatch for switching GitHub accounts. (#473)
• The apply-wait -race failure is fixed, the deadline message no longer tells you piperd did not come back about a daemon that answered every poll, and the enroll POST gets a budget that survives a Pi on a slow uplink. (#467)
• Device-flow hardening: a disabled account is refused at all three doors rather than handed an inert credential, and the in-flight flows map is now bounded by expiry instead of growing for the life of the process. (#81, which stays open for its remaining FakeVerifier ask)
• GitHub App installs reconcile: an App link that was left stranded now heals. (#470, #474)

piper agent status stops giving wrong upgrade advice (#499)

It compared the running daemon against whatever piperd resolved to in the caller's PATH, so a box with a current system service and a stale ~/.local/bin copy was told to restart into a downgrade. It now compares against the systemd unit's ExecStart, skips the comparison entirely when systemd won't report one, and prints restart piperd to apply only on positive evidence that the disk build is newer. Git-describe and dirty builds are never ordered. A shadowing copy on your PATH gets its own explicit warning. (Closes #472)

Fixes

• Relay reloads its wildcard TLS certificate without a restart. A renewed cert on disk is now picked up through a GetCertificate callback that re-stats and re-parses only on change; a failed reload keeps serving the last good pair. A serving the last good certificate line during a normal renewal is expected, not an incident. (#497, fixes #484)
• Custom-domain DNS target is the box's base domain, not the dial address. A box whose relay dial address isn't the relay's public name advertised a nonsense record and its issuance gate could never flip. The mirror-image bug is also closed: a domain resolving to loopback no longer "points at" a loopback dial address. Docs updated to match. (#493, #498, fixes #434)
• Monorepo pushes only rebuild the app they touch. A push that changes nothing under an app's root_dir is skipped. A push whose file list GitHub truncated still rebuilds — a needless build beats a swallowed change. PR previews still rebuild on any change. (#490, closes #319)
• Tunnel handshakes can no longer be pinned by a stalled peer, in either direction. Dial's handshake write is bounded (#457, fixes #436), Serve's success-ack write is bounded (#487, fixes #455), and both agent callers make the handshake context-interruptible so shutdown doesn't wait out the deadlines (#488, fixes #481).
• Relay heals a stranded target_login after a GitHub org rename, on the token-mint miss path only. (#464, fixes #433)
• installation_repositories webhooks link the installation, so re-saving an App's repository selection works; a failed link now names the installer instead of failing silently. (#489, fixes #471)
• piper env ls shows how long ago each variable was last set. (#495, fixes #442)
• Tests no longer pop a real browser tab at GitHub device login; PIPER_NO_BROWSER=1 suppresses the opener anywhere, which a headless box or SSH session wants anyway. (#491)
• Data race on the test log buffer in TestServeTunnelLogsRejectedHandshake. (#463, fixes #439)
• The installer one-liner is now curl -fsSL https://get.piperbox.dev/install.sh | sh. (#480)

Known issues shipped open

• #494 — token and installation lists sort RFC3339Nano timestamps lexically, so rows can come back mis-ordered.
• #485 — the relay does not speak PROXY protocol, so client IPs are lost behind an L4 ingress.
• #81 — FakeVerifier is neither one-shot nor pruned; it is reachable in production under PIPER_RELAY_FAKE_APPROVE=1 for the loopback e2e.

Installing this RC

curl -fsSL https://get.piperbox.dev/install.sh | PIPER_RC=1 sh

Without PIPER_RC=1 the installer resolves releases/latest, which skips prereleases by design.

v0.17.0 — PR preview URLs reach the API; agent status stops guessing

A PR preview's URL now reaches the control API, and piper agent status reports what the daemon actually loaded instead of guessing.

⚠️ Manual upgrade step — agent DB

deployments gained a hostname column. Pre-1.0 there are no migrations, and CREATE TABLE IF NOT EXISTS does not add columns to an existing table — so on an untouched DB, GET /v1/apps/{name}/deployments fails with no such column: hostname. That breaks piper deploy's progress follow, the TUI's deployment list, and the dashboard's app view. Deploys themselves keep working.

Add the column — one line, no data loss. Under the shipped systemd unit the DB is /var/lib/piper/piper.db; otherwise it is ~/.piper/piperd/piper.db, or wherever PIPER_DATA_DIR points. piper agent status prints the effective path (accurately, as of this release):

sudo systemctl stop piperd
sudo sqlite3 /var/lib/piper/piper.db "ALTER TABLE deployments ADD COLUMN hostname TEXT NOT NULL DEFAULT ''"
sudo systemctl start piperd

macOS/Homebrew: brew services stop piper, run the same sqlite3 line against ~/.piper/piperd/piper.db (no sudo), then brew services start piper. On Debian/Ubuntu/Pi install sqlite3 first if it is missing (sudo apt install sqlite3).

Existing deployment rows get an empty hostname — cosmetic, since previews re-record theirs on the next tunnel reconnect. Replacing the DB outright also works but costs you your apps, env vars, custom domains, tokens and enrollment, so prefer the ALTER.

Relays are unaffected: nothing in this release touches the relay schema or the agent↔relay wire protocol, so there is no relay-before-agents ordering to respect.

Upgrade as usual: brew upgrade piper && brew services restart piper, or sudo apt update && sudo apt install piperd piper.

What's new

• A preview's URL is now in the API (#479). Preview hostnames are keyed (agent, app, pr) and apps.hostname holds production's alone, so a preview's URL previously existed only on the PR's GitHub deployment — a client could tell a deployment was a preview but had nothing to link to. It is now persisted on the deployment row and served from GET /v1/apps/{name}/deployments as Hostname, and refreshed on tunnel reconnect so a relay-side rename can't leave a dead URL in the API. This is the preview counterpart of #137.
• piper agent status stops guessing (#477). Listen addresses and data dir came from /proc/<pid>/environ, which is root-only for a system piperd — so a non-root status printed the built-in defaults as fact and env-file overrides were invisible. It now prefers the daemon's own GET /v1/version self-report, falls back to /proc where readable, and marks the last resort (assumed — the agent did not report) rather than posing as truth.

Known issues shipped open

Carried over, none regressed by this release: piper agent status compares against the PATH-resolved binary rather than the unit's ExecStart (#472); piper login reruns the device flow even with a valid saved credential (#473); the relay learns App installations only from webhooks, so an already-installed App may need a suspend/unsuspend to re-link (#470, #471); relay-deploy runbook corrections (#474); prerelease-only deb checksum filename mismatches (#475).

Dashboard rendering of the new preview link is separate work in the dashboard repo and is not in this release — the data is available, the UI does not surface it yet.