v1.10.2

Completes the 1.10.0 release. The 1.10.0 and 1.10.1 tags failed the release
vulnerability gate on newly published CVEs, so not all images shipped under
them; use 1.10.2 for all four images. Includes everything listed under 1.10.0.

Security

• Alpine-based images (edge, node-agent, WireGuard) apply base package updates
  at build time and use a current Alpine 3.24 digest, picking up the zlib fix
  for CVE-2026-85091.
• The edge (Caddy) binary is built with Go 1.27.3 and go-jose 4.1.6
  (CVE-2026-111943, CVE-2026-116587).
• golang.org/x/net 0.62.0 in the panel, edge and Terraform provider
  (CVE-2026-78661).

Included from 1.10.0

Feature and bug-fix release driven by user reports. Adds REST API control over
upstream scheme, WAF and rate limiting, a per-route access log endpoint and a
configurable display timezone, and fixes host creation and the DNS check for
API-imported services and DDNS nodes. Rebuilt on Go 1.27.3 with patched
dependencies; updating is recommended. No migrations, no manual steps.

Added

• Route API: upstream_scheme on create/patch/get (#118); per-route WAF and
  rate-limit fields (module-gated like the UI) and
  GET /api/v1/routes/{id}/access-logs (#117).
• HPG_DISPLAY_TZ (IANA zone, default UTC) shows access-log, WAF-event and
  alert timestamps in that zone; storage stays UTC (#116).

Fixed

• Adding a host for a backend IP shared by several API-imported services picks
  the service whose port range contains the port instead of failing with "port
  not in allowed range" (#120).
• The host DNS check resolves a node's public address when it is a hostname or
  DDNS name instead of always reporting a mismatch (#119).
• Host edit keeps the node name and status in the page header after a
  validation error instead of showing them empty.

Security

• Images built with Go 1.27.3 (CVE-2026-111943, crypto/x509) and go-jose 4.1.6
  (CVE-2026-116587).

v1.9.0

Security and reliability release from a full internal code review. Upgrade
promptly, especially on shared, reseller or multi-node installs: many fixes
tighten tenant scoping, authentication and node isolation. Several defaults
now fail closed (GeoIP policies, WAF on nodes without Coraza, the plain-HTTP
installer, scoped admins without a resolvable scope). Read the upgrade notes
before updating: migrations 00166-00171 run on start, a new node-agent image
is needed, and a few API and Terraform behaviours change.

Added

• New clients can be invited by email instead of setting a password (needs SMTP).
• Issued mTLS client certificates have copy buttons and a .pem bundle download.
• Stream listen ports can be changed on edit.
• HPG_DRILL_DSN points the restore drill at a pre-created empty
  database, for MySQL users that cannot CREATE DATABASE (shared hosting).
• /api/v1/health reports the real build version.
• Terraform hpg_reseller.owner_password_wo (write-only, Terraform 1.11+)
  keeps the reseller owner password out of plan and state;
  owner_password is deprecated.
• POST /api/v1/resellers accepts the initial status, reseller_plan_id,
  overselling_allowed and can_create_plans, applied in the create
  transaction. The Terraform provider sends them on create.
• Terraform hpg_client.password_wo (write-only, Terraform 1.11+) keeps the
  client password out of plan and state; password is deprecated.
• Webhook POSTs carry X-HPG-Delivery: <id>, stable across retries, so
  receivers can drop duplicates.

Documentation

• Community health files: CODE_OF_CONDUCT.md, SUPPORT.md, CODEOWNERS, release-note
  categories; SECURITY.md now names the private advisory channel, supported
  versions and response targets. Issue forms add needs-triage.
• AI_USAGE.md states how AI coding assistants are used and how their output is
  checked.

Security

• Tunnel install tokens, webhook signing secrets and issued mTLS keys are shown
  once and never placed in redirect URLs.

• mTLS path rules match the decoded, cleaned request path; the most specific
  rule wins over rule order; unreadable rules block the request.

• Restricted and reseller admins fail closed when their scope can't be
  resolved (lists, exports, world map); only super_admin creates global access
  groups; CSV exports neutralise formula cells.

• Password login carries a CSRF token signed with a key derived from
  APP_SECRET.

• The install wizard loads only over TLS or from loopback (SSH port-forward);
  INSTALL_ALLOW_PLAINTEXT=1 opts out.

• Tunnel and node bootstrap tokens are no longer read from URLs or passed on
  the command line; /api/wg/* query strings are redacted from stored access
  logs and WAF events.

• Routes with a country policy deny on a node without GeoIP, and blocking-mode
  WAF routes answer 503 on a node without Coraza, instead of serving
  unfiltered. Only the affected route changes; waf.fail_open and the
  per-route geo setting are the explicit opt-outs.

• An undecryptable CAPTCHA secret refuses verification instead of switching
  CAPTCHA off; Telegram errors no longer log the bot token.

• The Terraform provider requires an https endpoint outside loopback, and
  omitted route/plan flags keep the secure server defaults. API route create
  without ssl/force_https now enables both, as documented.

• The provider release workflow pins actions by SHA with least privilege.

• Backup docs use a read-only hpg_backup user with an option file instead
  of the root password on the command line.

• Caddy node image 2.11.4 -> 2.11.7 (builder on Go 1.27.2) with
  golang.org/x/net forced to v0.60.0 via xcaddy --replace (Caddy resolves
  v0.59.0): CVE-2026-78667, CVE-2026-78669, CVE-2026-97031 (HIGH, DoS). All
  Caddy pins (compose, node-join, docs) moved together.

• API keys with an empty scope list no longer get full access; they now pass
  no scope check, and the admin key form rejects a key with no scope selected
  (#53).

• Customer-issued API keys (client:read/client:write) can reach the
  customer endpoints of API v1; admin access is not widened (#59).

• A slave panel is read-only on every surface, not only /admin; register and
  password reset work on the master only (#52).

• Runtime 2FA enforcement fails closed when the policy can't be read (503 until
  the database answers) and the admin grace window is issued once (#55, #56).

• Public auth POSTs stay rate limited for logged-in sessions, and passkey login
  challenges are throttled (#57, #58).

• Malformed IP allow/trust lists deny instead of allowing everyone; bad entries
  are logged as "invalid CIDR in IP allow/trust list" (#54).

• Auto-failover reserves capacity per route so a single healthy node is not
  overloaded, and plan/reseller domain quotas are checked inside the create
  transaction (#49, #50).

• Revoked mTLS client certificates are refused on the nodes, mTLS roles are
  bound to one certificate (serial + fingerprint), and deleting a CA pushes the
  change to every node that trusted it.

• The forward-auth portal enforces every enrolled second factor (TOTP, SMS,
  email, recovery code). Portal OAuth finishes on the host it started from and
  is bound to the browser.

• SMS/email codes, recovery codes, passkey challenges and TOTP steps are each
  accepted once, also under concurrent requests.

• OIDC issuers must use https (no localhost exemption).

• Reseller and restricted admins see only their own clients on the dashboard;
  resellers can place hosts and streams only on node pools their package
  grants; DNS steering is super_admin-only and verified domains only.

• Reseller quotas (clients, services, domains) hold under concurrent creates.

• Node decommission keeps the configured Caddy admin bind and clears the node
  before deleting it ("Force decommission" for an offline node).

• Customer WireGuard peers can only answer connections, not open them to the
  node, its LAN or the internet (needs the new node-agent image).

• L4 streams accept inbound PROXY headers only from listed balancer CIDRs.

• WebSocket requests no longer skip the WAF.

• The backend test pins the address it screened and refuses managed nodes and
  the control plane.

• The restore drill replays only an artifact whose SHA-256 matches its backup
  record and only statements our own dumps emit.

• Secret rotation aborts on any unreadable row, can be resumed after a crash
  at any step, refuses --apply without database access and takes the DB
  password from DB_PASSWORD or --db-pass-file.

• SQLite installs enforce foreign keys; WAL and busy_timeout now apply.

• Auth helper cookies are cleared with HttpOnly/SameSite/Secure; node-token
  mismatches log a hash instead of a token prefix; hpg-restore rejects
  non-local archive paths; form integers are bounded to 32 bits; remaining
  Dockerfile base images are pinned by digest; chi and klauspost/compress
  bumped.

• The OpenAPI spec no longer advertises ?node_token= (it was never accepted).

Fixed

• Wildcard DNS-01 certificates are actually ordered now: the *.zone cert is
  requested as soon as the config loads, and a route can take the alias
  *.<zone> to serve every subdomain on that one cert (no per-subdomain
  certificates, no TXT proof for the wildcard alias; platform admins only, on
  a route without a path prefix). Access-log and WAF events for those
  subdomains are attributed to the route. The SSL tab lists which
  DNS-01 requirement is missing, and "Renew certificate now" re-pushes the
  full node config.
• A Cloudflare token in a format the Caddy DNS module rejects (quotes,
  spaces, a Bearer prefix) is refused when saved, and a stored one is
  skipped instead of failing the node's whole config load.
• Host edit shows alias validation errors (invalid or already-used alias)
  instead of re-rendering the submitted value as if it was saved, and opens
  the tab that holds the error.
• The central node-agent now sees the GeoIP database, so country rules filter
  traffic instead of failing closed on the panel node.
• Saving a host after adding a redirect, rewrite or block location works again.
• Editing a stream keeps all its upstreams.
• Creating a webhook without a secret generates one instead of failing.
• The AI assistant's route detail lookup works and includes WAF, rate limit,
  geo and mTLS settings.
• Ops summary counts and chart hours (shown in browser local time).
• Refreshing the page after issuing an mTLS certificate no longer issues another.
• Error toasts stay until dismissed; success shows one notice, not two.
• mTLS CA form picks the client from a list; add-host form uses full width;
  client detail lists every custom field.
• Alert cooldown now applies to rules with changing measurements (stale WG
  peer, error rate, DB pool, WAF surge); they no longer fire every minute.
• Browsers no longer autofill the admin login into webhook, DNS provider,
  backup and other secret fields.
• Dashboard and Stats request counts match and survive node counter resets;
  the 24h chart no longer piles all traffic into the last hour.
• Row action menus stay inside the viewport; the Legacy aliases page renders;
  Settings sections follow the URL hash.
• Placeholders no longer look like saved values; the command palette is
  readable in dark mode; backup destination form shows only the fields for
  the chosen kind and the job list refreshes while a job runs.
• Editing a stream keeps its CIDR lists; creating a stream with extra
  upstreams is all-or-nothing; reissuing a tunnel keeps its key-rotation age;
  a failed PSK clear keeps the rekey token usable.
• Backup schedule saves are atomic; world map totals include traffic beyond
  the first 5000 IPs.
• Access-log ingest is all-or-nothing per batch (503 on a storage error, the
  agent resends), raw rows and rollups are written together, and the agent
  ships ~1 MiB batches. Prometheus samples with timestamps parse correctly;
  live log un...

v1.8.0

Closes the remaining findings of the September system review and the 1.7.1
audit, and folds in the security fixes merged after 1.7.1 that were never
tagged as 1.7.2. Upgrade promptly, especially on shared or multi-tenant
installs: several fixes harden tenant-controlled settings on the node. Also
new: ephemeral external routes over API v1, per-node applied-config state,
a durable push outbox, and a per-node GeoIP gate so one node without the
database no longer rejects its whole config. Read the upgrade notes before
updating; migrations 00159-00164 run on start.

Added

• Per-node publish state on the hosts list. The publish pill used to show
  only the last compile result, so with several nodes it could read
  published while a node had refused its /load. Each node now records the
  hash and time of its last accepted /load and its last push error, and the
  pill reports what the serving nodes actually hold: published 3/3,
  pending 2/3, push failed 2/3 (hover for the node and error). Migration
  00163 adds four nullable caddy_nodes columns. See
  ROUTES.md.

• Ephemeral external routes over API v1. POST /api/v1/routes accepts
  upstream_url (admin keys, allowlisted https origin, verified TLS, Host/SNI
  set to the origin), strip_path_prefix and ttl_seconds; expired routes
  stop being emitted at once and are deleted by the leader within ~30 s. The
  external allowlist accepts *.zone entries; a wildcard-only match must
  resolve to public addresses and is dialed at the address the panel
  screened (re-checked on every push). strip_path_prefix matches only whole
  path segments (/tok and /tok/..., never /tokX). Built for proxying provider web consoles
  (noVNC/KVM) under your own domain. Migration 00161 adds three nullable
  routes columns. See API.md.

• Caddy admin endpoint can run on a unix socket. A node's admin endpoint
  can be bound to a filesystem socket instead of a TCP port
  (HPG_CADDY_ADMIN_LISTEN=unix//sockets/caddy-admin.sock|0666), so it has no
  network address at all. The panel reaches it over a shared volume, a remote
  node through its node-agent, which prefers the socket whenever it is live and
  falls back to HPG_CADDY_ADMIN_URL otherwise. Opt-in per node with
  HPG_CADDY_ADMIN_LISTEN_NODES; an upgrade that changes nothing else keeps
  the current TCP bind. server doctor now prints each node's actual admin
  bind, read back from the node, and hpg-node-agent doctor reports which
  transport it is using - check both before removing a published port. Rollout
  order and rollback:
  MULTI_NODE.md.

• A node's Admin API URL is editable in /admin/nodes → node → edit.
  Previously it could only be set when the node was registered, so a node could
  not be repointed at a different admin endpoint without re-adding it.

• Routes show what the compiler actually did with them. The hosts list now
  carries a publish state of its own - published, quarantined, rejected, target
  rejected, not emitted - next to the DNS/SSL health pill. A route can be
  "active" and still not be served, and that contradiction is now visible
  instead of only reaching the audit log. Rows that have not been compiled
  since the upgrade read as unchecked, never as published.

• Portal-protected routes have an explicit public-path list
  (portal_public_paths on the route, editable on the host's portal tab).

Security

• Third-party images in the deploy/ compose files (mariadb, redis, busybox,
  caddy, coredns, whoami, integration-test images) are pinned by multi-arch
  digest (tag@sha256:...), so a moved tag cannot change what you pull.
• Node registration (admin UI, POST /api/v1/nodes, install wizard) refuses a
  single-label API host such as http://node2:2019, other than the bundled
  Caddy (caddy, localhost, the CADDY_ADMIN_URL host). Such names were
  treated as bridge-local and skipped the unauthenticated-admin guard.
• Upstream proxy targets are checked to be plain host:port
  (caddyapi.ScreenDialTarget): Caddy also accepts socket and
  file-descriptor upstream forms, which no address-based screen covers.
• Route owners could put Caddy placeholders into strings the node expands.
  Several tenant-editable fields reach parts of the node config that Caddy
  resolves at request time, which reached further than the intended text. Those
  fields now accept only request placeholders; everything else is refused when
  saved, and a route whose stored value is already unsafe is quarantined
  (503) rather than published. Shared nodes should be treated as having had
  their local secrets exposed and rotated.
• The access portal authorized a hostname, not a route. One host with
  several routes could be verified against the wrong route's policy. The
  verifier now resolves the exact route the same way the node does, from the
  panel's own gate config rather than anything the browser sends, and refuses
  when the route is ambiguous.
• Portal protection with nobody granted access served the route publicly.
  Removing the last grant widened access instead of narrowing it. Protection
  intent alone now denies; the gate is never dropped because the grant set is
  empty. The old implicit public bypass for static-looking paths is gone -
  see the upgrade notes.
• Disabling a user did not end their portal session. Portal sessions were a
  separate lifecycle that never saw the account's revocation epoch. They now
  share it and re-check that the account is active on every verification.
• A read error while compiling could publish a weaker config. Loading extra
  Basic Auth users and per-path mTLS rules treated a failed read as "no policy".
  Required policy reads now fail the build, so the node keeps its last good
  config instead of receiving one with the protection missing.
• Forward-auth changed the real request's method to GET. An authenticated
  POST, PUT or DELETE could reach the backend as a GET. The rewrite now happens
  inside the verifier subrequest, where it belongs.
• Passwordless passkeys did not require user verification although the
  session was recorded as MFA.
• An unauthenticated caller who knew an API key prefix could force expensive
  key-derivation work before any rate limit applied. The stored HMAC is now
  the verdict, and the legacy compatibility path has its own budget.
• Suspending a service did not reliably stop serving it, and one entry point
  deleted its routes outright. Suspend, resume and terminate now have a single
  implementation that every entry point calls - panel, bulk action, REST API and
  the FOSSBilling integration - in one transaction. Route rows are never
  deleted by a suspend, and the compiler independently refuses to emit a route
  whose service is not active.
• Updating a route through the API skipped the limits enforced when creating
  one. Update now validates the whole new state against the service's current
  plan and allocation, the same check the create path uses.
• Custom error pages: the background colour was spliced into an inline style
  block unescaped, and the logo link accepted any scheme.
• A route could make its node resolve a backend name through a resolver the
  route owner controlled. Panel-side screening was skipped whenever a tunnel,
  a bound peer or a custom DNS resolver was in play, which left the destination
  decided somewhere the panel could not check. The panel now resolves and pins
  those names itself; a tunnel-bound backend dials the peer directly. Choosing
  node-side resolution is still possible, but it is now an explicit super_admin
  decision per route rather than a side effect, and the resolver fields
  themselves require super_admin to set.
• The SSO provider URL was never checked. It is a route-owner-supplied
  destination that the node dials, including on an unauthenticated passthrough.
  It now goes through the same destination screening as every other backend, at
  save time and again before publishing.
• The node admin transport now defaults to authenticated. A key that cannot
  be decrypted no longer falls back to an unauthenticated connection - the push
  fails instead. Node registration no longer accepts an unauthenticated endpoint
  disguised by a remapped port or a hostname. Existing keyless nodes are
  grandfathered per node and clear themselves once a key works; see the upgrade
  notes.
• A single bad WAF directive took the whole node down. One route's rules
  could make the node reject its entire configuration, for every tenant on it.
  A route whose directives the node refuses is now quarantined on its own and
  the rest of the node still publishes. Arbitrary rule directives are now
  restricted to super_admin, are no longer copied by cloning a host, and are
  re-checked whenever a route is re-enabled. SecRemoteRules and
  SecRuleScript are refused outright.
• The geo-block redirect URL was validated as a URL but not against the
  placeholder policy, so a customer could hold back all of their own routes
  without seeing an error.
• Critical: tenant-controlled page text could read data from the node.
  Block, maintenance and error page text and redirect targets set by clients
  and scoped admins were not covered by the screening added in 1.6.0. They are
  now checked on save and neutralised again when the configuration is built.
  Mechanism details are withheld until operators have had time to upgrade;
  multi-tenant deployments should treat this as a priority upgrade.
• High: a second login factor the user never enrolled was accepted. 2FA now
  completes only with a method enabled on that account.
• High: social-login auto-provisioning defaulted to a role that can read
  every tenant. Accounts created by Google/GitHub/OIDC sign-in now...

v1.7.1 - the node-side resolve switch is super_admin only

Security

• The per-route "backend is resolved on the node" switch was available to
  scope-restricted accounts. That switch waives panel-side resolution, and
  with it the screen that checks where a backend actually points - so an
  account limited to its own routes could turn its own screening off. It now
  requires super_admin, the same bar as skipping upstream TLS verification.
  Existing routes are unaffected: tunnel-bound routes were marked
  automatically and keep working, and nothing already saved is re-evaluated.

v1.7.0 - review follow-up and the unix-socket admin transport

Follow-up to the 1.6.0 remediation: an independent review of that work found
three paths it left open, and this closes them, plus the architectural issue
underneath the whole class - the node's administrative endpoint being
reachable from the component that serves untrusted traffic. Mechanism details
are withheld until operators have had time to upgrade.

Security

• Backend addresses are now fixed when the configuration is built.
  Previously a backend hostname was checked when the route was saved and
  resolved again later by the node, so the address that was approved and the
  address actually used could differ. The panel now emits the address it
  screened and keeps presenting the original hostname for TLS verification.
• Every existing SSO route moves to strict mode. 1.6.0 made strict the
  default for new routes only; routes created before it kept a mode that did
  not cover write requests. Permissive remains selectable per route as a
  deliberate, audited choice.
• A backend name the panel cannot resolve is no longer accepted silently.
  It has to be either resolvable or explicitly marked as resolved by the node,
  per route. Existing tunnel-bound routes are marked automatically so nothing
  goes down on upgrade.
• Upstreams that name no address are refused. Only ordinary host:port
  targets are emitted.

Added

• The node administrative endpoint can run on a unix socket instead of a
  TCP port, opt-in per node with HPG_CADDY_ADMIN_LISTEN_NODES. It stops
  being reachable from anything that shares a network with the node, and from
  most of what could otherwise address it. Both server doctor and the node
  agent report which transport is actually in use, so the old path is only
  removed once the new one is proven. Rollout and rollback order are in
  docs/MULTI_NODE.md.
• A node's Admin API URL is now editable after registration - it used to be
  write-once, which left no way to move an existing node.
• server doctor lists every SSO route still in permissive mode, so the
  routes this release changes can be seen before upgrading.

Upgrade notes - read before deploying

1. Run server doctor with the new binary before starting it. It lists
   every SSO route the migration will change. Applications behind those routes
   that relied on unauthenticated write requests will start receiving 401.
2. DNS round-robin across a single backend name no longer spreads traffic -
   one address is chosen and used. Configure several upstreams instead.
   A changed backend address is picked up on the next push: immediately on a
   route change, otherwise within the five-minute drift sweep.
3. Saving a route whose backend name the panel cannot resolve now fails until
   either DNS is fixed or the per-route "resolved by the node" switch is set.
4. The unix socket mode is opt-in. A deployment that upgrades and changes
   nothing behaves exactly as before.

v1.6.0 - system review remediation

Remediation release from an external security review of the whole system.
Fourteen findings closed: one critical, three high, three medium, six low and
one architectural note. Details of the critical and high findings are withheld
until operators have had time to upgrade; if you run a multi-tenant
deployment, treat this as a priority upgrade.

Security

• Critical: a tenant could cross into the control plane. Reseller and
  scope-restricted accounts could influence where a node sends proxied
  traffic in a way that reached infrastructure they should never touch.
  Impact on an affected node is disclosure or replacement of its
  configuration. All tenant-supplied proxy targets are now screened
  fail-closed, at save time and again when the node configuration is built.
  Customer private ranges (RFC1918, CGNAT) remain valid origins.

• High: remote nodes exposed an administrative interface on the mesh. The
  authenticated agent admin proxy is now required for remote nodes and the
  port is bound to loopback. Registering a node without authentication is
  refused. See the upgrade notes - this one needs a rolling migration.

• High: external SSO did not cover every request. Protection depended in
  part on a value the client itself supplies, and did not apply uniformly
  across request methods. New routes are created in strict mode; existing
  routes keep their current setting and should be moved to strict.

• High: tenant-supplied values could be expanded by the proxy before being
  sent to an origin the tenant controls. All such values are now screened on
  save and again at build.

• Medium: a client could influence the source address the panel recorded
  and rate-limited on. The panel route now stamps the address the proxy
  actually saw; the middleware prefers the verified forwarded chain and trusts
  other headers only where an operator opts in with
  APP_TRUST_REALIP_HEADERS.

• Medium: one tenant's web-application-firewall rule could block
  configuration changes for every other tenant on the same node. Scoped
  admins are now limited to removing rules by numeric id; unparseable entries
  are dropped when the configuration is built.

• Low: the portal logout could be triggered cross-site. Logout is now a
  POST with a token bound to the portal session.

• Secret files are created 0600. install_state.json kept whatever mode it
  had when restored or synced, commonly 0644.

Added

• server healthcheck subcommand, and Compose healthchecks for the app and
  Caddy - the runtime image is distroless, so there is no shell for a curl
  probe. Start periods are sized for a long migration or certificate issuance.
• server doctor reports per-node admin API authentication, secret file modes
  and the process umask, and names every missing or empty required environment
  variable with a generated value to use.
• make pin-version V=x.y.z rewrites every image pin, and CI fails if the
  deploy profiles or the README disagree on the version.

Fixed

• The remote-node bootstrap could not start. It bound the admin endpoint
  to an address that does not exist inside a bridge container, so the Caddy
  configuration failed to load and the node never came up.
• The lite and Portainer profiles still deployed 1.3.2. A fresh install
  from either got code six months old, with none of the 1.4 or 1.5 fixes.
• The remote GeoIP profile could not work: the agent wrote the database into
  its own container and remote Caddy ran a stock image with no geolocation
  module. Agent and Caddy now share a read-only volume and the profile uses
  the custom edge image.
• make run claimed to load .env and did not; the config never parsed the
  file either.

Documentation

• README and internal/domain/routes/placement.go claimed Redis-backed shared
  certificate storage. The module was never built into the image and the
  Caddyfile line was commented out, so each node has always issued its own
  certificates. The claim is removed and per-node issuance documented,
  including what to budget for ACME rate limits on failover. Building the
  shared store was considered and rejected: it would put private keys in Redis
  and make every node depend on it.
• docs/SECURITY.md said rotating APP_SECRET needs no downtime, while
  cmd/rotate-secret requires the panel stopped and invalidates every HMAC API
  key. Replaced with a runbook that includes re-issuing the keys.
• docs/ARCHITECTURE.md now states the real recovery window for deferred work
  after a crash (up to about five minutes) instead of implying durability.

Upgrade notes - read before deploying

1. A route whose backend points at a managed node's own address, the panel, or
   the control mesh is now refused when the node configuration is pushed, with
   a route.blocked_target audit entry. Most plausible on single-box installs.
   The audit row is the diagnostic; the UI does not surface it.
2. Additional upstreams and path-rule upstreams must now resolve panel-side
   unless the route is tunnel-bound.
3. Editing a route whose header, rewrite or redirect contains a proxy
   placeholder expression blocks the save until the value is removed. At build
   such values are dropped, so the route keeps serving without that header.
4. Scoped admins can no longer save free-form firewall rules; stored values
   keep working until edited.
5. Closing the node admin interface is a four-step rolling migration: enable
   the tunnel and copy the proxy key, start the agent with it, repoint the
   node URL to the proxy port and confirm with server doctor, and only then
   move the Caddy mapping to loopback. Nothing breaks if you do nothing;
   HPG_ALLOW_UNAUTHENTICATED_NODE_ADMIN=1 keeps registration open during the
   migration.
6. Operators running large active_active groups should check their ACME
   rate-limit headroom against nodes x hosts - certificates have always been
   issued per node, the documentation just said otherwise.

v1.5.1 - mTLS fail-open and path RBAC

Three mTLS defects, all older than 1.5.0, found while bringing the
documentation in line with 1.5.0. Mechanism details are withheld until
operators have had time to upgrade. If you use client-certificate
authentication, upgrade before anything else.

Security

• Client-certificate "fail open" did not verify a presented certificate.
  The setting was meant to let a client with no certificate through while
  still validating one that is presented; it validated nothing. Because the
  certificate subject is passed onward, an affected route could treat a
  client as an identity it had not proven. Fail-open now requires a presented
  certificate to chain to the configured CA. The setting is super_admin
  break-glass and off by default.

• A client could supply the header that carries the certificate subject.
  It is now removed from every inbound request before anything else runs.

Fixed

• Path-based mTLS role checks denied every request, including legitimate
  ones. The stored subject and the value the proxy sends were compared in
  two different formats, so the lookup never matched and every rule-covered
  path returned 403. Both forms now match, so subjects already stored keep
  working. The existing test passed only because its fixture used a value the
  issuing path cannot produce.

v1.5.0 - post-quantum readiness

Post-quantum readiness (PQ-01). The threat is harvest-now-decrypt-later:
traffic recorded today and decrypted once a quantum computer exists. TLS
already negotiated hybrid ML-KEM by default on every hop; WireGuard did not,
because the protocol has no hybrid mode. Both WireGuard planes now mix a
preshared key into every handshake, a host can be pinned to post-quantum-only
TLS, and regressions are caught in CI and at release. Full detail, verification
commands and rollout order in docs/POST_QUANTUM.md.

Security

• An mTLS-enforced host was still reachable in the clear on port 80.
  Client-certificate enforcement lives in the TLS connection policy, so it
  applies only once a request arrives over TLS; the plaintext listener served
  the same host with no check at all. force_https was a separate stored flag
  that nothing set when enforcement was turned on. The route builder now
  derives it from require_client_cert, which also covers every row already in
  the database, and strict_sni_host is pinned explicitly so a request cannot
  reach the host under a different SNI. Migration 00145 backfills
  force_https = 1 on existing enforced routes.

• A host could require client certificates while serving without TLS. The
  client-authentication policy is part of the route's TLS connection policy,
  so with SSL off nothing is emitted: the host answered every request with a
  permanent 503 under the default fail-closed mode, or served everyone with no
  client-cert check at all under fail-open - while the panel listed it as
  locked. All three ways into that state are now refused: creating a host
  (checked after the plan gate settles SSL, so a plan without SSL cannot force
  it), saving the host edit page (checked after the external-upstream branch,
  the last thing that can turn SSL back on), and PATCH /api/v1/routes/{id}
  clearing ssl_enabled on an enforced route. The add form and the edit page
  now state the dependency, and a host already in this state is shown as
  "mTLS saved - SSL off, not enforced" instead of "mTLS enforced".

• "Require a client certificate" was silently discarded when adding a host.
  The create handler validated require_client_cert and the chosen CA, then
  never wrote either column, and the add-host form had no such inputs - so the
  check guarded a value that was thrown away. An operator who expected a new
  host to demand client certificates got one that accepted anyone, with no
  error and no warning. The add form now offers mTLS whenever an active CA
  exists, the flag and its trust anchor are persisted in the same INSERT as the
  route (so the host is enforced on its very first config push, never briefly
  open), and a create that asks for enforcement without a usable CA is refused
  instead of quietly downgraded. Existing hosts are unaffected.

• Enabling mTLS on one host broke the TLS handshake for every other host on
  the same node. Caddy only supplies a default connection policy when the
  policy list is nil; with any per-SNI policy present, an unmatched
  ClientHello gets no server TLS configuration available and the handshake
  dies. One mTLS route therefore took down plain HTTPS for every other site on
  that node - and TLS-ALPN-01 renewals with it, so certificates quietly stopped
  renewing. The emitted policy list now always ends with a catch-all {}.
  Verified against a real Caddy 2.11.4 both before and after. No configuration
  change is needed; the fix lands on the next config push.

• Rotating APP_SECRET orphaned two node secrets. cmd/rotate-secret was
  missing caddy_nodes.tunnel_privkey_e2 and caddy_nodes.admin_proxy_key_enc
  from its column list, so a rotation left the tunnel server key and the
  agent's admin-proxy key sealed under the old key - bricking WireGuard tunnels
  and authenticated config push on the next restart (security review CRYPTO-01,
  round two). Both are now rotated, along with the new preshared-key columns,
  and a column whose migration has not run yet is skipped with a log line
  instead of failing the run, so a newer image can rotate an older database.

Added

• wstunnel bumped 10.5.5 -> 11.0.0 on both pins. The WSS tunnel binary is
  pinned twice - in the node-agent image and in the customer install script -
  and a bump that lands on one side only leaves half the tunnel behind, so a
  CI gate now fails when the two version or sha256 pins disagree. Upstream
  calls v11 non-breaking; verified locally with a real WireGuard handshake and
  traffic over WSS through Caddy, in all four client/server version
  combinations, plus --restrict-to still rejecting a non-WireGuard
  destination.

• Preshared keys on the panel-node WireGuard mesh. The mesh carries Caddy
  config pushes, manual-certificate private keys included, so a recorded
  handshake is worth storing until X25519 falls. A node joining now gets a
  32-byte PresharedKey automatically, gated on the join script declaring
  support - a node bootstrapped from an older cached node-join.sh joins
  without one and keeps working. For nodes that joined earlier, the node detail
  page stages a key and prints a one-time command
  (curl -fsSL https://<panel>/install/node-psk.sh | sudo bash -s -- ...); the
  current key stays live until the node fetches, applies and confirms the new
  one, so the two sides are never more than one handshake apart. Confirm is
  idempotent for the token's 30-minute lifetime, so a lost response is retried
  rather than rolled back into a split mesh, and the staged key is cleared only
  once the panel has written its own WireGuard config - a repeat confirm can
  therefore never report success for a promotion that is still able to roll
  back. wg0.conf is replica-local and each replica feeds its own WireGuard
  sidecar, so every replica re-renders it from the database every 60 s, inside
  WireGuard's 120 s rekey window; that is what makes a rekey confirmed against
  one replica of a multi-replica panel safe. Clear PSK is the emergency exit
  and needs a matching edit on the node - and if the panel's own config cannot
  be rewritten, the clear is rolled back and reported as a failure instead of
  printing that node-side command, which on its own would take the mesh down.
  Note the timing: WireGuard holds an established session until the next
  handshake, so a mismatch takes up to ~120 s to appear, stops config pushes to
  that node, and does not interrupt traffic the node is already serving.

• Preshared keys on customer tunnels. Per-peer keys on wg-tun0, gated on
  both sides: a peer only gets one when every node in its group reports
  support, and node-agent declares its support on the peer pull itself, so
  the panel knows what it is talking to before it decides what to hand over. An
  agent that declares nothing while the node still has PSK-bearing peers is
  refused with 409: it would apply the peer set without the keys and drop
  every one of those tunnels on the next wg syncconf, whereas the config it
  is already running keeps them up. A node with no PSK-bearing peers is served
  as before, so a downgraded fleet is not bricked. The capability is also
  re-asserted on every stats report and an absent field means "not supported",
  so rolling an agent back clears the flag; the 1 -> 0 edge writes a
  node.psk_capability_lost audit entry with the number of affected peers,
  whose configs then have to be re-downloaded. Key rotation degrades instead of
  failing: if a group node lags, the rotation proceeds without a PSK, clears
  psk_enc so the node's pull and the rendered config still agree, and audits
  wg_peer.psk_dropped. Existing peers stay classical until their next
  rotation or config re-download.

• Post-quantum-only TLS per host. Opt-in on the host's SSL & Protocols tab;
  pins that hostname to curves=[x25519mlkem768] and TLS 1.3, so a client
  without hybrid ML-KEM fails the handshake instead of quietly negotiating
  classical X25519. Connection policies are merged per SNI, not per route,
  and the strictest wins - a TLS policy is matched before the request path is
  known, so enabling this (or mTLS) on one path enforces it for every route on
  that hostname, and two routes on one hostname with different mTLS CAs keep
  the first by route id and log the conflict. The policy is emitted only when
  SSL is on for the host and the node declares Caddy 2.10 or newer
  (caddy_nodes.caddy_version, operator-entered, not probed): an older Caddy
  rejects the entire /load on the unknown curve name. That gate is per node,
  so the host page checks every node serving the host - the anchor plus the
  route_node_assignments fan-out - and names the one holding it back instead
  of judging by the anchor alone. Saving a host pushes to that same set, and
  saving a node's capabilities (the Caddy version included) pushes that node:
  connection policies are rebuilt per node, so anything less left fan-out peers
  on the previous policy while the panel called PQ-only - or mTLS - enforced,
  and route-level drift detection cannot see a node-level difference. When
  either gate blocks the policy, the host page says so and the push logs it.
  Breaks old clients on purpose - Chrome < 131, Firefox < 132, Safari/iOS < 26,
  curl without OpenSSL 3.5+. TLS-ALPN-01 is disabled for such a host rather
  than left to fail silently at renewal time, so its certificate must be
  issuable over HTTP-01 (port 80 reachable) or DNS-01.

• Post-quantum readiness card in Settings. Read-only checklist of where
  hybrid protection is actually in effect: panel runtime, per-node Caddy
  versions, mesh PSK coverage, customer-tunnel PSK coverage, PQ-only hosts, and
  what is still classical upstream. PQ-only hosts are counted as distinct
  SSL-enabled domains: several path routes can share one hostname, and a route
  with SSL off gets no TLS connection policy at all, s...

v1.4.9

Two bugs behind one report (#14): the edge image could not stream a response
through the WAF, and suppressing a rule in the panel never reached the nodes.
Plus a CVE bump in the app image.

Fixed

• SSE and other streaming responses stalled behind the WAF (#14). The edge image now builds with coraza-caddy 2.6.1. Its 2.5.0 interceptor propagated Flush() with a bare http.Flusher type assertion, which fails on the recorder Caddy wraps around the response writer whenever access logging is on - and the panel enables access logs on every node. Every flush was dropped, so an EventSource behind a WAF-enabled host never received its headers, in detection-only mode as much as in blocking mode. Upstream fix: corazawaf/coraza-caddy#344. Reproduced and verified with the same pins as the edge image and a Beszel hub. Requires the v1.4.9 edge image on every node; no config change.
• WAF suppressions now reach the nodes (#14). Suppressing a rule only hid its events; in blocking mode the node kept returning 403. Active suppressions (global and per-route) are now emitted as SecRuleRemoveById after the CRS include on every WAF-enabled route, and saving or deleting one re-pushes all nodes. Rule IDs are validated (NNN or NNN-MMM) before they reach SecLang. The report itself was a CRS 942100 false positive on PocketBase/Beszel realtime subscription topics (systems/*), not an SSE transport problem; documented in docs/WAF.md.
• Spurious 503 on idempotent requests under load. The reservation INSERT that backs Idempotency-Key had a 300 ms deadline; a busy database or a runner mid-fsync missed it, and the fail-closed branch then rejected a request that would have succeeded. The deadline is now 2 s (still bounded) and the swallowed error is logged, so the next occurrence is diagnosable instead of a bare 503.

Security

• golang.org/x/crypto 0.55.0 in the app image, fixing CVE-2026-56854 (x/crypto/ssh source-address restrictions not enforced), flagged CRITICAL by the release Trivy gate. The rest of the x/ set moves with it.
• google.golang.org/grpc 1.83.2 in the Terraform provider (GHSA-2v4p-qf9q-27wj, xDS server DoS); indirect via terraform-plugin-go.

v1.4.8

Stabilization pass over the findings from an external review: the tenant
boundaries on the node-facing endpoints, the two write races in the control
plane, and the release process that could publish an untested image.

Added

• Remaining stock Caddy load-balancing policies (#13). The host LB selector was missing five policies stock Caddy already supports: first (send everything to the first healthy upstream, i.e. active/passive failover), random, random_choose (power-of-two-choices, sample size fixed at 2), client_ip_hash, and query (hash a query parameter). query reuses the existing header-name input - the field is relabelled when the policy is selected. No new Caddy modules are required, so no availability gate.

Security

• A node token could poison another node's - or another tenant's - access
  logs. /internal/access-log authenticated the per-node agent token and
  then resolved the route globally by domain (SELECT id FROM routes WHERE domain = ? LIMIT 1). A stolen or compromised node token could therefore
  write request history onto routes served by other nodes and other customers,
  and a domain with several path-routed routes had its lines attributed to
  whichever row LIMIT 1 happened to return. Authentication now yields the
  node id, and every batch is attributed through an index of the routes that
  node actually serves - anchor placement plus route_node_assignments fan-out
  peers, primary domain plus proven aliases, longest matching path_prefix
  first. A line for a host the node does not serve is dropped.
• mTLS path RBAC trusted the caller's header behind an IP allow-list.
  /internal/mtls-rbac/{route_id} took the client-certificate subject from
  X-Mtls-Subject and gated the caller with an allow-list spanning the whole
  WireGuard mesh plus every configured trusted proxy, so any host that reached
  that network layer could query the RBAC oracle for arbitrary routes and
  subjects. The panel now writes HMAC-SHA256(HKDF(APP_SECRET, "mtls-rbac"), node||route) into exactly one node's Caddy config; the check verifies it in
  constant time and re-reads placement from the database, so a token minted for
  another route - or for a node the route has since moved off - is refused.
  MTLS_RBAC_ALLOW_UNSIGNED=1 keeps token-less checks working during the
  upgrade window (the boot push stamps the token in immediately) and warns on
  every request it lets through.
• Purpose-scoped encryption was separated in name only. Scoped(purpose)
  derived a per-purpose sub-key, but Decrypt took the purpose from inside the
  ciphertext envelope, so a value sealed for one domain decrypted cleanly for a
  consumer of another. A scoped manager now requires the envelope's purpose to
  match its own; legacy pre-v2 ciphertext stays readable, and route secrets -
  sealed scoped but read unscoped - now decrypt through their own scope.
• A node name could inject WireGuard directives. caddy_nodes.name is
  rendered into wg0.conf as # Node #<id> (<name>); a name containing CR/LF
  ended the comment and could append arbitrary directives, including a [Peer]
  block with AllowedIPs 0.0.0.0/0. Names are validated on every write path
  (admin form, v1 API, install wizard, join-token name hint) and sanitized
  again where they are rendered, so a row written before this cannot inject
  either. A join token whose stored hint fails validation falls back to the
  derived node-<wg-ip> name.
• Argon2 parameters were taken from the stored hash unchecked. A corrupt
  row or an imported hash could ask one login attempt to allocate gigabytes,
  and t=0/p=0 panics inside argon2.IDKey. Parameters, salt length and
  hash length are now range-checked, and a malformed hash is reported as such
  rather than as a password mismatch.
• SESSION_COOKIE_SECURE_STRICT (new) keeps Secure on for an HTTPS-only
  deployment. Secure is still dropped for requests that do not look like
  HTTPS - that is what keeps first-run access over http://<ip> working - but
  a TLS proxy that forgets X-Forwarded-Proto was silently disabling it for
  every request. The downgrade now logs once per process.
• The NPM import upload was unbounded. ParseMultipartForm's argument caps
  only what is held in memory; the rest spills to a temp file. The request body
  is capped before parsing and the spill file is removed with the request.

Fixed

• An older config snapshot could overwrite a newer one. A full push built
  the node's config before taking the per-node lock, so two writers could
  interleave as: A builds generation 10, B mutates and pushes generation 11, A
  finally gets the lock and /loads its generation-10 snapshot - silently
  reverting B. The build now happens under the lock, and each node carries a
  config generation: a change that lands while a push is in flight triggers an
  immediate rebuild instead of leaving the node on the snapshot just applied.
  (The counters are per process; serializing pushes across several app
  replicas still needs the generation in the database.)
• A node could be pushed past max_routes. Placement read
  current_routes < max_routes and incremented the counter later, so
  concurrent creates handed out the same last free slot. The increment is now a
  conditional UPDATE inside the insert transaction; single-mode placement
  re-places onto another node with room, a fan-out peer that filled up is
  skipped rather than oversubscribed, and callers get a retryable error. A test
  runs 40 concurrent creates against a node capped at 10 and asserts exactly 10
  land - it counted 13 before the fix.
• About one API key in eight was dead on arrival. VerifyAPIKey split the
  token on _ to separate the 8-character prefix from the secret, but the
  prefix is base64url and can contain _. Those keys were rejected from the
  moment they were issued, with the same error as a forged token. Parsing is
  now positional, matching what the rate-limit middleware already did.
• WAF events from active-active peers lost their route. The WAF route index
  selected only routes anchored on the ingesting node, so a peer serving the
  route through route_node_assignments produced events with a NULL
  route_id, dropping them out of the per-route view, scoped-admin visibility
  and per-route pruning. Verified aliases had the same gap.
• Idempotency reservations failed open. Any INSERT error was read as "the
  key already exists", and a failed follow-up lookup then ran the handler
  unprotected - executing the very mutation the caller asked to be deduped, and
  again on each retry. Only a unique-key violation now means an existing entry;
  anything else answers 503 without running the handler, and an expired
  colliding row is reclaimed with a conditional update so exactly one racing
  request owns the key.
• Rate-limit counters could stick forever. INCR and EXPIRE were two
  commands; a counter incremented while the EXPIRE was lost kept its value
  and rate limited that IP or API key permanently. Both limiters now use one
  atomic script that sets the TTL on first use.

Added

• A node's Caddy admin API can now be authenticated. Caddy's admin API has
  no authentication of its own, so publishing it on the node's WireGuard
  address made "can route to <wg-ip>:2019" equivalent to root on that node -
  every tenant on it. The node-agent can now front it: the panel presents a
  per-node key (issued by Enable tunnel / Rotate, stored encrypted in
  caddy_nodes.admin_proxy_key_enc, migration 00141, shown once with the
  other agent credentials), the agent compares it in constant time and enforces
  a method+path allow-list covering exactly what the control plane does - so
  even a leaked key cannot POST /stop, /adapt, or rewrite the node's own
  admin config. The agent refuses to start on a wildcard bind or a key under 32
  characters rather than serving something that only looks protected.
  Opt-in per node (HPG_ADMIN_PROXY_LISTEN + HPG_ADMIN_PROXY_KEY); a node
  that has not been migrated is reached directly, exactly as before. Migration
  steps in docs/MULTI_NODE.md; server doctor probes with the key so a
  migrated node does not read as unreachable.
• NPM import: dry run and a real report. Preview runs every check (SSRF
  screening of forward hosts, duplicate domains, node availability) and writes
  nothing, so an import can be inspected first. Redirection hosts now import as
  redirect routes. The response is a per-entry report - imported / skipped /
  manual, each with a reason - and streams, access lists, non-ACME
  certificates, 404 hosts, location rules, advanced_config, NPM caching,
  block-common-exploits, HSTS and preserve_path are reported as manual with
  the nearest panel equivalent instead of being dropped silently.

Changed

• routes/service.go (~3.4k lines) is split by concern into service.go,
  lifecycle.go, build.go, streams.go, push.go, health.go and
  util.go - same package, pure code motion, every declaration body identical.

CI / release

• Release now gates every image build on the CI workflow, so nothing publishes
  from a ref that failed vet, race tests, the migrations-layout check or the
  image scan. latest tracks release tags only; a push to main publishes
  edge. Each per-arch build attaches an SBOM and max-mode provenance, and the
  merged manifest is signed keylessly with cosign. CI builds the app image and
  fails on fixable HIGH/CRITICAL findings. Still open: pinning actions to
  commit SHAs, and branch protection on main (a repository setting).
• New tests for the paths a release should not ship without: the N-1 -> N
  migration applied to live data, a backup restored into an empty installation
  (quotes, NULLs, binary values, indexes), edge failover movin...