v1.8.0

Closes the remaining findings of the September system review and the 1.7.1
audit, and folds in the security fixes merged after 1.7.1 that were never
tagged as 1.7.2. Upgrade promptly, especially on shared or multi-tenant
installs: several fixes harden tenant-controlled settings on the node. Also
new: ephemeral external routes over API v1, per-node applied-config state,
a durable push outbox, and a per-node GeoIP gate so one node without the
database no longer rejects its whole config. Read the upgrade notes before
updating; migrations 00159-00164 run on start.

Added

• Per-node publish state on the hosts list. The publish pill used to show
  only the last compile result, so with several nodes it could read
  published while a node had refused its /load. Each node now records the
  hash and time of its last accepted /load and its last push error, and the
  pill reports what the serving nodes actually hold: published 3/3,
  pending 2/3, push failed 2/3 (hover for the node and error). Migration
  00163 adds four nullable caddy_nodes columns. See
  ROUTES.md.

• Ephemeral external routes over API v1. POST /api/v1/routes accepts
  upstream_url (admin keys, allowlisted https origin, verified TLS, Host/SNI
  set to the origin), strip_path_prefix and ttl_seconds; expired routes
  stop being emitted at once and are deleted by the leader within ~30 s. The
  external allowlist accepts *.zone entries; a wildcard-only match must
  resolve to public addresses and is dialed at the address the panel
  screened (re-checked on every push). strip_path_prefix matches only whole
  path segments (/tok and /tok/..., never /tokX). Built for proxying provider web consoles
  (noVNC/KVM) under your own domain. Migration 00161 adds three nullable
  routes columns. See API.md.

• Caddy admin endpoint can run on a unix socket. A node's admin endpoint
  can be bound to a filesystem socket instead of a TCP port
  (HPG_CADDY_ADMIN_LISTEN=unix//sockets/caddy-admin.sock|0666), so it has no
  network address at all. The panel reaches it over a shared volume, a remote
  node through its node-agent, which prefers the socket whenever it is live and
  falls back to HPG_CADDY_ADMIN_URL otherwise. Opt-in per node with
  HPG_CADDY_ADMIN_LISTEN_NODES; an upgrade that changes nothing else keeps
  the current TCP bind. server doctor now prints each node's actual admin
  bind, read back from the node, and hpg-node-agent doctor reports which
  transport it is using - check both before removing a published port. Rollout
  order and rollback:
  MULTI_NODE.md.

• A node's Admin API URL is editable in /admin/nodes → node → edit.
  Previously it could only be set when the node was registered, so a node could
  not be repointed at a different admin endpoint without re-adding it.

• Routes show what the compiler actually did with them. The hosts list now
  carries a publish state of its own - published, quarantined, rejected, target
  rejected, not emitted - next to the DNS/SSL health pill. A route can be
  "active" and still not be served, and that contradiction is now visible
  instead of only reaching the audit log. Rows that have not been compiled
  since the upgrade read as unchecked, never as published.

• Portal-protected routes have an explicit public-path list
  (portal_public_paths on the route, editable on the host's portal tab).

Security

• Third-party images in the deploy/ compose files (mariadb, redis, busybox,
  caddy, coredns, whoami, integration-test images) are pinned by multi-arch
  digest (tag@sha256:...), so a moved tag cannot change what you pull.
• Node registration (admin UI, POST /api/v1/nodes, install wizard) refuses a
  single-label API host such as http://node2:2019, other than the bundled
  Caddy (caddy, localhost, the CADDY_ADMIN_URL host). Such names were
  treated as bridge-local and skipped the unauthenticated-admin guard.
• Upstream proxy targets are checked to be plain host:port
  (caddyapi.ScreenDialTarget): Caddy also accepts socket and
  file-descriptor upstream forms, which no address-based screen covers.
• Route owners could put Caddy placeholders into strings the node expands.
  Several tenant-editable fields reach parts of the node config that Caddy
  resolves at request time, which reached further than the intended text. Those
  fields now accept only request placeholders; everything else is refused when
  saved, and a route whose stored value is already unsafe is quarantined
  (503) rather than published. Shared nodes should be treated as having had
  their local secrets exposed and rotated.
• The access portal authorized a hostname, not a route. One host with
  several routes could be verified against the wrong route's policy. The
  verifier now resolves the exact route the same way the node does, from the
  panel's own gate config rather than anything the browser sends, and refuses
  when the route is ambiguous.
• Portal protection with nobody granted access served the route publicly.
  Removing the last grant widened access instead of narrowing it. Protection
  intent alone now denies; the gate is never dropped because the grant set is
  empty. The old implicit public bypass for static-looking paths is gone -
  see the upgrade notes.
• Disabling a user did not end their portal session. Portal sessions were a
  separate lifecycle that never saw the account's revocation epoch. They now
  share it and re-check that the account is active on every verification.
• A read error while compiling could publish a weaker config. Loading extra
  Basic Auth users and per-path mTLS rules treated a failed read as "no policy".
  Required policy reads now fail the build, so the node keeps its last good
  config instead of receiving one with the protection missing.
• Forward-auth changed the real request's method to GET. An authenticated
  POST, PUT or DELETE could reach the backend as a GET. The rewrite now happens
  inside the verifier subrequest, where it belongs.
• Passwordless passkeys did not require user verification although the
  session was recorded as MFA.
• An unauthenticated caller who knew an API key prefix could force expensive
  key-derivation work before any rate limit applied. The stored HMAC is now
  the verdict, and the legacy compatibility path has its own budget.
• Suspending a service did not reliably stop serving it, and one entry point
  deleted its routes outright. Suspend, resume and terminate now have a single
  implementation that every entry point calls - panel, bulk action, REST API and
  the FOSSBilling integration - in one transaction. Route rows are never
  deleted by a suspend, and the compiler independently refuses to emit a route
  whose service is not active.
• Updating a route through the API skipped the limits enforced when creating
  one. Update now validates the whole new state against the service's current
  plan and allocation, the same check the create path uses.
• Custom error pages: the background colour was spliced into an inline style
  block unescaped, and the logo link accepted any scheme.
• A route could make its node resolve a backend name through a resolver the
  route owner controlled. Panel-side screening was skipped whenever a tunnel,
  a bound peer or a custom DNS resolver was in play, which left the destination
  decided somewhere the panel could not check. The panel now resolves and pins
  those names itself; a tunnel-bound backend dials the peer directly. Choosing
  node-side resolution is still possible, but it is now an explicit super_admin
  decision per route rather than a side effect, and the resolver fields
  themselves require super_admin to set.
• The SSO provider URL was never checked. It is a route-owner-supplied
  destination that the node dials, including on an unauthenticated passthrough.
  It now goes through the same destination screening as every other backend, at
  save time and again before publishing.
• The node admin transport now defaults to authenticated. A key that cannot
  be decrypted no longer falls back to an unauthenticated connection - the push
  fails instead. Node registration no longer accepts an unauthenticated endpoint
  disguised by a remapped port or a hostname. Existing keyless nodes are
  grandfathered per node and clear themselves once a key works; see the upgrade
  notes.
• A single bad WAF directive took the whole node down. One route's rules
  could make the node reject its entire configuration, for every tenant on it.
  A route whose directives the node refuses is now quarantined on its own and
  the rest of the node still publishes. Arbitrary rule directives are now
  restricted to super_admin, are no longer copied by cloning a host, and are
  re-checked whenever a route is re-enabled. SecRemoteRules and
  SecRuleScript are refused outright.
• The geo-block redirect URL was validated as a URL but not against the
  placeholder policy, so a customer could hold back all of their own routes
  without seeing an error.
• Critical: tenant-controlled page text could read data from the node.
  Block, maintenance and error page text and redirect targets set by clients
  and scoped admins were not covered by the screening added in 1.6.0. They are
  now checked on save and neutralised again when the configuration is built.
  Mechanism details are withheld until operators have had time to upgrade;
  multi-tenant deployments should treat this as a priority upgrade.
• High: a second login factor the user never enrolled was accepted. 2FA now
  completes only with a method enabled on that account.
• High: social-login auto-provisioning defaulted to a role that can read
  every tenant. Accounts created by Google/GitHub/OIDC sign-in now default to
  client, and social-login provider settings require super_admin.
• The login rate limit could be skipped by the client; it is now exempted only
  for a live 2FA ticket on the 2FA endpoints.
• SSO copy-header names must be valid header names.
• The Gemini API key is sent in a header instead of the URL, so it no longer
  appears in logged transport errors.

Fixed

• A panel crash between saving a change and pushing it no longer leaves a
  node stale for up to 5 minutes. Pending pushes are now recorded durably
  (one row per node, written with the change itself where the write is
  transactional) and the leader retries them every 30s with backoff until the
  node accepts its config; the hosts list shows the retry count. Migration
  00164 adds the node_push_pending table. See
  ARCHITECTURE.md.
• A remote node without the GeoIP database no longer rejects its whole
  config. The geo matcher was gated on the panel's own copy of the mmdb, so a
  node missing the /data/geoip mount still got it and Caddy refused the
  entire /load. The node-agent now reports whether the file is present and
  the panel skips geo rules for that node only (fail-open on geo, never a
  bricked node), with a warning on the host's Geo tab and the node's page.
  Nodes without a reporting agent keep the old panel-side check. Migration
  00162 adds a nullable caddy_nodes.geoip_db_present; update the node-agent
  to get the per-node gate.
• A host with a fixed or random outbound IP emitted transport.local_addr,
  which Caddy does not know, so the node rejected the whole config load. It is
  now local_address.
• AI assistant: the default Anthropic model was claude-3-5-haiku-latest,
  which Anthropic retired on 2026-02-19, so chat failed on every install that
  had not picked a model. The default is now claude-haiku-4-5. The Gemini
  default moves from the retired gemini-1.5-flash to gemini-2.5-flash.
• The config compiler could block until its context deadline on SQLite, because
  it wrote audit rows while still holding the only pooled connection.
• Drift detection compared a different, smaller artifact than the one being
  published, so TLS and L4 changes on a node went unnoticed while per-node
  differences could trigger pointless full pushes. Push and reconcile now
  compare one canonical manifest of every managed section.
• A full push issued one UPDATE per route and ignored the results; it is now
  batched and checked.
• Secret rotation could leave the state file and the database under different
  keys. Rotation is now staged behind a preflight and a recovery journal.
• Database dumps read table after table with no shared snapshot, so a backup
  could contain a state that never existed. The dump now runs from one
  consistent snapshot.
• Backups reported success even when a required component was unreadable or the
  archive failed to close cleanly, and the "streaming" backup first buffered the
  whole dump in RAM. The dump now streams through a temp file, and the manifest
  records which components are required, present or skipped.
• An interrupted schema migration could leave an upgrade unable to continue.
  The reseller migrations now detect a partial apply and verify the shape of
  what they find, failing loudly on a mismatch rather than papering over it.
• Deleting a reseller left dangling references on SQLite, where the transform
  had silently dropped the foreign keys that MySQL relied on.
• A no-op PATCH could answer 404 for a route that exists.
• API list endpoints swallowed read errors, so a failed query looked like a
  short but successful list.
• An idempotency reservation could stay pending after its operation had already
  succeeded, blocking retries until the key expired.
• The command palette had two implementations fighting over the same DOM.
• The panel's own Content-Security-Policy blocked the inline confirmations and
  warnings the templates relied on, so those never ran. One of them, the ACME
  custom CA toggle, had no nonce at all.
• Form labels were not associated with their controls, the light theme's
  primary button did not meet WCAG AA contrast (3.74:1, now 5.47:1), and the
  document language was hard-coded regardless of the rendered language.
• A validation error on the host form no longer discards everything typed into
  it; the form comes back with the submitted values and the field marked.
• The Caddy healthcheck could pass on a leftover socket with nothing listening.
• Release images are pinned by digest and every published digest is scanned,
  rather than scanning a separate image built only for CI.
• CI provisions a real database, so the fail-closed tests run instead of
  skipping, and the Terraform provider module is covered.
• server doctor reported a node as reachable that the control plane would
  refuse to talk to, because it probed with a weaker client than the push path.
• server doctor gained a read-only inventory of stored routes that would not
  pass today's policies (backend destinations, placeholders, WAF directives),
  naming the route id and owner. It reports honestly when a check could not be
  performed rather than implying a clean result.
• hpg-node-agent doctor now checks that the GeoIP database is actually
  mounted on the node. A missing file previously made the node reject its whole
  configuration with no local signal.
• The Caddy healthcheck was fixed in the default and Portainer profiles, and
  added to the remote-node profile, which had none.

Upgrade notes

Read these before upgrading a multi-tenant install.

• Portal routes with no grants now deny. Any route left protected with an
  empty grant list was publicly reachable and will start returning 503. Grant
  access, or turn protection off, whichever you meant.
• The implicit public bypass for static-looking paths is gone. Requests that
  used to skip the portal because the path looked like an asset now hit the
  verifier. Fill portal_public_paths on each protected route before upgrading
  a single-page app, or expect a hard refresh to stampede the verifier.
• All portal sessions are logged out once. Existing sessions carry no epoch.
• Routes of suspended services stop serving. This is the fix, but if you
  have been relying on a suspend that never actually suspended, check those
  services first. Routes that the FOSSBilling path already deleted cannot be
  recovered by resuming - restore from a backup.
• Routes already outside their plan or port allocation will refuse to save
  until the configuration or the plan is corrected.
• Passkeys that cannot do user verification stop working for passwordless
  login. Inventory them with
  SELECT user_id, name FROM webauthn_credentials WHERE user_verified = 0, and
  make sure those users have a password or TOTP route back in before upgrading.
  Affected users must re-register the passkey.
• If APP_SECRET was ever rotated by hand rather than with
  hpg-rotate-secret, API keys now fail instead of quietly recovering. Run
  the rotation tool, or clear the stored HMAC so the keys re-derive.
• Idempotency keys left pending are marked unresolved, not deleted. A retry
  gets 409 with the operation id; reconcile against the audit log before
  clearing one.
• API lists: a failed read is now 500 rather than a short 200. Opt-in
  keyset pagination (?limit=, ?cursor=) was added to /api/v1/services and
  /api/v1/routes only; on those two an unpaginated result above 5000 rows is
  refused with 413 instead of being truncated. Every other list endpoint keeps
  its shape and got the error handling alone.
• Reconcile now reverts manual edits inside the config sections the panel
  manages. Certificates, logging and the admin listener stay unmanaged.
• Backups that were silently partial now fail. A backup reporting an
  unreadable component was producing an incomplete archive before.
• A backup's manifest.json reports components instead of contents.
• An interrupted hpg-rotate-secret leaves a .rotate-journal next to the
  state file. Re-run the same command to resume; a journal for a different
  secret pair blocks a new rotation until it is resolved.
• Node admin transport. Existing nodes with no admin-proxy key are
  grandfathered automatically, per node, and keep working; each such push logs
  it. The allowance clears itself on the first push where the node's key
  decrypts. Nodes added after this upgrade start fail-closed, so a remote node
  needs its agent admin proxy up before it can receive configuration. The
  allowance is now visible and editable on the node edit form.
  HPG_ALLOW_UNAUTHENTICATED_NODE_ADMIN=1 no longer authorizes a push - it only
  re-opens registration of a legacy URL. Use HPG_ADMIN_PROXY_PORTS if your
  agent proxy does not listen on 2021.
• Resolver fields are super_admin-only. A scoped admin saving a host that
  carries a custom DNS resolver will be refused until the field is cleared or a
  super_admin makes the change. Ordinary tunnel routes are unaffected: the panel
  still binds the peer itself.
• A backend name that the panel cannot resolve now holds its route back
  instead of being passed to the node to resolve. If a name only resolves inside
  a tunnel, either bind the tunnel peer or have a super_admin tick node-side
  resolution for that route.
• While an SSO provider name does not resolve from the panel, saving that host
  is refused - including edits to unrelated fields on it.
• SecRemoteRules and SecRuleScript stop working. Any route using them
  needs its rules rewritten before the upgrade.
• Run server doctor after upgrading: it now inventories stored rows that
  predate these policies. Nothing was migrated or rewritten, so an install can
  be holding values that today's checks would refuse.
• Social-login providers already saved with default role support keep it.
  If auto-provisioning is on for a public provider (Google, GitHub), review
  that choice in Settings - anyone with an account there gets that role.
• Pages or redirects containing Caddy placeholders outside {http.request.*}
  can no longer be saved. Existing ones are neutralised at build: a redirect
  answers 503, and custom maintenance HTML naming {env.*}, {file.*} or
  {system.*} falls back to the branded page.