v1.9.0

Security and reliability release from a full internal code review. Upgrade
promptly, especially on shared, reseller or multi-node installs: many fixes
tighten tenant scoping, authentication and node isolation. Several defaults
now fail closed (GeoIP policies, WAF on nodes without Coraza, the plain-HTTP
installer, scoped admins without a resolvable scope). Read the upgrade notes
before updating: migrations 00166-00171 run on start, a new node-agent image
is needed, and a few API and Terraform behaviours change.

Added

• New clients can be invited by email instead of setting a password (needs SMTP).
• Issued mTLS client certificates have copy buttons and a .pem bundle download.
• Stream listen ports can be changed on edit.
• HPG_DRILL_DSN points the restore drill at a pre-created empty
  database, for MySQL users that cannot CREATE DATABASE (shared hosting).
• /api/v1/health reports the real build version.
• Terraform hpg_reseller.owner_password_wo (write-only, Terraform 1.11+)
  keeps the reseller owner password out of plan and state;
  owner_password is deprecated.
• POST /api/v1/resellers accepts the initial status, reseller_plan_id,
  overselling_allowed and can_create_plans, applied in the create
  transaction. The Terraform provider sends them on create.
• Terraform hpg_client.password_wo (write-only, Terraform 1.11+) keeps the
  client password out of plan and state; password is deprecated.
• Webhook POSTs carry X-HPG-Delivery: <id>, stable across retries, so
  receivers can drop duplicates.

Documentation

• Community health files: CODE_OF_CONDUCT.md, SUPPORT.md, CODEOWNERS, release-note
  categories; SECURITY.md now names the private advisory channel, supported
  versions and response targets. Issue forms add needs-triage.
• AI_USAGE.md states how AI coding assistants are used and how their output is
  checked.

Security

• Tunnel install tokens, webhook signing secrets and issued mTLS keys are shown
  once and never placed in redirect URLs.

• mTLS path rules match the decoded, cleaned request path; the most specific
  rule wins over rule order; unreadable rules block the request.

• Restricted and reseller admins fail closed when their scope can't be
  resolved (lists, exports, world map); only super_admin creates global access
  groups; CSV exports neutralise formula cells.

• Password login carries a CSRF token signed with a key derived from
  APP_SECRET.

• The install wizard loads only over TLS or from loopback (SSH port-forward);
  INSTALL_ALLOW_PLAINTEXT=1 opts out.

• Tunnel and node bootstrap tokens are no longer read from URLs or passed on
  the command line; /api/wg/* query strings are redacted from stored access
  logs and WAF events.

• Routes with a country policy deny on a node without GeoIP, and blocking-mode
  WAF routes answer 503 on a node without Coraza, instead of serving
  unfiltered. Only the affected route changes; waf.fail_open and the
  per-route geo setting are the explicit opt-outs.

• An undecryptable CAPTCHA secret refuses verification instead of switching
  CAPTCHA off; Telegram errors no longer log the bot token.

• The Terraform provider requires an https endpoint outside loopback, and
  omitted route/plan flags keep the secure server defaults. API route create
  without ssl/force_https now enables both, as documented.

• The provider release workflow pins actions by SHA with least privilege.

• Backup docs use a read-only hpg_backup user with an option file instead
  of the root password on the command line.

• Caddy node image 2.11.4 -> 2.11.7 (builder on Go 1.27.2) with
  golang.org/x/net forced to v0.60.0 via xcaddy --replace (Caddy resolves
  v0.59.0): CVE-2026-78667, CVE-2026-78669, CVE-2026-97031 (HIGH, DoS). All
  Caddy pins (compose, node-join, docs) moved together.

• API keys with an empty scope list no longer get full access; they now pass
  no scope check, and the admin key form rejects a key with no scope selected
  (#53).

• Customer-issued API keys (client:read/client:write) can reach the
  customer endpoints of API v1; admin access is not widened (#59).

• A slave panel is read-only on every surface, not only /admin; register and
  password reset work on the master only (#52).

• Runtime 2FA enforcement fails closed when the policy can't be read (503 until
  the database answers) and the admin grace window is issued once (#55, #56).

• Public auth POSTs stay rate limited for logged-in sessions, and passkey login
  challenges are throttled (#57, #58).

• Malformed IP allow/trust lists deny instead of allowing everyone; bad entries
  are logged as "invalid CIDR in IP allow/trust list" (#54).

• Auto-failover reserves capacity per route so a single healthy node is not
  overloaded, and plan/reseller domain quotas are checked inside the create
  transaction (#49, #50).

• Revoked mTLS client certificates are refused on the nodes, mTLS roles are
  bound to one certificate (serial + fingerprint), and deleting a CA pushes the
  change to every node that trusted it.

• The forward-auth portal enforces every enrolled second factor (TOTP, SMS,
  email, recovery code). Portal OAuth finishes on the host it started from and
  is bound to the browser.

• SMS/email codes, recovery codes, passkey challenges and TOTP steps are each
  accepted once, also under concurrent requests.

• OIDC issuers must use https (no localhost exemption).

• Reseller and restricted admins see only their own clients on the dashboard;
  resellers can place hosts and streams only on node pools their package
  grants; DNS steering is super_admin-only and verified domains only.

• Reseller quotas (clients, services, domains) hold under concurrent creates.

• Node decommission keeps the configured Caddy admin bind and clears the node
  before deleting it ("Force decommission" for an offline node).

• Customer WireGuard peers can only answer connections, not open them to the
  node, its LAN or the internet (needs the new node-agent image).

• L4 streams accept inbound PROXY headers only from listed balancer CIDRs.

• WebSocket requests no longer skip the WAF.

• The backend test pins the address it screened and refuses managed nodes and
  the control plane.

• The restore drill replays only an artifact whose SHA-256 matches its backup
  record and only statements our own dumps emit.

• Secret rotation aborts on any unreadable row, can be resumed after a crash
  at any step, refuses --apply without database access and takes the DB
  password from DB_PASSWORD or --db-pass-file.

• SQLite installs enforce foreign keys; WAL and busy_timeout now apply.

• Auth helper cookies are cleared with HttpOnly/SameSite/Secure; node-token
  mismatches log a hash instead of a token prefix; hpg-restore rejects
  non-local archive paths; form integers are bounded to 32 bits; remaining
  Dockerfile base images are pinned by digest; chi and klauspost/compress
  bumped.

• The OpenAPI spec no longer advertises ?node_token= (it was never accepted).

Fixed

• Wildcard DNS-01 certificates are actually ordered now: the *.zone cert is
  requested as soon as the config loads, and a route can take the alias
  *.<zone> to serve every subdomain on that one cert (no per-subdomain
  certificates, no TXT proof for the wildcard alias; platform admins only, on
  a route without a path prefix). Access-log and WAF events for those
  subdomains are attributed to the route. The SSL tab lists which
  DNS-01 requirement is missing, and "Renew certificate now" re-pushes the
  full node config.
• A Cloudflare token in a format the Caddy DNS module rejects (quotes,
  spaces, a Bearer prefix) is refused when saved, and a stored one is
  skipped instead of failing the node's whole config load.
• Host edit shows alias validation errors (invalid or already-used alias)
  instead of re-rendering the submitted value as if it was saved, and opens
  the tab that holds the error.
• The central node-agent now sees the GeoIP database, so country rules filter
  traffic instead of failing closed on the panel node.
• Saving a host after adding a redirect, rewrite or block location works again.
• Editing a stream keeps all its upstreams.
• Creating a webhook without a secret generates one instead of failing.
• The AI assistant's route detail lookup works and includes WAF, rate limit,
  geo and mTLS settings.
• Ops summary counts and chart hours (shown in browser local time).
• Refreshing the page after issuing an mTLS certificate no longer issues another.
• Error toasts stay until dismissed; success shows one notice, not two.
• mTLS CA form picks the client from a list; add-host form uses full width;
  client detail lists every custom field.
• Alert cooldown now applies to rules with changing measurements (stale WG
  peer, error rate, DB pool, WAF surge); they no longer fire every minute.
• Browsers no longer autofill the admin login into webhook, DNS provider,
  backup and other secret fields.
• Dashboard and Stats request counts match and survive node counter resets;
  the 24h chart no longer piles all traffic into the last hour.
• Row action menus stay inside the viewport; the Legacy aliases page renders;
  Settings sections follow the URL hash.
• Placeholders no longer look like saved values; the command palette is
  readable in dark mode; backup destination form shows only the fields for
  the chosen kind and the job list refreshes while a job runs.
• Editing a stream keeps its CIDR lists; creating a stream with extra
  upstreams is all-or-nothing; reissuing a tunnel keeps its key-rotation age;
  a failed PSK clear keeps the rekey token usable.
• Backup schedule saves are atomic; world map totals include traffic beyond
  the first 5000 IPs.
• Access-log ingest is all-or-nothing per batch (503 on a storage error, the
  agent resends), raw rows and rollups are written together, and the agent
  ships ~1 MiB batches. Prometheus samples with timestamps parse correctly;
  live log unsubscribe no longer races publication.
• Webhook deliveries are claimed atomically before sending; WireGuard
  rotation alerts are marked only after they are recorded; alert cooldown
  changes apply without a restart; the Cloudflare settings cache reloads once
  per expiry.
• The client service detail page renders; the status page traffic chart works
  under the CSP; registration states the real 12-character minimum.
• Nodes created through POST /api/v1/nodes are approved and receive routes;
  reseller creation from Terraform no longer orphans the tenant on a failed
  read.
• The Portainer external-DB profile publishes the app on loopback so the
  installer is reachable.
• Streams with more than one upstream load on nodes again: the panel now sends
  the load-balancing key caddy-l4 v0.1.1 accepts and no per-upstream weights.
• Terraform resources keep their id when the read right after create fails.
• SQLite installs on Go 1.27 builds: the alert cooldown, the stale restore
  drill alert and the alert page's last-fired time read timestamps correctly
  again (repeat alerts were not suppressed).
• Drain and decommission move a node's active-active/failover replica
  assignments too, without double-counting a destination that already serves
  the route; a drain re-syncs the drained node and rolls back if it can't.
• SSE log and AI chat streams are no longer cut off by the global 30s request
  timeout (#51).
• Auto-failover measures an outage from the last successful probe, so it and
  the node_offline alert fire after their grace period.
• Deleted, disabled or un-steered routes no longer leave managed DNS records
  live; DNS steering and alias rechecks cover every route, not the first 500;
  Cloudflare record listing follows pagination.
• Fan-out peers get wildcard DNS-01 credentials, route edits, maintenance and
  basic-auth changes, and fleet/client-wide pushes; replacing an unverified
  route frees capacity on every node.
• HA WireGuard creation is all-or-nothing, and an install token is used up
  only when the config download succeeds.
• A database error while reading stream upstreams or WAF suppressions fails
  the push (the node keeps its last good config) instead of dropping them.
• Tenants can't add a path prefix on route edit when their plan lacks path
  routing; narrowing a service port range is refused while routes use the
  removed ports (API: 409); the OpenAPI spec lists POST /api/v1/services
  again.
• Port-range changes and route writes lock the service row, so a concurrent
  route edit can't land outside a narrowed range; an aborted node migration
  re-syncs every destination it touched.
• SSO-created client users get their client record; self-registration is
  all-or-nothing.
• Bulk approve enables nodes; drain/decommission spread routes over peers by
  free capacity and roll back on a failed push; "Send test email" no longer
  saves SMTP settings; multi-key settings saves (OIDC) are atomic.
• Overlapping tcp/udp/both streams on one port are rejected instead of
  silently overwritten; concurrent first-time WireGuard keypair creation can
  no longer store a mismatched pair.
• GDPR erasure is all-or-nothing and also removes phone, OTP enrolment, SSO
  links, passkeys, email verifications, AI chat history and the client's
  display name, notes and custom fields.
• MySQL backups keep binary columns (WebAuthn, TOTP) intact; verification no
  longer poisons later verifications and drills; retention deletes artifacts
  at the destination too; the drill fails on any SQL error or empty core
  tables.
• The install wizard no longer reports completion when the state file can't be
  written; GeoIP refresh reports undecryptable MaxMind credentials as a
  failure; WAF events on external routes with prefix stripping go to the right
  route; the legacy /api/node/wg/handshakes endpoint applies JSON reports.

Upgrade notes

• Portainer/compose installs: add - geoip_data:/data/geoip:ro under
  hpg-node-agent volumes, or GeoIP rules on the panel node keep denying all
  traffic (fail-closed) or skip filtering (fail-open).
• Migration 00171 adds a short-lived ledger that makes access-log batch
  retries idempotent.
• Stream upstream weights are no longer sent to nodes; saving a stream with a
  weight other than 1 is refused. Remove weights from existing streams on
  their next edit.
• Terraform: move hpg_reseller.owner_password to owner_password_wo.
• Migrations 00169-00170 run on start: stored /api/wg/* access-log queries
  are redacted, and every route's GeoIP policy becomes fail-closed (untick it
  per route to keep serving on nodes without GeoIP).
• Plain-HTTP access to /install from a remote address gets 403. Use
  ssh -L 8080:127.0.0.1:8080, HTTPS, or INSTALL_ALLOW_PLAINTEXT=1.
• Non-browser clients posting /auth/login must first GET the page and send
  back csrf_token with its cookie.
• Blocking-mode WAF routes on nodes without Coraza answer 503 until the node
  runs the WAF image or waf.fail_open is set.
• Old install.sh?token= commands prompt for the token; scripts calling
  /api/wg/* with ?token= must send Authorization: Bearer. Node scripts'
  --token is deprecated in favour of the prompt or --token-file.
• API callers relying on omitted ssl/force_https (routes) or
  ssl_enabled/websocket_enabled (plans) meaning off must send false.
• Terraform: switch http:// endpoints to https and move
  hpg_client.password to password_wo.
• Rotate the Telegram bot token if older logs may contain it.
• If every login fails after an APP_SECRET change, see
  TROUBLESHOOTING.md.
• The Portainer external-DB profile now binds 127.0.0.1:8080; set APP_PORT
  or APP_BIND_HOST if that clashes.
• Path rules on mTLS routes: the most specific rule now decides. Review rule
  sets that relied on an older broad rule taking precedence.
• Release the new node-agent image for the smaller access-log batches.
• Migration 00165 gives every existing unscoped API key the explicit scopes it
  effectively had: client:read,client:write for customer-owned keys,
  services,routes,nodes,admin:read,admin:write for every other owner. Keys
  keep working unchanged; review the migrated keys under Admin > API keys and
  narrow any that should not carry the full set.
• Migrations 00166-00168 run on start: DNS steering record ownership (backfilled
  from existing steering state), a verification kind for backup jobs, and a
  per-stream trusted PROXY sender list.
• After upgrading, DNS records the panel created for routes that are deleted,
  disabled, unverified or no longer steered are deleted at the provider on the
  first reconcile pass. Records you created by hand are never touched.
• Streams with inbound PROXY enabled close every connection until their
  balancer CIDRs are entered under "Trusted PROXY senders".
• Release a new node-agent image to every node for the WireGuard peer
  isolation. WebSockets on WAF routes need the v1.8.0 edge image or newer.
• Resellers without a package granting node pools can no longer create hosts
  or streams. Only super_admin can see or change DNS steering.
• An http:// OIDC issuer stops OIDC login until it is switched to https.
• Until a node receives the new config, mTLS paths with role rules return 403;
  the normal push or drift reconcile clears it.
• Decommission fails if the node can't be cleared; use "Force decommission"
  for an offline node and stop Caddy there by hand. Nodes bulk-approved before
  this release stay disabled until enabled.
• hpg-rotate-secret --apply requires the DB flags; --db-pass is deprecated.
• SQLite: on first start, rows pointing at deleted parents are purged
  (sqlite: purged orphan row) or logged (orphan row left in place). Take a
  backup first.
• Restore drills on backups with no recorded SHA-256 fail; take a new backup.
  The first retention run deletes expired artifacts that were left behind.
  Older MySQL backups may hold binary columns in the wrong encoding.
• Portal OAuth sign-ins in progress during the upgrade must be retried.
• Node-token log field token_prefix is now token_fp.