Skip to content

v1.6.0 - system review remediation

Choose a tag to compare

@marcoome marcoome released this 21 Sep 08:26
· 159 commits to main since this release
8c9cb59

Remediation release from an external security review of the whole system.
Fourteen findings closed: one critical, three high, three medium, six low and
one architectural note. Details of the critical and high findings are withheld
until operators have had time to upgrade; if you run a multi-tenant
deployment, treat this as a priority upgrade.

Security

  • Critical: a tenant could cross into the control plane. Reseller and
    scope-restricted accounts could influence where a node sends proxied
    traffic in a way that reached infrastructure they should never touch.
    Impact on an affected node is disclosure or replacement of its
    configuration. All tenant-supplied proxy targets are now screened
    fail-closed, at save time and again when the node configuration is built.
    Customer private ranges (RFC1918, CGNAT) remain valid origins.

  • High: remote nodes exposed an administrative interface on the mesh. The
    authenticated agent admin proxy is now required for remote nodes and the
    port is bound to loopback. Registering a node without authentication is
    refused. See the upgrade notes - this one needs a rolling migration.

  • High: external SSO did not cover every request. Protection depended in
    part on a value the client itself supplies, and did not apply uniformly
    across request methods. New routes are created in strict mode; existing
    routes keep their current setting and should be moved to strict.

  • High: tenant-supplied values could be expanded by the proxy before being
    sent to an origin the tenant controls. All such values are now screened on
    save and again at build.

  • Medium: a client could influence the source address the panel recorded
    and rate-limited on. The panel route now stamps the address the proxy
    actually saw; the middleware prefers the verified forwarded chain and trusts
    other headers only where an operator opts in with
    APP_TRUST_REALIP_HEADERS.

  • Medium: one tenant's web-application-firewall rule could block
    configuration changes for every other tenant on the same node.
    Scoped
    admins are now limited to removing rules by numeric id; unparseable entries
    are dropped when the configuration is built.

  • Low: the portal logout could be triggered cross-site. Logout is now a
    POST with a token bound to the portal session.

  • Secret files are created 0600. install_state.json kept whatever mode it
    had when restored or synced, commonly 0644.

Added

  • server healthcheck subcommand, and Compose healthchecks for the app and
    Caddy - the runtime image is distroless, so there is no shell for a curl
    probe. Start periods are sized for a long migration or certificate issuance.
  • server doctor reports per-node admin API authentication, secret file modes
    and the process umask, and names every missing or empty required environment
    variable with a generated value to use.
  • make pin-version V=x.y.z rewrites every image pin, and CI fails if the
    deploy profiles or the README disagree on the version.

Fixed

  • The remote-node bootstrap could not start. It bound the admin endpoint
    to an address that does not exist inside a bridge container, so the Caddy
    configuration failed to load and the node never came up.
  • The lite and Portainer profiles still deployed 1.3.2. A fresh install
    from either got code six months old, with none of the 1.4 or 1.5 fixes.
  • The remote GeoIP profile could not work: the agent wrote the database into
    its own container and remote Caddy ran a stock image with no geolocation
    module. Agent and Caddy now share a read-only volume and the profile uses
    the custom edge image.
  • make run claimed to load .env and did not; the config never parsed the
    file either.

Documentation

  • README and internal/domain/routes/placement.go claimed Redis-backed shared
    certificate storage. The module was never built into the image and the
    Caddyfile line was commented out, so each node has always issued its own
    certificates. The claim is removed and per-node issuance documented,
    including what to budget for ACME rate limits on failover. Building the
    shared store was considered and rejected: it would put private keys in Redis
    and make every node depend on it.
  • docs/SECURITY.md said rotating APP_SECRET needs no downtime, while
    cmd/rotate-secret requires the panel stopped and invalidates every HMAC API
    key. Replaced with a runbook that includes re-issuing the keys.
  • docs/ARCHITECTURE.md now states the real recovery window for deferred work
    after a crash (up to about five minutes) instead of implying durability.

Upgrade notes - read before deploying

  1. A route whose backend points at a managed node's own address, the panel, or
    the control mesh is now refused when the node configuration is pushed, with
    a route.blocked_target audit entry. Most plausible on single-box installs.
    The audit row is the diagnostic; the UI does not surface it.
  2. Additional upstreams and path-rule upstreams must now resolve panel-side
    unless the route is tunnel-bound.
  3. Editing a route whose header, rewrite or redirect contains a proxy
    placeholder expression blocks the save until the value is removed. At build
    such values are dropped, so the route keeps serving without that header.
  4. Scoped admins can no longer save free-form firewall rules; stored values
    keep working until edited.
  5. Closing the node admin interface is a four-step rolling migration: enable
    the tunnel and copy the proxy key, start the agent with it, repoint the
    node URL to the proxy port and confirm with server doctor, and only then
    move the Caddy mapping to loopback. Nothing breaks if you do nothing;
    HPG_ALLOW_UNAUTHENTICATED_NODE_ADMIN=1 keeps registration open during the
    migration.
  6. Operators running large active_active groups should check their ACME
    rate-limit headroom against nodes x hosts - certificates have always been
    issued per node, the documentation just said otherwise.