Skip to content

v2.20.0

Choose a tag to compare

@AltanS AltanS released this 11 Nov 22:50
· 16 commits to main since this release

This release closes the Headscale admin API to every client outside a short allowlist (loopback by default), makes Headscale log the real client IP, and makes a dry run validate a new ACL policy. Traefik no longer logs a missing middleware at start for each tailnet route. It also fixes six smaller findings: a stale docker group on the debug user, a nightly prune that deleted volumes, a dry run that failed on a new timer, a sidecar that ignored SIGTERM, an always-changed buildkit task, and a new validator warning.

Added

  • bay validate warns when a tag in headscale_acl_policy.tagOwners has more than one owner, an owner that is a group: or an autogroup:*, or the owner *. The warning names the tag and says: every owner can enrol a node with that tag and so gets every grant of the tag. It never fails the run. See docs/tailnet-naming.md.
  • New variable docker_prune_volumes (see Fixed).

Changed

  • Headscale admin API: new variable headscale_admin_api_allowed_ips (access_gateway defaults, default ["127.0.0.1/32", "::1/128"]). The headscale-admin router (/api, /swagger) no longer uses vpn-chain. It uses public-chain, the headers and compression of the main Headscale router, plus its own IP allowlist headscale-admin-allow from the new variable. In 2.19.0 and 2.19.1 every tailnet node reached the admin API, also the nodes of an external tenant, and an API key was the only guard there. Now every client outside the list gets 403, tailnet nodes included. Nothing in Bay needs the path: bay gateway and registration use docker exec on the control host, and the identity sidecar uses the docker network. headscale_admin_api_public: true still opens the API to everyone and adds no label.
  • Headscale admin API: the deploy stops when headscale_admin_api_allowed_ips is empty, is not a list, or holds a value that is not an IP or a CIDR. Traefik drops a router whose allowlist it cannot parse.
  • Headscale admin API, fail-safe: while the API is closed, the main Headscale router leaves out the admin paths (Host(`<domain>`) && !PathPrefix(`/api`) && !PathPrefix(`/swagger`)). If the headscale-admin router is ever dropped, /api and /swagger get 404. Before, they fell through to the public main router. With headscale_admin_api_public: true the main rule stays Host(`<domain>`). The admin router keeps priority 1000, which is still above the length of the longer main rule.
  • Headscale: config.yaml now has trusted_proxies when Traefik fronts Headscale. New variable headscale_trusted_proxies (role headscale). Its default is computed on each run: the gateway of traefik_docker_network, read with docker network inspect, as a /32 (a /128 for an IPv6 gateway). Traefik runs with host networking, so it reaches the container from that address. Before, Headscale logged the gateway for every request, not the client. Only the gateway is trusted, not the subnet, so another container on the network cannot set the headers. The key and its rules are those of Headscale 0.29.7: prefixes only, 0.0.0.0/0 and ::/0 refused. The role stops on a bare IP or a /0, because Headscale would crash-loop. [] turns it off. The role creates the docker network before it reads it, so a fresh host gets the key on its first deploy.
  • Headscale: both headscale routers now remove a True-Client-IP header sent by the client (new middleware headscale-client-ip). Headscale reads that header before X-Real-IP and X-Forwarded-For, and Traefik passes it through, so with trusted_proxies a client could otherwise choose the IP Headscale logs. Traefik already replaces X-Real-IP and X-Forwarded-For.
  • Headscale admin API, docs: docs/tailnet-ingress.md renames the section to "The Headscale admin API is closed" and shows how to open the API for one device. docs/access-gateways.md, docs/multi-region.md, docs/tailnet-naming.md, the headscale role defaults header and the bay gateway apikey hints say the same. The 2.19.0 text that said the API answers over the tailnet is gone.

Fixed

  • Traefik: at each start, Traefik logged middleware "vpn-only@docker" does not exist once for each tailnet route (<route>-tailnet@file). The route file is loaded by the file provider, and the file provider was ready before the docker provider had read the vpn-only label on the traefik container. The routes worked a second later. The route file now defines its own copy of the allowlist, and each route names it as vpn-only@file. The docker-label routers keep vpn-only and vpn-chain as before. Both copies read vpn_allowed_ips with the same default, and a test renders both and compares the ranges.
  • Access gateway: the task that adds the Headscale tailnet range to vpn_allowed_ips is now tagged always. Before, bay deploy <env> --tags traefik skipped it, so the route file could get a shorter list than the label. It only sets a fact.
  • Headscale ACL: a dry run (-- --check) now validates a new headscale_acl_policy and fails on an invalid one. Before, check mode skipped "Check headscale container is running", then printed "SKIPPING ACL validation, the headscale container is not running", which was false, and the policy was first validated in the real run. The container probe, the staging write, headscale policy check and the removal of the staged file now run in check mode too (check_mode: false). They only read, or write the hidden staged file that the dry run removes again, also when the check fails. The install of the policy stays skipped in check mode. On a host where the container does not run, the dry run stages nothing and the warning is now true.
  • Debug agent: the debugbot account kept its membership of the docker group on a host that an older Bay version provisioned. The group is root-equivalent. The role only ever added groups (append: true), so it never took the membership away. With debug_agent_docker_access: false the role now reads id -nG and runs gpasswd -d <user> docker when the account is a member. The step is idempotent. The primary group and the other groups stay. It runs in check mode as a read only and skips the removal. docs/debug-agent.md says so.
  • Cron jobs: the nightly or weekly docker system prune -af --volumes deleted every volume that no container used at that moment. The data volume of a service whose deploy failed shortly before is such a volume. New variable docker_prune_volumes (default false). Without it the job runs docker system prune -af. The cost: an anonymous volume of a removed container stays on disk, a small leak. Set docker_prune_volumes: true to get the old command back. The comments in roles/cronjobs and the docs that quoted --volumes say the same.
  • Dry runs: bay provision <env> --tags outbound_monitor -- --check failed on a host that had no bay-reboot-alert.timer yet, with Could not find the requested service. Check mode writes no unit file, so the systemd module could not find the unit. The enable and the disable tasks of the three timers of outbound_monitor (outbound check, disk alert, reboot alert) now skip in check mode when the task that writes their unit file reports changed. The same flaw was in the backup timers and the maintenance timers (backup), the trigger watchdog timer (git_deploy) and the CrowdSec data refresh timer (crowdsec). They follow the same rule. A real run is not affected.
  • Tailnet identity: the tailnet-identity sidecar ignored SIGTERM. The server is PID 1 in its container, Linux does not apply the default SIGTERM action to PID 1, so docker stop waited 30 s and then killed it. The server now handles SIGTERM and SIGINT, stops listening and exits 0. A test starts it as PID 1 of a fresh PID namespace and checks that SIGTERM ends it in under 3 s.
  • Build server: the task Apply memory limit to the bay buildx container (and Apply memory limit to builder container for local builds) had changed_when: rc == 0. docker update exits 0 also when nothing moves, so the task said changed on every deploy. It now reads memory and swap of the container before and after, and reports changed only when they differ. See Upgrade notes for the part that is not a bug.

Upgrade notes

  • Apply on existing hosts, in this order. Run each command against the fleet of that env.
    1. Check your own Headscale API clients first (see "Headscale admin API" below). A client that calls /api/... over the tailnet gets 403 after step 3.
    2. bay provision <env> --tags agent-debug on each host with debug_agent_enabled: true. No container is touched.
    3. bay deploy <env> --tags headscale on the env that runs Headscale. Recreates headscale once; every node reconnects once.
    4. bay deploy <env> --tags tailnet_identity on the control and ingress env. Recreates tailnet-identity (new image); requests with identity_inject fail for a few seconds.
    5. bay deploy <env> --tags traefik on the ingress env. Rewrites dynamic/tailnet-proxies.yml; no container is recreated.
    6. bay deploy <env> --tags cronjobs on each env. Rewrites one cron line; no container is touched.
    7. Optional: bay deploy <env> --tags git_deploy on each env with builds, for the new buildkit memory reporting. No container is recreated.
      No other container is recreated by this release. The dry-run and validator changes need no command.
  • Headscale (admin API allowlist, trusted_proxies, True-Client-IP strip): run bay deploy <env> --tags headscale against the fleet that runs Headscale. The router labels change, so the headscale container is recreated once and every node reconnects once. This happens also with headscale_admin_api_public: true, because the main router gets the headscale-client-ip middleware. The same run renders trusted_proxies into config.yaml before the recreate, so there is no second restart. No other container is recreated. A plain bay up of the control box env also recreates the container for the labels, but it does not render config.yaml; run --tags headscale for trusted_proxies (one more restart).
  • Headscale trusted_proxies: after the deploy, docker logs headscale shows client IPs in the request lines instead of the docker gateway. If a line still shows the gateway, compare headscale_trusted_proxies with docker network inspect <network>.
  • Headscale admin API: check your own API clients first. A client that calls /api/... over the tailnet gets 403 after the deploy. Add the tailnet IP of its device to headscale_admin_api_allowed_ips as a /32, and keep the loopback entries. docs/tailnet-ingress.md has the steps.
  • Headscale ACL dry run: nothing to run. The next bay deploy <env> --tags headscale -- --check validates the policy. A dry run now writes /opt/headscale/config/.policy.hujson.staged for a moment and removes it. No container is recreated.
  • Traefik start noise: run bay deploy <env> --tags traefik on the ingress box (or the next bay up of that box env). It rewrites dynamic/tailnet-proxies.yml, and Traefik reloads the file without a restart. No container is recreated: no label changes.
  • Tailnet identity: run bay deploy <env> --tags tailnet_identity on the control and ingress env. The build context changed, so the role rebuilds the image, and the reconciler recreates the tailnet-identity sidecar (and only that container). Requests that use identity_inject fail for a few seconds while it restarts. Later stops are fast. A plain bay up of that box env does the same as a recreate step.
  • Debug agent: the role is in provision.yml (tags users and agent-debug). Run bay provision <env> --tags agent-debug on each existing host that has debug_agent_enabled: true. No container is touched. An SSH session that the account already has keeps its old groups until it logs in again.
  • Cron jobs: the role is in deploy.yml and is a rig role. Run bay deploy <env> --tags cronjobs. It rewrites one cron line. No container is touched. A fleet that wants --volumes back sets docker_prune_volumes: true in group_vars.
  • Build server: run bay deploy <env> --tags git_deploy to see the new reporting. No container is recreated. Fleets that share one build server share one buildkit container, so they share one memory cap. If two such fleets set different values of git_deploy_buildkit_memory_max, each deploy resets the cap that the other fleet set. The task also sets swap equal to memory, which turns container swap off. A fleet that pins memory and swap by hand sets git_deploy_buildkit_memory_max: "" at the all scope, and every other fleet on that build server must set the same value. docs/services.md says so.
  • Dry runs: nothing to apply. The change is in the task files. outbound_monitor and crowdsec are in provision.yml, backup and git_deploy are in deploy.yml.
  • Validator: nothing to run. Run bay validate and read the new warning, if any.