Repository navigation
v2.20.0
This release closes the Headscale admin API to every client outside a short allowlist (loopback by default), makes Headscale log the real client IP, and makes a dry run validate a new ACL policy. Traefik no longer logs a missing middleware at start for each tailnet route. It also fixes six smaller findings: a stale docker group on the debug user, a nightly prune that deleted volumes, a dry run that failed on a new timer, a sidecar that ignored SIGTERM, an always-changed buildkit task, and a new validator warning.
Added
bay validatewarns when a tag inheadscale_acl_policy.tagOwnershas more than one owner, an owner that is agroup:or anautogroup:*, or the owner*. The warning names the tag and says: every owner can enrol a node with that tag and so gets every grant of the tag. It never fails the run. Seedocs/tailnet-naming.md.- New variable
docker_prune_volumes(see Fixed).
Changed
- Headscale admin API: new variable
headscale_admin_api_allowed_ips(access_gateway defaults, default["127.0.0.1/32", "::1/128"]). Theheadscale-adminrouter (/api,/swagger) no longer usesvpn-chain. It usespublic-chain, the headers and compression of the main Headscale router, plus its own IP allowlistheadscale-admin-allowfrom the new variable. In 2.19.0 and 2.19.1 every tailnet node reached the admin API, also the nodes of an external tenant, and an API key was the only guard there. Now every client outside the list gets 403, tailnet nodes included. Nothing in Bay needs the path:bay gatewayand registration usedocker execon the control host, and the identity sidecar uses the docker network.headscale_admin_api_public: truestill opens the API to everyone and adds no label. - Headscale admin API: the deploy stops when
headscale_admin_api_allowed_ipsis empty, is not a list, or holds a value that is not an IP or a CIDR. Traefik drops a router whose allowlist it cannot parse. - Headscale admin API, fail-safe: while the API is closed, the main Headscale router leaves out the admin paths (
Host(`<domain>`) && !PathPrefix(`/api`) && !PathPrefix(`/swagger`)). If theheadscale-adminrouter is ever dropped,/apiand/swaggerget 404. Before, they fell through to the public main router. Withheadscale_admin_api_public: truethe main rule staysHost(`<domain>`). The admin router keeps priority 1000, which is still above the length of the longer main rule. - Headscale:
config.yamlnow hastrusted_proxieswhen Traefik fronts Headscale. New variableheadscale_trusted_proxies(roleheadscale). Its default is computed on each run: the gateway oftraefik_docker_network, read withdocker network inspect, as a/32(a/128for an IPv6 gateway). Traefik runs with host networking, so it reaches the container from that address. Before, Headscale logged the gateway for every request, not the client. Only the gateway is trusted, not the subnet, so another container on the network cannot set the headers. The key and its rules are those of Headscale 0.29.7: prefixes only,0.0.0.0/0and::/0refused. The role stops on a bare IP or a/0, because Headscale would crash-loop.[]turns it off. The role creates the docker network before it reads it, so a fresh host gets the key on its first deploy. - Headscale: both headscale routers now remove a
True-Client-IPheader sent by the client (new middlewareheadscale-client-ip). Headscale reads that header beforeX-Real-IPandX-Forwarded-For, and Traefik passes it through, so withtrusted_proxiesa client could otherwise choose the IP Headscale logs. Traefik already replacesX-Real-IPandX-Forwarded-For. - Headscale admin API, docs:
docs/tailnet-ingress.mdrenames the section to "The Headscale admin API is closed" and shows how to open the API for one device.docs/access-gateways.md,docs/multi-region.md,docs/tailnet-naming.md, theheadscalerole defaults header and thebay gateway apikeyhints say the same. The 2.19.0 text that said the API answers over the tailnet is gone.
Fixed
- Traefik: at each start, Traefik logged
middleware "vpn-only@docker" does not existonce for each tailnet route (<route>-tailnet@file). The route file is loaded by the file provider, and the file provider was ready before the docker provider had read thevpn-onlylabel on the traefik container. The routes worked a second later. The route file now defines its own copy of the allowlist, and each route names it asvpn-only@file. The docker-label routers keepvpn-onlyandvpn-chainas before. Both copies readvpn_allowed_ipswith the same default, and a test renders both and compares the ranges. - Access gateway: the task that adds the Headscale tailnet range to
vpn_allowed_ipsis now taggedalways. Before,bay deploy <env> --tags traefikskipped it, so the route file could get a shorter list than the label. It only sets a fact. - Headscale ACL: a dry run (
-- --check) now validates a newheadscale_acl_policyand fails on an invalid one. Before, check mode skipped "Check headscale container is running", then printed "SKIPPING ACL validation, the headscale container is not running", which was false, and the policy was first validated in the real run. The container probe, the staging write,headscale policy checkand the removal of the staged file now run in check mode too (check_mode: false). They only read, or write the hidden staged file that the dry run removes again, also when the check fails. The install of the policy stays skipped in check mode. On a host where the container does not run, the dry run stages nothing and the warning is now true. - Debug agent: the
debugbotaccount kept its membership of thedockergroup on a host that an older Bay version provisioned. The group is root-equivalent. The role only ever added groups (append: true), so it never took the membership away. Withdebug_agent_docker_access: falsethe role now readsid -nGand runsgpasswd -d <user> dockerwhen the account is a member. The step is idempotent. The primary group and the other groups stay. It runs in check mode as a read only and skips the removal.docs/debug-agent.mdsays so. - Cron jobs: the nightly or weekly
docker system prune -af --volumesdeleted every volume that no container used at that moment. The data volume of a service whose deploy failed shortly before is such a volume. New variabledocker_prune_volumes(defaultfalse). Without it the job runsdocker system prune -af. The cost: an anonymous volume of a removed container stays on disk, a small leak. Setdocker_prune_volumes: trueto get the old command back. The comments inroles/cronjobsand the docs that quoted--volumessay the same. - Dry runs:
bay provision <env> --tags outbound_monitor -- --checkfailed on a host that had nobay-reboot-alert.timeryet, withCould not find the requested service. Check mode writes no unit file, so the systemd module could not find the unit. The enable and the disable tasks of the three timers ofoutbound_monitor(outbound check, disk alert, reboot alert) now skip in check mode when the task that writes their unit file reportschanged. The same flaw was in the backup timers and the maintenance timers (backup), the trigger watchdog timer (git_deploy) and the CrowdSec data refresh timer (crowdsec). They follow the same rule. A real run is not affected. - Tailnet identity: the
tailnet-identitysidecar ignored SIGTERM. The server is PID 1 in its container, Linux does not apply the default SIGTERM action to PID 1, sodocker stopwaited 30 s and then killed it. The server now handles SIGTERM and SIGINT, stops listening and exits 0. A test starts it as PID 1 of a fresh PID namespace and checks that SIGTERM ends it in under 3 s. - Build server: the task
Apply memory limit to the bay buildx container(andApply memory limit to builder containerfor local builds) hadchanged_when: rc == 0.docker updateexits 0 also when nothing moves, so the task saidchangedon every deploy. It now reads memory and swap of the container before and after, and reportschangedonly when they differ. See Upgrade notes for the part that is not a bug.
Upgrade notes
- Apply on existing hosts, in this order. Run each command against the fleet of that env.
- Check your own Headscale API clients first (see "Headscale admin API" below). A client that calls
/api/...over the tailnet gets 403 after step 3. bay provision <env> --tags agent-debugon each host withdebug_agent_enabled: true. No container is touched.bay deploy <env> --tags headscaleon the env that runs Headscale. Recreatesheadscaleonce; every node reconnects once.bay deploy <env> --tags tailnet_identityon the control and ingress env. Recreatestailnet-identity(new image); requests withidentity_injectfail for a few seconds.bay deploy <env> --tags traefikon the ingress env. Rewritesdynamic/tailnet-proxies.yml; no container is recreated.bay deploy <env> --tags cronjobson each env. Rewrites one cron line; no container is touched.- Optional:
bay deploy <env> --tags git_deployon each env with builds, for the new buildkit memory reporting. No container is recreated.
No other container is recreated by this release. The dry-run and validator changes need no command.
- Check your own Headscale API clients first (see "Headscale admin API" below). A client that calls
- Headscale (admin API allowlist,
trusted_proxies,True-Client-IPstrip): runbay deploy <env> --tags headscaleagainst the fleet that runs Headscale. The router labels change, so theheadscalecontainer is recreated once and every node reconnects once. This happens also withheadscale_admin_api_public: true, because the main router gets theheadscale-client-ipmiddleware. The same run renderstrusted_proxiesintoconfig.yamlbefore the recreate, so there is no second restart. No other container is recreated. A plainbay upof the control box env also recreates the container for the labels, but it does not renderconfig.yaml; run--tags headscalefortrusted_proxies(one more restart). - Headscale
trusted_proxies: after the deploy,docker logs headscaleshows client IPs in the request lines instead of the docker gateway. If a line still shows the gateway, compareheadscale_trusted_proxieswithdocker network inspect <network>. - Headscale admin API: check your own API clients first. A client that calls
/api/...over the tailnet gets 403 after the deploy. Add the tailnet IP of its device toheadscale_admin_api_allowed_ipsas a/32, and keep the loopback entries.docs/tailnet-ingress.mdhas the steps. - Headscale ACL dry run: nothing to run. The next
bay deploy <env> --tags headscale -- --checkvalidates the policy. A dry run now writes/opt/headscale/config/.policy.hujson.stagedfor a moment and removes it. No container is recreated. - Traefik start noise: run
bay deploy <env> --tags traefikon the ingress box (or the nextbay upof that box env). It rewritesdynamic/tailnet-proxies.yml, and Traefik reloads the file without a restart. No container is recreated: no label changes. - Tailnet identity: run
bay deploy <env> --tags tailnet_identityon the control and ingress env. The build context changed, so the role rebuilds the image, and the reconciler recreates thetailnet-identitysidecar (and only that container). Requests that useidentity_injectfail for a few seconds while it restarts. Later stops are fast. A plainbay upof that box env does the same as a recreate step. - Debug agent: the role is in
provision.yml(tagsusersandagent-debug). Runbay provision <env> --tags agent-debugon each existing host that hasdebug_agent_enabled: true. No container is touched. An SSH session that the account already has keeps its old groups until it logs in again. - Cron jobs: the role is in
deploy.ymland is a rig role. Runbay deploy <env> --tags cronjobs. It rewrites one cron line. No container is touched. A fleet that wants--volumesback setsdocker_prune_volumes: truein group_vars. - Build server: run
bay deploy <env> --tags git_deployto see the new reporting. No container is recreated. Fleets that share one build server share one buildkit container, so they share one memory cap. If two such fleets set different values ofgit_deploy_buildkit_memory_max, each deploy resets the cap that the other fleet set. The task also sets swap equal to memory, which turns container swap off. A fleet that pins memory and swap by hand setsgit_deploy_buildkit_memory_max: ""at theallscope, and every other fleet on that build server must set the same value.docs/services.mdsays so. - Dry runs: nothing to apply. The change is in the task files.
outbound_monitorandcrowdsecare inprovision.yml,backupandgit_deployare indeploy.yml. - Validator: nothing to run. Run
bay validateand read the new warning, if any.