Repository navigation
Releases
v2.22.0
This release lets an egress container load the fleet's own public websites.
Added
- bay.toml: new key
egress_fleet = "web"on a[services.<name>]withegress = "public". The egress container may then also open new tcp connections to ports 80 and 443 of the public address of every fleet box, its own box too. The addresses are theansible_host(or the inventory name when it is an address) and thenetplan_addressof every inventory host, the same list that feeds the blocked set. Every other port (SSH too), UDP, IPv6, the tailnet, the private ranges, the gateway and the other containers stay closed. The validator refuses the key on a service withoutegress, any value but"web", and the key at the top level or in[deploy.<env>]. It is never inherited. - Compile: the egress entry in
services.ymlgetsfleet: web. The key is absent when unset, so every compiledservices.ymlstays the same, byte for byte. - Firewall (
roles/container_lifecycle):/etc/nftables.d/bay-egress.confgets two accept rules for each flagged container. Chainfrom_egressopens the other boxes (forward hook). Chainegress_to_hostopens this box's own address (input hook). Both match the container's bridge and its own address, so a client on the same egress network gains nothing. Both run before every drop. The rule for other boxes never sends the request out through the tailnet interface (tailscale*), where it would carry a tailnet address thatvpnroutes trust. Only a single public IPv4 address opens. A host name, a network, and a tailnet, private or reserved address are skipped. An address inbay_egress_blocked_extrastays closed: the block wins. Without the key the file is the same, byte for byte. bay plan: addingegress_fleetgives a secondcontainerstep with riskshared. Its reason names the container and says that it may open tcp 80 and 443 on the public address of every fleet box. So the plan needsbay approve. Removing the key gives the same step with risksafe.docs/plan.mdhas both rows in the risk table.- Docs:
docs/bay-toml.mdhas a new section "egress_fleet: the fleet's own websites": what opens, what stays closed, two side effects to check first, and the verify commands.docs/services.mddescribesfleet: web.
Upgrade notes
- Nothing changes until a service sets the key. The compiled
services.ymland the egress firewall file stay the same, so the upgrade reloads no firewall and recreates no container. - To turn it on, add
egress_fleet = "web"to the egress service and runbay plan, thenbay approve <plan-id> --reason "<why>", thenbay up. The plan showschanged: egressand the stepegress_fleet = "web" added: <container> may open tcp 80 and 443 on the public address of every fleet box, its own box too; bay up opens the box firewall for it, without a recreate.bay uploads the new rules. The container is not recreated. - Check two side effects first. Another box sees the requests from this box's public address, and the CrowdSec allowlist of the fleet's own addresses never bans it. Keep box addresses out of
vpn_allowed_ipsand every app allowlist. On its own box, Traefik sees the container's private address (inbay_egress_subnet_pool) as the client.vpnroutes stay 403, but an app that trusts private client addresses would serve the crawler as an internal user. - Do not set the key on a box that routes fleet addresses through another overlay or VPN interface (one not named
tailscale*). The request would then arrive from an address thatvpnroutes may trust. - Behind a cloud 1:1 NAT (the public address is not on the box's own interface), the crawler cannot reach its own box. The rules fail closed there. The other boxes still answer.
- A new box opens at the next
bay upon the egress container's box, like the blocked set.
v2.21.0
This release gives the Zot registry longer request time limits and adds pull-only registry accounts. It also stops a verbose run from printing the registry admin password.
Added
- Zot: new variables
zot_http_read_timeoutandzot_http_write_timeout(rolezot, default"30m"). They renderhttp.readTimeoutandhttp.writeTimeoutinconfig.json. Zot's own default is 60s, an absolute limit for the whole request or response body. A layer upload that still streamed at 60s was cut: zot loggedi/o timeoutand the proxy answered 502. A large pull on a slow link was cut the same way.""renders nothing (zot's 60s)."0s"turns a limit off. The deploy refuses a value that is not a Go duration. The keys need zot v2.1.17 or later; the role pins v2.1.22. - Zot: new variable
zot_pull_users(rolezot, default[]), a list ofusernameandpassword. Each entry is an htpasswd account that can only read (pull) every repository. Thedocker_registriesentry forzot_domainstays the only account that can push and delete. With at least one entry,config.jsongetshttp.accessControl:readon**for the pull accounts, and every action for the admin throughadminPolicy. With the list empty the block is absent and zot has one account, as before. The deploy refuses an entry without a username or a password, a username that does not match^[a-z0-9][a-z0-9._-]{0,62}$, a duplicate, and the admin username. A machine logs in withdocker login <zot_domain> -u <username>. A pull account cannot write a buildx registry cache (git_deploy_registry_cache). - Traefik: new variable
traefik_public_read_timeout(roletraefik, default""). It sets the request read limit of the publicwebsecureentrypoint, with and withouttraefik_split_entrypoints. It applies to every router on that host. Set it only on the host that serves the registry, when machines push over the public address. Empty renders nothing, sotraefik.ymlis unchanged and Traefik keeps its 60s default. The deploy refuses a value that is not whole seconds or a Go duration. - Docs:
docs/build-strategies.mdhas a new section "Registry accounts and time limits".docs/build-pipeline.mdhas a troubleshooting entry for a push that fails at exactly 60 seconds.
Changed
- Zot:
htpasswdnow holds the admin line plus one line per pull account. It still changes only when something changed. The role verifies each account withhtpasswd -iv(password on stdin), and a read-only probe compares the user names in the file with the wanted names. When either check fails, the role writes the whole file again and restarts zot. So a removed pull account and an unknown line added by hand also disappear. Both probes run in check mode too.
Fixed
- Zot: the task that reads the admin account from
docker_registrieshad nono_log, so a run with-vprinted the registry admin password. It now hasno_log: true.
Upgrade notes
- Zot restarts once. The first run that applies the
zotrole on the registry host after the upgrade writes the two new time limits intoconfig.jsonand restarts zot. That isbay upon that env, or abay deploythat runs the rig roles. Other regions pull from this registry, so run the registry host's env first, or runbay deploy <env> --tags zotthere before the others. To keep zot's old 60s limits and skip the restart, set bothzot_http_read_timeoutandzot_http_write_timeoutto"". - Pull accounts are opt-in. Put each password in the vault, add
zot_pull_usersto the group_vars of the registry host, and runbay deploy <env> --tags zot. The role rewriteshtpasswdandconfig.jsonand restarts zot once. A deployment host can use a pull account through its owndocker_registriesentry forzot_domain. The registry host and the build server must keep the admin entry. - The public entrypoint limit is opt-in. Nothing changes until you set
traefik_public_read_timeout. When machines push to the registry over its public address, set it (for example"30m") on the registry host and runbay deploy <env> --tags traefik. Traefik restarts once.
v2.20.0
This release closes the Headscale admin API to every client outside a short allowlist (loopback by default), makes Headscale log the real client IP, and makes a dry run validate a new ACL policy. Traefik no longer logs a missing middleware at start for each tailnet route. It also fixes six smaller findings: a stale docker group on the debug user, a nightly prune that deleted volumes, a dry run that failed on a new timer, a sidecar that ignored SIGTERM, an always-changed buildkit task, and a new validator warning.
Added
bay validatewarns when a tag inheadscale_acl_policy.tagOwnershas more than one owner, an owner that is agroup:or anautogroup:*, or the owner*. The warning names the tag and says: every owner can enrol a node with that tag and so gets every grant of the tag. It never fails the run. Seedocs/tailnet-naming.md.- New variable
docker_prune_volumes(see Fixed).
Changed
- Headscale admin API: new variable
headscale_admin_api_allowed_ips(access_gateway defaults, default["127.0.0.1/32", "::1/128"]). Theheadscale-adminrouter (/api,/swagger) no longer usesvpn-chain. It usespublic-chain, the headers and compression of the main Headscale router, plus its own IP allowlistheadscale-admin-allowfrom the new variable. In 2.19.0 and 2.19.1 every tailnet node reached the admin API, also the nodes of an external tenant, and an API key was the only guard there. Now every client outside the list gets 403, tailnet nodes included. Nothing in Bay needs the path:bay gatewayand registration usedocker execon the control host, and the identity sidecar uses the docker network.headscale_admin_api_public: truestill opens the API to everyone and adds no label. - Headscale admin API: the deploy stops when
headscale_admin_api_allowed_ipsis empty, is not a list, or holds a value that is not an IP or a CIDR. Traefik drops a router whose allowlist it cannot parse. - Headscale admin API, fail-safe: while the API is closed, the main Headscale router leaves out the admin paths (
Host(`<domain>`) && !PathPrefix(`/api`) && !PathPrefix(`/swagger`)). If theheadscale-adminrouter is ever dropped,/apiand/swaggerget 404. Before, they fell through to the public main router. Withheadscale_admin_api_public: truethe main rule staysHost(`<domain>`). The admin router keeps priority 1000, which is still above the length of the longer main rule. - Headscale:
config.yamlnow hastrusted_proxieswhen Traefik fronts Headscale. New variableheadscale_trusted_proxies(roleheadscale). Its default is computed on each run: the gateway oftraefik_docker_network, read withdocker network inspect, as a/32(a/128for an IPv6 gateway). Traefik runs with host networking, so it reaches the container from that address. Before, Headscale logged the gateway for every request, not the client. Only the gateway is trusted, not the subnet, so another container on the network cannot set the headers. The key and its rules are those of Headscale 0.29.7: prefixes only,0.0.0.0/0and::/0refused. The role stops on a bare IP or a/0, because Headscale would crash-loop.[]turns it off. The role creates the docker network before it reads it, so a fresh host gets the key on its first deploy. - Headscale: both headscale routers now remove a
True-Client-IPheader sent by the client (new middlewareheadscale-client-ip). Headscale reads that header beforeX-Real-IPandX-Forwarded-For, and Traefik passes it through, so withtrusted_proxiesa client could otherwise choose the IP Headscale logs. Traefik already replacesX-Real-IPandX-Forwarded-For. - Headscale admin API, docs:
docs/tailnet-ingress.mdrenames the section to "The Headscale admin API is closed" and shows how to open the API for one device.docs/access-gateways.md,docs/multi-region.md,docs/tailnet-naming.md, theheadscalerole defaults header and thebay gateway apikeyhints say the same. The 2.19.0 text that said the API answers over the tailnet is gone.
Fixed
- Traefik: at each start, Traefik logged
middleware "vpn-only@docker" does not existonce for each tailnet route (<route>-tailnet@file). The route file is loaded by the file provider, and the file provider was ready before the docker provider had read thevpn-onlylabel on the traefik container. The routes worked a second later. The route file now defines its own copy of the allowlist, and each route names it asvpn-only@file. The docker-label routers keepvpn-onlyandvpn-chainas before. Both copies readvpn_allowed_ipswith the same default, and a test renders both and compares the ranges. - Access gateway: the task that adds the Headscale tailnet range to
vpn_allowed_ipsis now taggedalways. Before,bay deploy <env> --tags traefikskipped it, so the route file could get a shorter list than the label. It only sets a fact. - Headscale ACL: a dry run (
-- --check) now validates a newheadscale_acl_policyand fails on an invalid one. Before, check mode skipped "Check headscale container is running", then printed "SKIPPING ACL validation, the headscale container is not running", which was false, and the policy was first validated in the real run. The container probe, the staging write,headscale policy checkand the removal of the staged file now run in check mode too (check_mode: false). They only read, or write the hidden staged file that the dry run removes again, also when the check fails. The install of the policy stays skipped in check mode. On a host where the container does not run, the dry run stages nothing and the warning is now true. - Debug agent: the
debugbotaccount kept its membership of thedockergroup on a host that an older Bay version provisioned. The group is root-equivalent. The role only ever added groups (append: true), so it never took the membership away. Withdebug_agent_docker_access: falsethe role now readsid -nGand runsgpasswd -d <user> dockerwhen the account is a member. The step is idempotent. The primary group and the other groups stay. It runs in check mode as a read only and skips the removal.docs/debug-agent.mdsays so. - Cron jobs: the nightly or weekly
docker system prune -af --volumesdeleted every volume that no container used at that moment. The data volume of a service whose deploy failed shortly before is such a volume. New variabledocker_prune_volumes(defaultfalse). Without it the job runsdocker system prune -af. The cost: an anonymous volume of a removed container stays on disk, a small leak. Setdocker_prune_volumes: trueto get the old command back. The comments inroles/cronjobsand the docs that quoted--volumessay the same. - Dry runs:
bay provision <env> --tags outbound_monitor -- --checkfailed on a host that had nobay-reboot-alert.timeryet, withCould not find the requested service. Check mode writes no unit file, so the systemd module could not find the unit. The enable and the disable tasks of the three timers ofoutbound_monitor(outbound check, disk alert, reboot alert) now skip in check mode when the task that writes their unit file reportschanged. The same flaw was in the backup timers and the maintenance timers (backup), the trigger watchdog timer (git_deploy) and the CrowdSec data refresh timer (crowdsec). They follow the same rule. A real run is not affected. - Tailnet identity: the
tailnet-identitysidecar ignored SIGTERM. The server is PID 1 in its container, Linux does not apply the default SIGTERM action to PID 1, sodocker stopwaited 30 s and then killed it. The server now handles SIGTERM and SIGINT, stops listening and exits 0. A test starts it as PID 1 of a fresh PID namespace and checks that SIGTERM ends it in under 3 s. - Build server: the task
Apply memory limit to the bay buildx container(andApply memory limit to builder containerfor local builds) hadchanged_when: rc == 0.docker updateexits 0 also when nothing moves, so the task saidchangedon every deploy. It now reads memory and swap of the container before and after, and reportschangedonly when they differ. See Upgrade notes for the part that is not a bug.
Upgrade notes
- Apply on existing hosts, in this order. Run each command against the fleet of that env.
- Check your own Headscale API clients first (see "Headscale admin API" below). A client that calls
/api/...over the tailnet gets 403 after step 3. bay provision <env> --tags agent-debugon each host withdebug_agent_enabled: true. No container is touched.bay deploy <env> --tags headscaleon the env that runs Headscale. Recreatesheadscaleonce; every node reconnects once.bay deploy <env> --tags tailnet_identityon the control and ingress env. Recreatestailnet-identity(new image); requests withidentity_injectfail for a few seconds.bay deploy <env> --tags traefikon the ingress env. Rewritesdynamic/tailnet-proxies.yml; no container is recreated.bay deploy <env> --tags cronjobson each env. Rewrites one cron line; no container is touched.- Optional:
bay deploy <env> --tags git_deployon each env with builds, for the new buildkit memory reporting. No container is recreated.
No other container is recreated by this release. The dry-run and validator changes need no command.
- Check your own Headscale API clients first (see "Headscale admin API" below). A client that calls
- Headscale (admin API allowlist,
trusted_proxies,True-Client-IPstrip): runbay deploy <env> --tags headscaleagainst the fleet that runs Headscale. The router labels change, so theheadscalecontainer is recreated once and every node reconnects once. This happens also withheadscale_admin_api_public: true, because the main router gets theheadscale-client-ipmiddleware. The same run renderstrusted_proxiesintoconfig.yamlbefore the recreate, so there is no second restart. No other container is recreated. A plainbay upof the control box env also re...
v2.19.1
Fixed
bay deploy --log <file>andbay provision --log <file>crashed at start withModuleNotFoundError: No module named 'click'. This was so from 2.19.0 on a machine installed withuv tool install. Newer Typer carries Click inside itself and does not install a top-levelclick. The tests did not see it, because the test environment hasclickthrough its dev dependencies. The crash came before the run began, so no box was touched and nothing was deployed or provisioned. The same import sat in an error path ofbay up --jsonandbay rollback --json, which hit it only on an unexpected error. Bay now gets the exit and error classes through Typer and no longer importsclick. A new test bansimport clickinsrc/and runs the--logsummary withclickunimportable.- Without
--log,bay deployandbay provisionwere not affected.
v2.19.0
This release adds three things. Hosts alert when a reboot stays pending. The Headscale admin API is closed to the internet, and a box in another region registers through the control host over SSH. bay host upgrade upgrades the OS packages of one box, and bay deploy --tags tailnet_identity rotates the sidecar key alone.
Added
- Reboot alert: new alert
host.reboot_pending(levelwarn, on by default). A daily timer,bay-reboot-alert.timer, checks/var/run/reboot-required.unattended-upgradesinstalls a kernel update but never reboots, so a host could run for months on an old kernel with no word to anyone. When the file is older than the threshold, the alert names the host, the days pending, the first five packages from/var/run/reboot-required.pkgs, and the running kernel against the newest installed one. - Reboot alert: new
outbound_monitorvariables:reboot_alert_enabled(defaulttrue),reboot_alert_threshold_days(default7) andreboot_alert_repeat_days(default7). The first alert comes when the file reaches the threshold. After that, one reminder per repeat interval while it stays pending. The age counts from the first time the probe saw the file, so a later package that rewrites the file does not restart the clock. The state (first_seenandlast_alert) is kept in/var/lib/bay-outbound-check/reboot-alert.state, and it is cleared when the file is gone. Setreboot_alert_enabled: falseto stop and mask the timer. - Reboot alert, docs:
docs/alerting.mdhas a new section "Pending reboot alert". - Headscale admin API: new variable
headscale_admin_api_public(defaultfalse). A second Traefik router on the Headscale host name takesPathPrefix(/api)andPathPrefix(/swagger)and sends them throughvpn-chain, the chain of everyaccess: vpnservice. The REST API and the Swagger page answer only tovpn_allowed_ips(loopback and the tailnet range). From the internet they answer 403. The router has priority 1000, above the main Headscale router, whose priority is the length of its rule. Undertraefik_split_entrypointsit also bindswebsecure_tailnet. The tailscale protocol paths stay public:/ts2021,/key,/register/...,/auth/...,/oidc/callback,/verify,/derp,/bootstrap-dns,/machine/ping-response,/apple,/windows,/health,/version. A test checks this list against Headscale 0.29.7.truekeeps the old behaviour and adds no label. - Headscale admin API:
bay gateway apikey listshows the ID, prefix, expiry and creation time of every API key.bay gateway apikey expire <prefix-or-id>expires one key. It takes the prefix aslistprints it (hskey-api-<prefix>-***), the bare prefix or the ID. Both run the Headscale CLI in the container on the control host, like every otherbay gatewayverb. - Headscale admin API: new variable
headscale_register_via_api(defaultfalse, roletailscale_register).truekeeps the old API registration path for a fleet whose deploy machine cannot reach the control host over SSH. - Host upgrade:
bay host upgrade <env> --box <box> [--reboot] [--json] [--log <file>]upgrades the OS packages of one box. It runsapt-get dist-upgrade(keeping local config files) in a detached systemd unit,bay-host-upgrade, so a dropped SSH session does not stop it, thendpkg --configure -a(with one repair pass when apt fails, see below). It checks Docker, every container that ran before (healthy where it has a healthcheck), nftables and the Bay tables, Docker's chains, one published port, CrowdSec, the bouncer and tailscale. A container that isrestarting, or whose restart count grew during the wait, fails the run (container_restarts), and its last log lines are printed: after a cold boot, Traefik looped until tailscaled had its address. Where tailscale is installed,tailscale ip -4must give an address. A failed check fails the run with its name.--rebootreboots when a reboot is needed and waits for the box. Without it, the command says that a reboot is pending.-- --checkchanges nothing on the box. It writes no receipt and no lock record. Seedocs/host-upgrade.md. - Host upgrade: the upgrade refuses to start while a
bay upholds a claim on the box env (env_busy, exit 43), while the box holds the deploy lock (box_locked, exit 43), or while an earlier upgrade unit still runs (upgrade_running, exit 43). It holds the deploy lock while it runs and touches it at each poll round. - Host upgrade: when
apt-get dist-upgradeexits with an error, the run does not fail yet. It runs one repair pass: it waits up tohost_upgrade_crowdsec_wait(60 s) untilcscli lapi statusanswers (when CrowdSec is installed), runsdpkg --configure -a, thenapt-get -f install -yand one moredist-upgradewith the same flags, in the same detached unit and the same log. The apt step fails only when that pass fails ordpkg --auditstill lists a package. The result hasapt_repair,apt_first_rc,apt_rcanddpkg_audit. A live trial needed it: the postinst ofcrowdsec-firewall-bouncer-nftablesstarted the bouncer while CrowdSec restarted, and apt exited 100 on an upgrade of 78 packages. - Host upgrade: once the apt unit started, the firewall repair and the checks always run, also when the apt step failed. The result always has
checksandcontainers.after. The run still fails, andfailed_tasknames the apt step. The deploy lock is released on every path. - Host upgrade:
--rebootnever reboots when the apt step failed ordpkg --auditis not empty.reboot_notein the result says why. bay deploy,bay provisionandbay host upgradewith--log <file>and no--jsonend with a short summary on the terminal: the result word (okorFAILED), the PLAY RECAP counts per host (ok, changed, failed, and unreachable when not 0) and the log path.bay host upgradeadds its key facts: packages upgraded, a repair pass, a pending or done reboot, the checks. Before, the terminal showed only thefleet:line, also for a failed run.bay updoes not change.- Host upgrade: an upgrade of the
nftablespackage flushes the whole ruleset (ExecStop=nft flush ruleset), Docker's rules included. When the package was upgraded, whennftables.servicestarted again, or when a Docker chain is missing, the upgrade loads/etc/nftables.confagain, then restarts Docker, then the CrowdSec bouncer.
Changed
- Headscale admin API:
tailscale_registerno longer calls the public REST API for a box outside the control region. It delegates to the control host, the first host of the inventory group named byheadscale_control_region, and runsdocker exec headscale headscale ...there as root: create the user, list users as JSON, make a 5-minute pre-auth key as JSON. The box then runstailscale upwith the key. No API key is needed. The deploy stops with a message that says what to do when the control host is not in the inventory, does not answer over SSH, or does not run Headscale. In check mode nothing runs, as before. - Headscale admin API: on the API path (
headscale_register_via_api: true), a 403 from Traefik stops the deploy with a message that namesheadscale_admin_api_public. - Headscale admin API:
bay gateway apikeysays that Bay does not need the key. The next steps ofbay region addno longer nameheadscale_api_key; they say to check SSH access to the control host. Docs:docs/tailnet-ingress.mdhas a new section "The Headscale admin API is tailnet-only".docs/multi-region.mdhas "Registration and the admin API" and new troubleshooting lines.docs/access-gateways.mdand theheadscalerole defaults header say the same. - Headscale admin API, limit: a deploy machine with no SSH to the control host cannot enrol a box of another region, unless the fleet opts back in with
headscale_register_via_api: true,headscale_admin_api_public: trueandheadscale_api_keyin the vault. - Host upgrade:
deploy.ymlandprovision.ymlstart with a guard: they stop before any role touches the box while a host upgrade runs on it (a fresh lock of kindhost_upgrade, or the unitbay-host-upgradeactive). Before, a deploy died on the apt lock half way. - Host upgrade: the
deploy_stacklock message says when a host upgrade holds the lock. - Tailnet identity:
bay deploy <env> --tags tailnet_identitynow rotates the sidecar's Headscale API key alone. When the run does not includedeploy_stack, the role reconciles only thetailnet-identitycontainer on the control host (container_lifecycle_only, no orphan removal, no receipt), as the headscale role does for its container. The env file is in the sidecar's config hash, so a new key recreates it and an unchanged key is a no-op. No other container is touched. The role then waits until/healthzanswers, and with theapisource it checks the key against Headscale from inside the sidecar and fails the run unless the answer is200. Before, a rotation needed--tags tailnet_identity,deploy_stack, which reconciles every container on the box. Steps:docs/tailnet-ingress.md, "Rotating the sidecar's API key". bay deployandbay provisiontake--log <file>, with the meaning it has forbay up. Progress and the Ansible output go to the file, and the terminal gets only the end summary. It works before or after the environment name. All ofbay up,bay plan,bay remove,bay host upgrade,bay deployandbay provisionnow use one implementation of the redirect (bay_cli.commands.output), and one help text.
Fixed
bay deployandbay upprinted "This deploy changed no containers" after a run that had recreated containers. The reconciler runs once per pass, and a run can have several passes on one host: the narrow headscale, tailnet-identity and webhook passes, thendeploy_stack. Each pass wrote the same report file, so the last pass replaced the others. Each pass now writes its own file (`..json...
v2.18.1
A HALF project stays HALF until every box it waits for got a deploy. A later run that skips such a box no longer clears it.
Fixed
- A
bay upthat failed on one box left the project HALF, but the next run cleared it even when that run did not deploy the failed box. The failed run commits its compile, so the next plan of the project often has a smaller scope: no build change (no build server), no route change (no ingress box), no move (no old box), no planned step of another project. That run passed on the project's own box and wroteresult: ok. The fleet then said the project was fine, while the failed box never got the deploy.bay rollbackandbay up --take-overcleared it the same way. - A run with no host map at receipt time took the env-wide result for every box of the box env, also for the boxes its scope skipped. Only the boxes of the scope get a result now.
- With
traefik_split_entrypoints: true, a cold boot could leave Traefik down. Traefik bindswebsecure_tailnetto the tailnet address, and tailscaled has no address until it reaches Headscale. Traefik exited withlisten tcp <ip>:443: bind: cannot assign requested address, which stopped the public 80 and 443 too. On the Headscale host it was a deadlock, because Traefik serves Headscale. Thetraefikrole now setsnet.ipv4.ip_nonlocal_bind=1(live, and in/etc/sysctl.d/99-bay-traefik-nonlocal-bind.conf) before it writes the Traefik config. The listener still binds only the tailnet address. The variabletraefik_nonlocal_bindfollowstraefik_split_entrypoints; set it to false to opt out (the file is removed, the live value stays until the next reboot). IPv4 only. - The boot unit
bay-infra-boot.service(bay-infra-ensure.sh) removed Traefik when port 80 was not bound 30s after boot, and made a new one from the infra compose file. The removed container came from the Bay reconciler (labelcom.bay.config-hash). The compose copy has no such label and may differ from the Bay spec, so the next rig deploy planned a Traefik recreate ("the container has no config-hash label"). The recreate could not help anyway when the cause was a missing tailnet address. The script now never removes or replaces a container with that label. A labelled container that is dead or restarting, or Traefik with port 80 unbound, gets onedocker restart, then a WARNING in the journal that names the likely cause. An exited container stays stopped, so a deliberate stop survives a boot. Compose create stays for a container that does not exist, and the log says the nextbay deploy --rig <env>recreates it under the reconciler. A container without the label keeps the old behaviour.
Changed
- A
pendingorfailedlock record has a new keyhalf_boxes: the boxes that still need a deploy.bay upwrites it with the pending record, before it touches a box. After the deploy it keeps the boxes that failed or wrote no receipt from this deploy. A one-project run counts every box of its scope; a run of the whole environment counts the project's own box and the old box of its move. - A later run removes a box from the list only when its scope covered the box and the box passed. The record stays
failedwhile one box is left. Such a run exits 0 and adds a note:<project> <env> stays HALF: this run did not deploy box <box>, ... Run bay up <env> --project <project> --all-boxes.pinnedrows of a HALF record carryhalf_boxes. bay showends the HALF reason with the boxes and the command that covers them (--all-boxeswhen a box other than the project's own box is left).bay show --jsonhasenvs[].half_boxesandenvs[].next.bay planadds a note when its scope skips a box that the project still waits for.- Docs:
docs/plan.mdhas a new section "A HALF record names its boxes", and theHALFrow of thebay showtable names the boxes.
Upgrade notes
- Every machine that deploys a fleet needs 2.18.1. An older CLI refuses
half_boxesin a lock as an unknown key, so its compile, plan and deploy stop with a lock schema error.bay.compat.jsonstops it earlier: the first 2.18.1bay upwritesdeployed_with: 2.18.1in the same claim commit that first writeshalf_boxes, so a 2.16.0 to 2.18.0 machine stops withbay_older(exit 45) before it writes anything.lock_versionstays 2. - A record that is HALF before the upgrade has no list, so it waits for every box of its box env. A one-project run no longer clears it. Run
bay up <env>in the fleet (the whole environment), orbay up <env> --project <name> --all-boxes, once per such project.bay shownames the command. - Split-entrypoint hosts: run
bay deploy <env> --tags traefikon every such host. The sysctl takes effect at once and recreates no container. This tag restarts Traefik only when the rendered Traefik config changes, and this release does not change it, so Traefik keeps running. The fix matters at the next cold boot. - Every rig host: run
bay deploy <env> --tags boot_safety(--tagsturns the rig roles on) to install the new/usr/local/bin/bay-infra-ensure.sh. Theboot_safetyrole runs fromdeploy.yml(rig roles only), not fromprovision.yml. It only rewrites the script. The unit file does not change and no container restarts.
v2.18.0
Headscale hardening: random IP allocation for new nodes, a consistent and checked DB backup, a locked-down container, and tighter file modes.
Changed
- New role variable
headscale_ip_allocationfeedsprefixes.allocationin the Headscale config. The default israndom. The value must berandomorsequential; theheadscalerole stops on any other value. Reason: ACLhosts:aliases and grants point at IPs, and withsequentialHeadscale gives the IP of a deleted node, and every grant on it, to the next node that joins. tailscale_registernow checks a host right after it registers: when its tailnet IP is notheadscale_server_tailnet_ip, the deploy stops and the message names the IP to set. Underrandom, a fresh control host no longer gets100.64.0.1, the default of that variable. A host that is already registered is never stopped by this check.- The Headscale backup no longer copies the live database with
docker cp. The DB runs in WAL mode, and a file copy can catchdb.sqliteanddb.sqlite-walat different moments. A root helper (/usr/local/sbin/bay-headscale-db-snapshot) runssqlite3 .backupon the live DB, runsPRAGMA integrity_checkon the copy, and fails unless it printsok. The snapshot and the noise and DERP private keys go to restic as before (headscale.tar, sameheadscale/layout). The live-waland-shmfiles are no longer in the archive. The temp copy is 0600 in the root-only/var/lib/bay-headscale-backupand is removed on every exit. A failed check stores no snapshot and sends the usualbackup.failedalert. The backup unit still runs as the app user and calls the helper through one sudoers rule (/etc/sudoers.d/bay-headscale-backup) that allows no arguments. Restic, retention, the size check, maintenance and alerts are unchanged.backup_headscale_source_pathis gone;backup_headscale_data_dir,backup_headscale_snapshot_binandbackup_headscale_snapshot_dirare new. - The Headscale container runs with a read-only root filesystem, a tmpfs for its unix socket (
/var/run/headscale, sodocker exec headscale headscale ...keeps working), the config folder mounted:ro,cap_drop: [ALL]pluscap_add: [DAC_OVERRIDE], andno-new-privileges. It still runs as root. DAC_OVERRIDE is the one capability it needs: the config and data folders belong to the app user, so root without it cannot readconfig.yamlor write the data folder (checked againstheadscale/headscale:0.29.7: with every capability dropped it does not find its config; with DAC_READ_SEARCH alone it cannot write its noise key). The reconciler learned theread_only,tmpfsandcap_addspec keys; a spec without them hashes as before, so no other container is recreated. - File modes:
/opt/headscale/datais 0700 and/opt/headscale/config0750 (both were 0755).db.sqlite, its-waland-shmfiles and every DB copy in the data folder or in/opt/headscale/backupsbecome 0600 (Headscale created them 0644). The role fixes the modes on every run and deletes nothing. SQLite creates new-waland-shmfiles with the mode ofdb.sqlite, so they stay 0600 after a restart. - The
headscalerole refusestailnet_identity_source: sqlite: the sidecar runs as a non-root user and can no longer read the DB. The defaultapisource is unaffected. - Docs:
docs/backups.mddescribes the snapshot helper, moves the old-waland-shmfiles away before a restore, and has a new "Restore test (Headscale)" section.docs/access-gateways.mdsays how to setheadscale_server_tailnet_ipunder random allocation.
Upgrade notes
- Run
bay deploy <env> --tags headscale,backupagainst the fleet that runs Headscale (both roles are in the deploy playbook, not in provision).headscalesets the file modes and recreates the container with the hardened spec.backupinstalls thesqlite3package on the host, the helper, its folder and the sudoers rule, and renders the new backup script. A fullbay deploy --rig <env>does the same. - The Headscale container restarts once (the spec hash changes). Nodes reconnect within seconds. Nothing else is recreated.
- A plain
bay upon that fleet also recreates the Headscale container, becausebay upbuilds the same spec.bay planshows it as aheadscalerecreate step. So the firstbay upafter the upgrade is not a no-op on the control host. Run the--tags headscale,backupdeploy first, so the modes, the helper and the container change in one step. - IP allocation: existing nodes keep their addresses; only nodes that join after the upgrade get a random free IP. Headscale reads every assigned IP from its database as used, and its startup backfill only fills a node that has no IP. To keep the old behaviour, set
headscale_ip_allocation: sequentialin group_vars before the deploy. - A fleet that sets
tailnet_identity_source: sqlitemust switch toapifirst, or theheadscalerole stops. - The first backup after the upgrade is a new-format archive. Restore older snapshots as before;
docs/backups.mdcovers both. - An old copy such as
/opt/headscale/backups/db.sqlite.pre-v0292-*becomes 0600 but stays. Delete it by hand when you no longer need it.
v2.17.3
A test fix. Nothing changes for a fleet or a box.
Fixed
tests/test_log_retention_integration.pyfailed now and then withsubprocess.TimeoutExpiredwhen the Docker daemon of the test machine was busy. Its Docker calls had limits of 5 to 30 seconds. They are now 60 to 120 seconds. This test stopped one release gate.
Upgrade notes
- None needed.
v2.17.2
A forced push to a local build no longer fails with no reason. [build] repo is checked for its shape. A service with no token no longer names a credential helper that does not exist.
Fixed
- A
localbuild with the same repo URL and a replaced history (a forced push with an unrelated root commit) rangit pull, which stopped with exit 128 (fatal: Need to specify how to reconcile divergent branches). The failure was the generic "Webhook deploy failed" with no reason, and the circuit breaker counted it. The only way out was a manual full-build mark on the box. Alocalbuild now updates its checkout as aremotebuild does:git fetchof the deploy branch, thengit reset --hard FETCH_HEAD. The checkout is build-only, so any history moves it. A fetch that fails ends in the existing loud path ("Repo fetch failed", onebuild.failedalert with the likely cause), not in the ERR trap. rebuild.shnow asks after the reset whether the old head (the checkout's HEAD before the fetch, as the service saw it) is an ancestor of the new head (git merge-base --is-ancestor). When it is not, or when git cannot find the old commit, the history was rewritten: the build is a full build, with the mark of a new checkout (.git/bay-full-build/<container>). No config-only tag and no no-input-change tag use the old commit. The log sayshistory of <container> was rewritten (<old12> is not in <new12>), full build. When the old head is an ancestor, nothing changes. Aremotebuild did not fail on this, but it did not look either: it built because nothing used the old commit (the receiver writes no no-input-change marker for a force push, and the old commit stays in the checkout forgit diff). A trigger file with that marker, after a rewrite that kept the same tree, was a tag-only push of the old image. It now takes the same explicit full-build path.- A service with no token had
GIT_ASKPASSset to a helper file that Ansible writes only for a service with a token. A publichttps://repo that answered 401 or 404 made git run the missing file, and the failure reason readfatal: cannot exec '<...>/.git-askpass-<name>': No such file or directory.rebuild.shnow setsGIT_ASKPASSonly for a service with a token, in both strategies. A service with no token keepsGIT_TERMINAL_PROMPT=0, so git fails at once with its own message. The lines of a service with a token are unchanged.
Changed
bay toml validate,bay planandbay upnow refuse a[build] repothat is not a git URL or an absolute path. Accepted:https://,http://,ssh://,git://andfile://URLs with a host and a path,user@host:path, and an absolute local path. Refused: an empty value, a space inside, a bare word, a relative path. Message:build.repo: "<value>" is not a git URL or an absolute path; use for example git@github.com:org/repo.git or https://github.com/org/repo.git.- A URL with a password or a token inside (
https://user:secret@host/...,https://token@host/...) is refused too. The message shows the URL redacted and says to give the token inrepo_tokensinbay.fleet.tomlinstead. The secret is never printed. Thebay.lockrepois not checked: it is written by the CLI from the origin of a checkout, and a check there could stop abay upafter it touched a box. - Docs:
docs/build-pipeline.mdreplaces the "known limit" of a local build with a replaced history by what now happens (no manual step), and explains the ancestor rule and the no-token case.docs/bay-toml.mdnames the accepted forms at therepokey.
Upgrade notes
- The build script fixes reach a box with the next
bay up(orbay deploy), which rendersrebuild.shagain. The upgrade recreates no container:rebuild.shis a plain file that no container hash or handler reads. The container config hash, thecom.bay.receiver-hash(the receiver files) and thecom.bay.receiver-config-hash(the receiver config and the image map) read none of the files this release changes. - The next push after the upgrade is a normal one. A
localbuild moves fromgit pullto fetch and reset, with the same result for a push that adds commits. - Run
bay toml validatein a project beforebay up. A[build] repothat is a relative path or has a space in it was never usable by git on the box, andbay plannow stops on it. A token in a repo URL moves torepo_tokensinbay.fleet.toml.
v2.17.1
A changed [build] repo no longer leaves the box on the old repo. The next build uses the new repo, or fails loud.
Fixed
- After a change of
[build] repoandbay up, the box kept the old repo. Aremotebuild reused its checkout, whoseoriginstill named the old repo, so the build server went on building the old code with no error (with a token, only a fullbay deploy --tags git_deployresetorigin; with an SSH URL, nothing did). Alocalbuild keys its checkout folder on repo and branch, so the new URL named a folder that only a full deploy clones: the next push failed at the repo lock with a bare "Webhook deploy failed". A changed deploy branch hit the samelocalfailure. rebuild.shnow holds the expected repo of each container (EXPECTED_ORIGIN) and the credentials of a fetch. Before the fetch it clones a missing checkout (into a temp folder, moved into place when complete), and it repoints a checkout whoseoriginnames another repo: firstgit ls-remoteof the new repo and branch, thenoriginis set, and the fetch andreset --hard FETCH_HEADmove the checkout to the new head, whatever its history. Log line:repo of <container> changed from <old> to <new>, checkout repointed. The repos are compared in the CLI's normal form (host/owner/repo, lower case, no user, password, port or.git), so the same repo in another form is no change.- The first build from a new or repointed checkout is a full build: the commit of the old repo decides no config-only or no-input-change tag, and a
localbuild fetches and resets instead ofgit pull, which refuses an unrelated history. A mark in.git/bay-full-build/keeps it so until a run has built or found the image of the head. - A repo that the box cannot clone, check or fetch ends with one
build.failedalert, "Repo check failed (N/MAX)" (or clone, or fetch). It names the container, the repo, the branch and the likely cause: the token cannot read it, the deploy key is not on it (GitHub takes a key on one repo only), the box has no deploy key yet, an https repo is private or missing, the branch is missing, or the host key is not pinned. Nothing is built, and the checkout and itsoriginstay as they were. The circuit breaker counts it like any failed build. Before, a failed fetch of aremotebuild fell to the ERR trap with no reason, and the HTTP 403 hint after it never ran. - A token in a repo URL is redacted (
***) in every log line, alert, state file and plan note. bay build resetwrote an invalid state file. The JSON went through the remote shell inside double quotes, so the shell removed the quotes ({version: 1, consecutive_failures: 0, ...}).bay build statusthen failed to parse it and printed "No build state found", until the next build rewrote the file. The state now travels as base64, so no shell can change it. The file has the same shape asrebuild.shwrites (last_failureisnull, not a block of empty strings), the owner and group of the state directory (the app user and the build group) and mode 0664, asrebuild.shdoes. Before, it was root-owned with mode 0644:rebuild.shcould still replace it (it needs write access to the directory, not to the file), but the build group could not write it. The command no longer creates a missing state directory, which would have been root-owned and would have lockedrebuild.shout.bay build statusnames a state file that is not valid JSON: "the state file of is not valid JSON; runbay build reset <service> --env <env>or wait for the next build to rewrite it". The other services still show. Before, one bad file made the whole listing say "No build state found".bay up(--tags deploy_stack) now also writes the GIT_ASKPASS helper of eachlocalbuild with a token. The helper path holds the checkout slug, so a new repo or branch needed a helper that only a full deploy wrote.
Changed
bay planandbay upadd a note when the build repo of a container changes:<container>: the build repo changes from <old> to <new>. The box builds from the new repo from the next push. Add the box's deploy key to the new repo and create its push webhook first.For aremotebuild the second sentence says: the build server builds from the new repo from the next push or trigger file, and the deployment box recreates the container when it gets the pull signal. With a token the note names its vault key. Thechanged: buildstep stays. Notes are not part of the plan id. Aregistrybuild gets no note: Bay does not clone its repo.- The plan after
bay toml move(andbay adopt) shows onebay-webhookrecreate step for each box of the box environment that runs a webhook receiver, not one step in all.bay updeploys the whole box environment, and each receiver holds the config (four such boxes give four steps). The docs (docs/plan.md,docs/bay-toml.md,docs/layout-scenarios.md) and the output and JSONnotesof the verb now say: expect onlybay-webhookrecreate steps, one per box with a webhook receiver. Any other step: stop. - A missing
localcheckout is now cloned by the build script itself, with the key or token the box has. The SSH deploy key of a newlocalbuild app still comes only frombay deploy <env>with no--tags. - Docs:
docs/build-pipeline.mdhas a new section "When the repo URL changes" (what the box does, the loud failure, the operator checklist for the deploy key, the webhook, the old hook and the old checkout folder, the circuit breaker, rollback, branch changes).docs/bay-toml.mdlinks it from therepokey. The observability contract has row 25 for the new exit. The section also says who builds and who recreates with the remote strategy, how to check the switch without waiting for a recreate (the journal line andgit remote get-url origin), why a repointed checkout may skip the image build (the image of the head is already in the registry), and that a same-URL history replacement needs no step on a remote build. A local build still fails there atgit pull: the section gives the manual step.
Upgrade notes
- The fix reaches a box with the next
bay up(orbay deploy), which rendersrebuild.shagain. No step is needed, and the upgrade recreates no container:rebuild.shis a plain file that no container hash or handler reads, and the receiver config and image map are unchanged. On a box with alocalbuild that uses a token,bay upwrites its GIT_ASKPASS helper file, with the content a full deploy already wrote. - A box that already lost a
localcheckout this way clones it on the next push. - When you change a repo URL: give the box read access to the new repo and create its push webhook before
bay up, then push or write a trigger file. Remove the old repo's hook and deploy key afterwards. The oldlocalcheckout folder stays; remove it by hand. See the checklist indocs/build-pipeline.md.