Skip to content

Releases

v2.22.0

Choose a tag to compare

@AltanS AltanS released this 13 Nov 10:36

This release lets an egress container load the fleet's own public websites.

Added

  • bay.toml: new key egress_fleet = "web" on a [services.<name>] with egress = "public". The egress container may then also open new tcp connections to ports 80 and 443 of the public address of every fleet box, its own box too. The addresses are the ansible_host (or the inventory name when it is an address) and the netplan_address of every inventory host, the same list that feeds the blocked set. Every other port (SSH too), UDP, IPv6, the tailnet, the private ranges, the gateway and the other containers stay closed. The validator refuses the key on a service without egress, any value but "web", and the key at the top level or in [deploy.<env>]. It is never inherited.
  • Compile: the egress entry in services.yml gets fleet: web. The key is absent when unset, so every compiled services.yml stays the same, byte for byte.
  • Firewall (roles/container_lifecycle): /etc/nftables.d/bay-egress.conf gets two accept rules for each flagged container. Chain from_egress opens the other boxes (forward hook). Chain egress_to_host opens this box's own address (input hook). Both match the container's bridge and its own address, so a client on the same egress network gains nothing. Both run before every drop. The rule for other boxes never sends the request out through the tailnet interface (tailscale*), where it would carry a tailnet address that vpn routes trust. Only a single public IPv4 address opens. A host name, a network, and a tailnet, private or reserved address are skipped. An address in bay_egress_blocked_extra stays closed: the block wins. Without the key the file is the same, byte for byte.
  • bay plan: adding egress_fleet gives a second container step with risk shared. Its reason names the container and says that it may open tcp 80 and 443 on the public address of every fleet box. So the plan needs bay approve. Removing the key gives the same step with risk safe. docs/plan.md has both rows in the risk table.
  • Docs: docs/bay-toml.md has a new section "egress_fleet: the fleet's own websites": what opens, what stays closed, two side effects to check first, and the verify commands. docs/services.md describes fleet: web.

Upgrade notes

  • Nothing changes until a service sets the key. The compiled services.yml and the egress firewall file stay the same, so the upgrade reloads no firewall and recreates no container.
  • To turn it on, add egress_fleet = "web" to the egress service and run bay plan, then bay approve <plan-id> --reason "<why>", then bay up. The plan shows changed: egress and the step egress_fleet = "web" added: <container> may open tcp 80 and 443 on the public address of every fleet box, its own box too; bay up opens the box firewall for it, without a recreate. bay up loads the new rules. The container is not recreated.
  • Check two side effects first. Another box sees the requests from this box's public address, and the CrowdSec allowlist of the fleet's own addresses never bans it. Keep box addresses out of vpn_allowed_ips and every app allowlist. On its own box, Traefik sees the container's private address (in bay_egress_subnet_pool) as the client. vpn routes stay 403, but an app that trusts private client addresses would serve the crawler as an internal user.
  • Do not set the key on a box that routes fleet addresses through another overlay or VPN interface (one not named tailscale*). The request would then arrive from an address that vpn routes may trust.
  • Behind a cloud 1:1 NAT (the public address is not on the box's own interface), the crawler cannot reach its own box. The rules fail closed there. The other boxes still answer.
  • A new box opens at the next bay up on the egress container's box, like the blocked set.

v2.21.0

Choose a tag to compare

@AltanS AltanS released this 13 Nov 00:11

This release gives the Zot registry longer request time limits and adds pull-only registry accounts. It also stops a verbose run from printing the registry admin password.

Added

  • Zot: new variables zot_http_read_timeout and zot_http_write_timeout (role zot, default "30m"). They render http.readTimeout and http.writeTimeout in config.json. Zot's own default is 60s, an absolute limit for the whole request or response body. A layer upload that still streamed at 60s was cut: zot logged i/o timeout and the proxy answered 502. A large pull on a slow link was cut the same way. "" renders nothing (zot's 60s). "0s" turns a limit off. The deploy refuses a value that is not a Go duration. The keys need zot v2.1.17 or later; the role pins v2.1.22.
  • Zot: new variable zot_pull_users (role zot, default []), a list of username and password. Each entry is an htpasswd account that can only read (pull) every repository. The docker_registries entry for zot_domain stays the only account that can push and delete. With at least one entry, config.json gets http.accessControl: read on ** for the pull accounts, and every action for the admin through adminPolicy. With the list empty the block is absent and zot has one account, as before. The deploy refuses an entry without a username or a password, a username that does not match ^[a-z0-9][a-z0-9._-]{0,62}$, a duplicate, and the admin username. A machine logs in with docker login <zot_domain> -u <username>. A pull account cannot write a buildx registry cache (git_deploy_registry_cache).
  • Traefik: new variable traefik_public_read_timeout (role traefik, default ""). It sets the request read limit of the public websecure entrypoint, with and without traefik_split_entrypoints. It applies to every router on that host. Set it only on the host that serves the registry, when machines push over the public address. Empty renders nothing, so traefik.yml is unchanged and Traefik keeps its 60s default. The deploy refuses a value that is not whole seconds or a Go duration.
  • Docs: docs/build-strategies.md has a new section "Registry accounts and time limits". docs/build-pipeline.md has a troubleshooting entry for a push that fails at exactly 60 seconds.

Changed

  • Zot: htpasswd now holds the admin line plus one line per pull account. It still changes only when something changed. The role verifies each account with htpasswd -iv (password on stdin), and a read-only probe compares the user names in the file with the wanted names. When either check fails, the role writes the whole file again and restarts zot. So a removed pull account and an unknown line added by hand also disappear. Both probes run in check mode too.

Fixed

  • Zot: the task that reads the admin account from docker_registries had no no_log, so a run with -v printed the registry admin password. It now has no_log: true.

Upgrade notes

  • Zot restarts once. The first run that applies the zot role on the registry host after the upgrade writes the two new time limits into config.json and restarts zot. That is bay up on that env, or a bay deploy that runs the rig roles. Other regions pull from this registry, so run the registry host's env first, or run bay deploy <env> --tags zot there before the others. To keep zot's old 60s limits and skip the restart, set both zot_http_read_timeout and zot_http_write_timeout to "".
  • Pull accounts are opt-in. Put each password in the vault, add zot_pull_users to the group_vars of the registry host, and run bay deploy <env> --tags zot. The role rewrites htpasswd and config.json and restarts zot once. A deployment host can use a pull account through its own docker_registries entry for zot_domain. The registry host and the build server must keep the admin entry.
  • The public entrypoint limit is opt-in. Nothing changes until you set traefik_public_read_timeout. When machines push to the registry over its public address, set it (for example "30m") on the registry host and run bay deploy <env> --tags traefik. Traefik restarts once.

v2.20.0

Choose a tag to compare

@AltanS AltanS released this 11 Nov 22:50

This release closes the Headscale admin API to every client outside a short allowlist (loopback by default), makes Headscale log the real client IP, and makes a dry run validate a new ACL policy. Traefik no longer logs a missing middleware at start for each tailnet route. It also fixes six smaller findings: a stale docker group on the debug user, a nightly prune that deleted volumes, a dry run that failed on a new timer, a sidecar that ignored SIGTERM, an always-changed buildkit task, and a new validator warning.

Added

  • bay validate warns when a tag in headscale_acl_policy.tagOwners has more than one owner, an owner that is a group: or an autogroup:*, or the owner *. The warning names the tag and says: every owner can enrol a node with that tag and so gets every grant of the tag. It never fails the run. See docs/tailnet-naming.md.
  • New variable docker_prune_volumes (see Fixed).

Changed

  • Headscale admin API: new variable headscale_admin_api_allowed_ips (access_gateway defaults, default ["127.0.0.1/32", "::1/128"]). The headscale-admin router (/api, /swagger) no longer uses vpn-chain. It uses public-chain, the headers and compression of the main Headscale router, plus its own IP allowlist headscale-admin-allow from the new variable. In 2.19.0 and 2.19.1 every tailnet node reached the admin API, also the nodes of an external tenant, and an API key was the only guard there. Now every client outside the list gets 403, tailnet nodes included. Nothing in Bay needs the path: bay gateway and registration use docker exec on the control host, and the identity sidecar uses the docker network. headscale_admin_api_public: true still opens the API to everyone and adds no label.
  • Headscale admin API: the deploy stops when headscale_admin_api_allowed_ips is empty, is not a list, or holds a value that is not an IP or a CIDR. Traefik drops a router whose allowlist it cannot parse.
  • Headscale admin API, fail-safe: while the API is closed, the main Headscale router leaves out the admin paths (Host(`<domain>`) && !PathPrefix(`/api`) && !PathPrefix(`/swagger`)). If the headscale-admin router is ever dropped, /api and /swagger get 404. Before, they fell through to the public main router. With headscale_admin_api_public: true the main rule stays Host(`<domain>`). The admin router keeps priority 1000, which is still above the length of the longer main rule.
  • Headscale: config.yaml now has trusted_proxies when Traefik fronts Headscale. New variable headscale_trusted_proxies (role headscale). Its default is computed on each run: the gateway of traefik_docker_network, read with docker network inspect, as a /32 (a /128 for an IPv6 gateway). Traefik runs with host networking, so it reaches the container from that address. Before, Headscale logged the gateway for every request, not the client. Only the gateway is trusted, not the subnet, so another container on the network cannot set the headers. The key and its rules are those of Headscale 0.29.7: prefixes only, 0.0.0.0/0 and ::/0 refused. The role stops on a bare IP or a /0, because Headscale would crash-loop. [] turns it off. The role creates the docker network before it reads it, so a fresh host gets the key on its first deploy.
  • Headscale: both headscale routers now remove a True-Client-IP header sent by the client (new middleware headscale-client-ip). Headscale reads that header before X-Real-IP and X-Forwarded-For, and Traefik passes it through, so with trusted_proxies a client could otherwise choose the IP Headscale logs. Traefik already replaces X-Real-IP and X-Forwarded-For.
  • Headscale admin API, docs: docs/tailnet-ingress.md renames the section to "The Headscale admin API is closed" and shows how to open the API for one device. docs/access-gateways.md, docs/multi-region.md, docs/tailnet-naming.md, the headscale role defaults header and the bay gateway apikey hints say the same. The 2.19.0 text that said the API answers over the tailnet is gone.

Fixed

  • Traefik: at each start, Traefik logged middleware "vpn-only@docker" does not exist once for each tailnet route (<route>-tailnet@file). The route file is loaded by the file provider, and the file provider was ready before the docker provider had read the vpn-only label on the traefik container. The routes worked a second later. The route file now defines its own copy of the allowlist, and each route names it as vpn-only@file. The docker-label routers keep vpn-only and vpn-chain as before. Both copies read vpn_allowed_ips with the same default, and a test renders both and compares the ranges.
  • Access gateway: the task that adds the Headscale tailnet range to vpn_allowed_ips is now tagged always. Before, bay deploy <env> --tags traefik skipped it, so the route file could get a shorter list than the label. It only sets a fact.
  • Headscale ACL: a dry run (-- --check) now validates a new headscale_acl_policy and fails on an invalid one. Before, check mode skipped "Check headscale container is running", then printed "SKIPPING ACL validation, the headscale container is not running", which was false, and the policy was first validated in the real run. The container probe, the staging write, headscale policy check and the removal of the staged file now run in check mode too (check_mode: false). They only read, or write the hidden staged file that the dry run removes again, also when the check fails. The install of the policy stays skipped in check mode. On a host where the container does not run, the dry run stages nothing and the warning is now true.
  • Debug agent: the debugbot account kept its membership of the docker group on a host that an older Bay version provisioned. The group is root-equivalent. The role only ever added groups (append: true), so it never took the membership away. With debug_agent_docker_access: false the role now reads id -nG and runs gpasswd -d <user> docker when the account is a member. The step is idempotent. The primary group and the other groups stay. It runs in check mode as a read only and skips the removal. docs/debug-agent.md says so.
  • Cron jobs: the nightly or weekly docker system prune -af --volumes deleted every volume that no container used at that moment. The data volume of a service whose deploy failed shortly before is such a volume. New variable docker_prune_volumes (default false). Without it the job runs docker system prune -af. The cost: an anonymous volume of a removed container stays on disk, a small leak. Set docker_prune_volumes: true to get the old command back. The comments in roles/cronjobs and the docs that quoted --volumes say the same.
  • Dry runs: bay provision <env> --tags outbound_monitor -- --check failed on a host that had no bay-reboot-alert.timer yet, with Could not find the requested service. Check mode writes no unit file, so the systemd module could not find the unit. The enable and the disable tasks of the three timers of outbound_monitor (outbound check, disk alert, reboot alert) now skip in check mode when the task that writes their unit file reports changed. The same flaw was in the backup timers and the maintenance timers (backup), the trigger watchdog timer (git_deploy) and the CrowdSec data refresh timer (crowdsec). They follow the same rule. A real run is not affected.
  • Tailnet identity: the tailnet-identity sidecar ignored SIGTERM. The server is PID 1 in its container, Linux does not apply the default SIGTERM action to PID 1, so docker stop waited 30 s and then killed it. The server now handles SIGTERM and SIGINT, stops listening and exits 0. A test starts it as PID 1 of a fresh PID namespace and checks that SIGTERM ends it in under 3 s.
  • Build server: the task Apply memory limit to the bay buildx container (and Apply memory limit to builder container for local builds) had changed_when: rc == 0. docker update exits 0 also when nothing moves, so the task said changed on every deploy. It now reads memory and swap of the container before and after, and reports changed only when they differ. See Upgrade notes for the part that is not a bug.

Upgrade notes

  • Apply on existing hosts, in this order. Run each command against the fleet of that env.
    1. Check your own Headscale API clients first (see "Headscale admin API" below). A client that calls /api/... over the tailnet gets 403 after step 3.
    2. bay provision <env> --tags agent-debug on each host with debug_agent_enabled: true. No container is touched.
    3. bay deploy <env> --tags headscale on the env that runs Headscale. Recreates headscale once; every node reconnects once.
    4. bay deploy <env> --tags tailnet_identity on the control and ingress env. Recreates tailnet-identity (new image); requests with identity_inject fail for a few seconds.
    5. bay deploy <env> --tags traefik on the ingress env. Rewrites dynamic/tailnet-proxies.yml; no container is recreated.
    6. bay deploy <env> --tags cronjobs on each env. Rewrites one cron line; no container is touched.
    7. Optional: bay deploy <env> --tags git_deploy on each env with builds, for the new buildkit memory reporting. No container is recreated.
      No other container is recreated by this release. The dry-run and validator changes need no command.
  • Headscale (admin API allowlist, trusted_proxies, True-Client-IP strip): run bay deploy <env> --tags headscale against the fleet that runs Headscale. The router labels change, so the headscale container is recreated once and every node reconnects once. This happens also with headscale_admin_api_public: true, because the main router gets the headscale-client-ip middleware. The same run renders trusted_proxies into config.yaml before the recreate, so there is no second restart. No other container is recreated. A plain bay up of the control box env also re...
Read more

v2.19.1

Choose a tag to compare

@AltanS AltanS released this 11 Nov 19:40

Fixed

  • bay deploy --log <file> and bay provision --log <file> crashed at start with ModuleNotFoundError: No module named 'click'. This was so from 2.19.0 on a machine installed with uv tool install. Newer Typer carries Click inside itself and does not install a top-level click. The tests did not see it, because the test environment has click through its dev dependencies. The crash came before the run began, so no box was touched and nothing was deployed or provisioned. The same import sat in an error path of bay up --json and bay rollback --json, which hit it only on an unexpected error. Bay now gets the exit and error classes through Typer and no longer imports click. A new test bans import click in src/ and runs the --log summary with click unimportable.
  • Without --log, bay deploy and bay provision were not affected.

v2.19.0

Choose a tag to compare

@AltanS AltanS released this 11 Nov 14:59

This release adds three things. Hosts alert when a reboot stays pending. The Headscale admin API is closed to the internet, and a box in another region registers through the control host over SSH. bay host upgrade upgrades the OS packages of one box, and bay deploy --tags tailnet_identity rotates the sidecar key alone.

Added

  • Reboot alert: new alert host.reboot_pending (level warn, on by default). A daily timer, bay-reboot-alert.timer, checks /var/run/reboot-required. unattended-upgrades installs a kernel update but never reboots, so a host could run for months on an old kernel with no word to anyone. When the file is older than the threshold, the alert names the host, the days pending, the first five packages from /var/run/reboot-required.pkgs, and the running kernel against the newest installed one.
  • Reboot alert: new outbound_monitor variables: reboot_alert_enabled (default true), reboot_alert_threshold_days (default 7) and reboot_alert_repeat_days (default 7). The first alert comes when the file reaches the threshold. After that, one reminder per repeat interval while it stays pending. The age counts from the first time the probe saw the file, so a later package that rewrites the file does not restart the clock. The state (first_seen and last_alert) is kept in /var/lib/bay-outbound-check/reboot-alert.state, and it is cleared when the file is gone. Set reboot_alert_enabled: false to stop and mask the timer.
  • Reboot alert, docs: docs/alerting.md has a new section "Pending reboot alert".
  • Headscale admin API: new variable headscale_admin_api_public (default false). A second Traefik router on the Headscale host name takes PathPrefix(/api) and PathPrefix(/swagger) and sends them through vpn-chain, the chain of every access: vpn service. The REST API and the Swagger page answer only to vpn_allowed_ips (loopback and the tailnet range). From the internet they answer 403. The router has priority 1000, above the main Headscale router, whose priority is the length of its rule. Under traefik_split_entrypoints it also binds websecure_tailnet. The tailscale protocol paths stay public: /ts2021, /key, /register/..., /auth/..., /oidc/callback, /verify, /derp, /bootstrap-dns, /machine/ping-response, /apple, /windows, /health, /version. A test checks this list against Headscale 0.29.7. true keeps the old behaviour and adds no label.
  • Headscale admin API: bay gateway apikey list shows the ID, prefix, expiry and creation time of every API key. bay gateway apikey expire <prefix-or-id> expires one key. It takes the prefix as list prints it (hskey-api-<prefix>-***), the bare prefix or the ID. Both run the Headscale CLI in the container on the control host, like every other bay gateway verb.
  • Headscale admin API: new variable headscale_register_via_api (default false, role tailscale_register). true keeps the old API registration path for a fleet whose deploy machine cannot reach the control host over SSH.
  • Host upgrade: bay host upgrade <env> --box <box> [--reboot] [--json] [--log <file>] upgrades the OS packages of one box. It runs apt-get dist-upgrade (keeping local config files) in a detached systemd unit, bay-host-upgrade, so a dropped SSH session does not stop it, then dpkg --configure -a (with one repair pass when apt fails, see below). It checks Docker, every container that ran before (healthy where it has a healthcheck), nftables and the Bay tables, Docker's chains, one published port, CrowdSec, the bouncer and tailscale. A container that is restarting, or whose restart count grew during the wait, fails the run (container_restarts), and its last log lines are printed: after a cold boot, Traefik looped until tailscaled had its address. Where tailscale is installed, tailscale ip -4 must give an address. A failed check fails the run with its name. --reboot reboots when a reboot is needed and waits for the box. Without it, the command says that a reboot is pending. -- --check changes nothing on the box. It writes no receipt and no lock record. See docs/host-upgrade.md.
  • Host upgrade: the upgrade refuses to start while a bay up holds a claim on the box env (env_busy, exit 43), while the box holds the deploy lock (box_locked, exit 43), or while an earlier upgrade unit still runs (upgrade_running, exit 43). It holds the deploy lock while it runs and touches it at each poll round.
  • Host upgrade: when apt-get dist-upgrade exits with an error, the run does not fail yet. It runs one repair pass: it waits up to host_upgrade_crowdsec_wait (60 s) until cscli lapi status answers (when CrowdSec is installed), runs dpkg --configure -a, then apt-get -f install -y and one more dist-upgrade with the same flags, in the same detached unit and the same log. The apt step fails only when that pass fails or dpkg --audit still lists a package. The result has apt_repair, apt_first_rc, apt_rc and dpkg_audit. A live trial needed it: the postinst of crowdsec-firewall-bouncer-nftables started the bouncer while CrowdSec restarted, and apt exited 100 on an upgrade of 78 packages.
  • Host upgrade: once the apt unit started, the firewall repair and the checks always run, also when the apt step failed. The result always has checks and containers.after. The run still fails, and failed_task names the apt step. The deploy lock is released on every path.
  • Host upgrade: --reboot never reboots when the apt step failed or dpkg --audit is not empty. reboot_note in the result says why.
  • bay deploy, bay provision and bay host upgrade with --log <file> and no --json end with a short summary on the terminal: the result word (ok or FAILED), the PLAY RECAP counts per host (ok, changed, failed, and unreachable when not 0) and the log path. bay host upgrade adds its key facts: packages upgraded, a repair pass, a pending or done reboot, the checks. Before, the terminal showed only the fleet: line, also for a failed run. bay up does not change.
  • Host upgrade: an upgrade of the nftables package flushes the whole ruleset (ExecStop=nft flush ruleset), Docker's rules included. When the package was upgraded, when nftables.service started again, or when a Docker chain is missing, the upgrade loads /etc/nftables.conf again, then restarts Docker, then the CrowdSec bouncer.

Changed

  • Headscale admin API: tailscale_register no longer calls the public REST API for a box outside the control region. It delegates to the control host, the first host of the inventory group named by headscale_control_region, and runs docker exec headscale headscale ... there as root: create the user, list users as JSON, make a 5-minute pre-auth key as JSON. The box then runs tailscale up with the key. No API key is needed. The deploy stops with a message that says what to do when the control host is not in the inventory, does not answer over SSH, or does not run Headscale. In check mode nothing runs, as before.
  • Headscale admin API: on the API path (headscale_register_via_api: true), a 403 from Traefik stops the deploy with a message that names headscale_admin_api_public.
  • Headscale admin API: bay gateway apikey says that Bay does not need the key. The next steps of bay region add no longer name headscale_api_key; they say to check SSH access to the control host. Docs: docs/tailnet-ingress.md has a new section "The Headscale admin API is tailnet-only". docs/multi-region.md has "Registration and the admin API" and new troubleshooting lines. docs/access-gateways.md and the headscale role defaults header say the same.
  • Headscale admin API, limit: a deploy machine with no SSH to the control host cannot enrol a box of another region, unless the fleet opts back in with headscale_register_via_api: true, headscale_admin_api_public: true and headscale_api_key in the vault.
  • Host upgrade: deploy.yml and provision.yml start with a guard: they stop before any role touches the box while a host upgrade runs on it (a fresh lock of kind host_upgrade, or the unit bay-host-upgrade active). Before, a deploy died on the apt lock half way.
  • Host upgrade: the deploy_stack lock message says when a host upgrade holds the lock.
  • Tailnet identity: bay deploy <env> --tags tailnet_identity now rotates the sidecar's Headscale API key alone. When the run does not include deploy_stack, the role reconciles only the tailnet-identity container on the control host (container_lifecycle_only, no orphan removal, no receipt), as the headscale role does for its container. The env file is in the sidecar's config hash, so a new key recreates it and an unchanged key is a no-op. No other container is touched. The role then waits until /healthz answers, and with the api source it checks the key against Headscale from inside the sidecar and fails the run unless the answer is 200. Before, a rotation needed --tags tailnet_identity,deploy_stack, which reconciles every container on the box. Steps: docs/tailnet-ingress.md, "Rotating the sidecar's API key".
  • bay deploy and bay provision take --log <file>, with the meaning it has for bay up. Progress and the Ansible output go to the file, and the terminal gets only the end summary. It works before or after the environment name. All of bay up, bay plan, bay remove, bay host upgrade, bay deploy and bay provision now use one implementation of the redirect (bay_cli.commands.output), and one help text.

Fixed

  • bay deploy and bay up printed "This deploy changed no containers" after a run that had recreated containers. The reconciler runs once per pass, and a run can have several passes on one host: the narrow headscale, tailnet-identity and webhook passes, then deploy_stack. Each pass wrote the same report file, so the last pass replaced the others. Each pass now writes its own file (`..json...
Read more

v2.18.1

Choose a tag to compare

@AltanS AltanS released this 11 Nov 13:09

A HALF project stays HALF until every box it waits for got a deploy. A later run that skips such a box no longer clears it.

Fixed

  • A bay up that failed on one box left the project HALF, but the next run cleared it even when that run did not deploy the failed box. The failed run commits its compile, so the next plan of the project often has a smaller scope: no build change (no build server), no route change (no ingress box), no move (no old box), no planned step of another project. That run passed on the project's own box and wrote result: ok. The fleet then said the project was fine, while the failed box never got the deploy. bay rollback and bay up --take-over cleared it the same way.
  • A run with no host map at receipt time took the env-wide result for every box of the box env, also for the boxes its scope skipped. Only the boxes of the scope get a result now.
  • With traefik_split_entrypoints: true, a cold boot could leave Traefik down. Traefik binds websecure_tailnet to the tailnet address, and tailscaled has no address until it reaches Headscale. Traefik exited with listen tcp <ip>:443: bind: cannot assign requested address, which stopped the public 80 and 443 too. On the Headscale host it was a deadlock, because Traefik serves Headscale. The traefik role now sets net.ipv4.ip_nonlocal_bind=1 (live, and in /etc/sysctl.d/99-bay-traefik-nonlocal-bind.conf) before it writes the Traefik config. The listener still binds only the tailnet address. The variable traefik_nonlocal_bind follows traefik_split_entrypoints; set it to false to opt out (the file is removed, the live value stays until the next reboot). IPv4 only.
  • The boot unit bay-infra-boot.service (bay-infra-ensure.sh) removed Traefik when port 80 was not bound 30s after boot, and made a new one from the infra compose file. The removed container came from the Bay reconciler (label com.bay.config-hash). The compose copy has no such label and may differ from the Bay spec, so the next rig deploy planned a Traefik recreate ("the container has no config-hash label"). The recreate could not help anyway when the cause was a missing tailnet address. The script now never removes or replaces a container with that label. A labelled container that is dead or restarting, or Traefik with port 80 unbound, gets one docker restart, then a WARNING in the journal that names the likely cause. An exited container stays stopped, so a deliberate stop survives a boot. Compose create stays for a container that does not exist, and the log says the next bay deploy --rig <env> recreates it under the reconciler. A container without the label keeps the old behaviour.

Changed

  • A pending or failed lock record has a new key half_boxes: the boxes that still need a deploy. bay up writes it with the pending record, before it touches a box. After the deploy it keeps the boxes that failed or wrote no receipt from this deploy. A one-project run counts every box of its scope; a run of the whole environment counts the project's own box and the old box of its move.
  • A later run removes a box from the list only when its scope covered the box and the box passed. The record stays failed while one box is left. Such a run exits 0 and adds a note: <project> <env> stays HALF: this run did not deploy box <box>, ... Run bay up <env> --project <project> --all-boxes. pinned rows of a HALF record carry half_boxes.
  • bay show ends the HALF reason with the boxes and the command that covers them (--all-boxes when a box other than the project's own box is left). bay show --json has envs[].half_boxes and envs[].next.
  • bay plan adds a note when its scope skips a box that the project still waits for.
  • Docs: docs/plan.md has a new section "A HALF record names its boxes", and the HALF row of the bay show table names the boxes.

Upgrade notes

  • Every machine that deploys a fleet needs 2.18.1. An older CLI refuses half_boxes in a lock as an unknown key, so its compile, plan and deploy stop with a lock schema error. bay.compat.json stops it earlier: the first 2.18.1 bay up writes deployed_with: 2.18.1 in the same claim commit that first writes half_boxes, so a 2.16.0 to 2.18.0 machine stops with bay_older (exit 45) before it writes anything. lock_version stays 2.
  • A record that is HALF before the upgrade has no list, so it waits for every box of its box env. A one-project run no longer clears it. Run bay up <env> in the fleet (the whole environment), or bay up <env> --project <name> --all-boxes, once per such project. bay show names the command.
  • Split-entrypoint hosts: run bay deploy <env> --tags traefik on every such host. The sysctl takes effect at once and recreates no container. This tag restarts Traefik only when the rendered Traefik config changes, and this release does not change it, so Traefik keeps running. The fix matters at the next cold boot.
  • Every rig host: run bay deploy <env> --tags boot_safety (--tags turns the rig roles on) to install the new /usr/local/bin/bay-infra-ensure.sh. The boot_safety role runs from deploy.yml (rig roles only), not from provision.yml. It only rewrites the script. The unit file does not change and no container restarts.

v2.18.0

Choose a tag to compare

@AltanS AltanS released this 11 Nov 10:49

Headscale hardening: random IP allocation for new nodes, a consistent and checked DB backup, a locked-down container, and tighter file modes.

Changed

  • New role variable headscale_ip_allocation feeds prefixes.allocation in the Headscale config. The default is random. The value must be random or sequential; the headscale role stops on any other value. Reason: ACL hosts: aliases and grants point at IPs, and with sequential Headscale gives the IP of a deleted node, and every grant on it, to the next node that joins.
  • tailscale_register now checks a host right after it registers: when its tailnet IP is not headscale_server_tailnet_ip, the deploy stops and the message names the IP to set. Under random, a fresh control host no longer gets 100.64.0.1, the default of that variable. A host that is already registered is never stopped by this check.
  • The Headscale backup no longer copies the live database with docker cp. The DB runs in WAL mode, and a file copy can catch db.sqlite and db.sqlite-wal at different moments. A root helper (/usr/local/sbin/bay-headscale-db-snapshot) runs sqlite3 .backup on the live DB, runs PRAGMA integrity_check on the copy, and fails unless it prints ok. The snapshot and the noise and DERP private keys go to restic as before (headscale.tar, same headscale/ layout). The live -wal and -shm files are no longer in the archive. The temp copy is 0600 in the root-only /var/lib/bay-headscale-backup and is removed on every exit. A failed check stores no snapshot and sends the usual backup.failed alert. The backup unit still runs as the app user and calls the helper through one sudoers rule (/etc/sudoers.d/bay-headscale-backup) that allows no arguments. Restic, retention, the size check, maintenance and alerts are unchanged. backup_headscale_source_path is gone; backup_headscale_data_dir, backup_headscale_snapshot_bin and backup_headscale_snapshot_dir are new.
  • The Headscale container runs with a read-only root filesystem, a tmpfs for its unix socket (/var/run/headscale, so docker exec headscale headscale ... keeps working), the config folder mounted :ro, cap_drop: [ALL] plus cap_add: [DAC_OVERRIDE], and no-new-privileges. It still runs as root. DAC_OVERRIDE is the one capability it needs: the config and data folders belong to the app user, so root without it cannot read config.yaml or write the data folder (checked against headscale/headscale:0.29.7: with every capability dropped it does not find its config; with DAC_READ_SEARCH alone it cannot write its noise key). The reconciler learned the read_only, tmpfs and cap_add spec keys; a spec without them hashes as before, so no other container is recreated.
  • File modes: /opt/headscale/data is 0700 and /opt/headscale/config 0750 (both were 0755). db.sqlite, its -wal and -shm files and every DB copy in the data folder or in /opt/headscale/backups become 0600 (Headscale created them 0644). The role fixes the modes on every run and deletes nothing. SQLite creates new -wal and -shm files with the mode of db.sqlite, so they stay 0600 after a restart.
  • The headscale role refuses tailnet_identity_source: sqlite: the sidecar runs as a non-root user and can no longer read the DB. The default api source is unaffected.
  • Docs: docs/backups.md describes the snapshot helper, moves the old -wal and -shm files away before a restore, and has a new "Restore test (Headscale)" section. docs/access-gateways.md says how to set headscale_server_tailnet_ip under random allocation.

Upgrade notes

  • Run bay deploy <env> --tags headscale,backup against the fleet that runs Headscale (both roles are in the deploy playbook, not in provision). headscale sets the file modes and recreates the container with the hardened spec. backup installs the sqlite3 package on the host, the helper, its folder and the sudoers rule, and renders the new backup script. A full bay deploy --rig <env> does the same.
  • The Headscale container restarts once (the spec hash changes). Nodes reconnect within seconds. Nothing else is recreated.
  • A plain bay up on that fleet also recreates the Headscale container, because bay up builds the same spec. bay plan shows it as a headscale recreate step. So the first bay up after the upgrade is not a no-op on the control host. Run the --tags headscale,backup deploy first, so the modes, the helper and the container change in one step.
  • IP allocation: existing nodes keep their addresses; only nodes that join after the upgrade get a random free IP. Headscale reads every assigned IP from its database as used, and its startup backfill only fills a node that has no IP. To keep the old behaviour, set headscale_ip_allocation: sequential in group_vars before the deploy.
  • A fleet that sets tailnet_identity_source: sqlite must switch to api first, or the headscale role stops.
  • The first backup after the upgrade is a new-format archive. Restore older snapshots as before; docs/backups.md covers both.
  • An old copy such as /opt/headscale/backups/db.sqlite.pre-v0292-* becomes 0600 but stays. Delete it by hand when you no longer need it.

v2.17.3

Choose a tag to compare

@AltanS AltanS released this 11 Nov 09:15

A test fix. Nothing changes for a fleet or a box.

Fixed

  • tests/test_log_retention_integration.py failed now and then with subprocess.TimeoutExpired when the Docker daemon of the test machine was busy. Its Docker calls had limits of 5 to 30 seconds. They are now 60 to 120 seconds. This test stopped one release gate.

Upgrade notes

  • None needed.

v2.17.2

Choose a tag to compare

@AltanS AltanS released this 11 Nov 08:59

A forced push to a local build no longer fails with no reason. [build] repo is checked for its shape. A service with no token no longer names a credential helper that does not exist.

Fixed

  • A local build with the same repo URL and a replaced history (a forced push with an unrelated root commit) ran git pull, which stopped with exit 128 (fatal: Need to specify how to reconcile divergent branches). The failure was the generic "Webhook deploy failed" with no reason, and the circuit breaker counted it. The only way out was a manual full-build mark on the box. A local build now updates its checkout as a remote build does: git fetch of the deploy branch, then git reset --hard FETCH_HEAD. The checkout is build-only, so any history moves it. A fetch that fails ends in the existing loud path ("Repo fetch failed", one build.failed alert with the likely cause), not in the ERR trap.
  • rebuild.sh now asks after the reset whether the old head (the checkout's HEAD before the fetch, as the service saw it) is an ancestor of the new head (git merge-base --is-ancestor). When it is not, or when git cannot find the old commit, the history was rewritten: the build is a full build, with the mark of a new checkout (.git/bay-full-build/<container>). No config-only tag and no no-input-change tag use the old commit. The log says history of <container> was rewritten (<old12> is not in <new12>), full build. When the old head is an ancestor, nothing changes. A remote build did not fail on this, but it did not look either: it built because nothing used the old commit (the receiver writes no no-input-change marker for a force push, and the old commit stays in the checkout for git diff). A trigger file with that marker, after a rewrite that kept the same tree, was a tag-only push of the old image. It now takes the same explicit full-build path.
  • A service with no token had GIT_ASKPASS set to a helper file that Ansible writes only for a service with a token. A public https:// repo that answered 401 or 404 made git run the missing file, and the failure reason read fatal: cannot exec '<...>/.git-askpass-<name>': No such file or directory. rebuild.sh now sets GIT_ASKPASS only for a service with a token, in both strategies. A service with no token keeps GIT_TERMINAL_PROMPT=0, so git fails at once with its own message. The lines of a service with a token are unchanged.

Changed

  • bay toml validate, bay plan and bay up now refuse a [build] repo that is not a git URL or an absolute path. Accepted: https://, http://, ssh://, git:// and file:// URLs with a host and a path, user@host:path, and an absolute local path. Refused: an empty value, a space inside, a bare word, a relative path. Message: build.repo: "<value>" is not a git URL or an absolute path; use for example git@github.com:org/repo.git or https://github.com/org/repo.git.
  • A URL with a password or a token inside (https://user:secret@host/..., https://token@host/...) is refused too. The message shows the URL redacted and says to give the token in repo_tokens in bay.fleet.toml instead. The secret is never printed. The bay.lock repo is not checked: it is written by the CLI from the origin of a checkout, and a check there could stop a bay up after it touched a box.
  • Docs: docs/build-pipeline.md replaces the "known limit" of a local build with a replaced history by what now happens (no manual step), and explains the ancestor rule and the no-token case. docs/bay-toml.md names the accepted forms at the repo key.

Upgrade notes

  • The build script fixes reach a box with the next bay up (or bay deploy), which renders rebuild.sh again. The upgrade recreates no container: rebuild.sh is a plain file that no container hash or handler reads. The container config hash, the com.bay.receiver-hash (the receiver files) and the com.bay.receiver-config-hash (the receiver config and the image map) read none of the files this release changes.
  • The next push after the upgrade is a normal one. A local build moves from git pull to fetch and reset, with the same result for a push that adds commits.
  • Run bay toml validate in a project before bay up. A [build] repo that is a relative path or has a space in it was never usable by git on the box, and bay plan now stops on it. A token in a repo URL moves to repo_tokens in bay.fleet.toml.

v2.17.1

Choose a tag to compare

@AltanS AltanS released this 10 Nov 08:10

A changed [build] repo no longer leaves the box on the old repo. The next build uses the new repo, or fails loud.

Fixed

  • After a change of [build] repo and bay up, the box kept the old repo. A remote build reused its checkout, whose origin still named the old repo, so the build server went on building the old code with no error (with a token, only a full bay deploy --tags git_deploy reset origin; with an SSH URL, nothing did). A local build keys its checkout folder on repo and branch, so the new URL named a folder that only a full deploy clones: the next push failed at the repo lock with a bare "Webhook deploy failed". A changed deploy branch hit the same local failure.
  • rebuild.sh now holds the expected repo of each container (EXPECTED_ORIGIN) and the credentials of a fetch. Before the fetch it clones a missing checkout (into a temp folder, moved into place when complete), and it repoints a checkout whose origin names another repo: first git ls-remote of the new repo and branch, then origin is set, and the fetch and reset --hard FETCH_HEAD move the checkout to the new head, whatever its history. Log line: repo of <container> changed from <old> to <new>, checkout repointed. The repos are compared in the CLI's normal form (host/owner/repo, lower case, no user, password, port or .git), so the same repo in another form is no change.
  • The first build from a new or repointed checkout is a full build: the commit of the old repo decides no config-only or no-input-change tag, and a local build fetches and resets instead of git pull, which refuses an unrelated history. A mark in .git/bay-full-build/ keeps it so until a run has built or found the image of the head.
  • A repo that the box cannot clone, check or fetch ends with one build.failed alert, "Repo check failed (N/MAX)" (or clone, or fetch). It names the container, the repo, the branch and the likely cause: the token cannot read it, the deploy key is not on it (GitHub takes a key on one repo only), the box has no deploy key yet, an https repo is private or missing, the branch is missing, or the host key is not pinned. Nothing is built, and the checkout and its origin stay as they were. The circuit breaker counts it like any failed build. Before, a failed fetch of a remote build fell to the ERR trap with no reason, and the HTTP 403 hint after it never ran.
  • A token in a repo URL is redacted (***) in every log line, alert, state file and plan note.
  • bay build reset wrote an invalid state file. The JSON went through the remote shell inside double quotes, so the shell removed the quotes ({version: 1, consecutive_failures: 0, ...}). bay build status then failed to parse it and printed "No build state found", until the next build rewrote the file. The state now travels as base64, so no shell can change it. The file has the same shape as rebuild.sh writes (last_failure is null, not a block of empty strings), the owner and group of the state directory (the app user and the build group) and mode 0664, as rebuild.sh does. Before, it was root-owned with mode 0644: rebuild.sh could still replace it (it needs write access to the directory, not to the file), but the build group could not write it. The command no longer creates a missing state directory, which would have been root-owned and would have locked rebuild.sh out.
  • bay build status names a state file that is not valid JSON: "the state file of is not valid JSON; run bay build reset <service> --env <env> or wait for the next build to rewrite it". The other services still show. Before, one bad file made the whole listing say "No build state found".
  • bay up (--tags deploy_stack) now also writes the GIT_ASKPASS helper of each local build with a token. The helper path holds the checkout slug, so a new repo or branch needed a helper that only a full deploy wrote.

Changed

  • bay plan and bay up add a note when the build repo of a container changes: <container>: the build repo changes from <old> to <new>. The box builds from the new repo from the next push. Add the box's deploy key to the new repo and create its push webhook first. For a remote build the second sentence says: the build server builds from the new repo from the next push or trigger file, and the deployment box recreates the container when it gets the pull signal. With a token the note names its vault key. The changed: build step stays. Notes are not part of the plan id. A registry build gets no note: Bay does not clone its repo.
  • The plan after bay toml move (and bay adopt) shows one bay-webhook recreate step for each box of the box environment that runs a webhook receiver, not one step in all. bay up deploys the whole box environment, and each receiver holds the config (four such boxes give four steps). The docs (docs/plan.md, docs/bay-toml.md, docs/layout-scenarios.md) and the output and JSON notes of the verb now say: expect only bay-webhook recreate steps, one per box with a webhook receiver. Any other step: stop.
  • A missing local checkout is now cloned by the build script itself, with the key or token the box has. The SSH deploy key of a new local build app still comes only from bay deploy <env> with no --tags.
  • Docs: docs/build-pipeline.md has a new section "When the repo URL changes" (what the box does, the loud failure, the operator checklist for the deploy key, the webhook, the old hook and the old checkout folder, the circuit breaker, rollback, branch changes). docs/bay-toml.md links it from the repo key. The observability contract has row 25 for the new exit. The section also says who builds and who recreates with the remote strategy, how to check the switch without waiting for a recreate (the journal line and git remote get-url origin), why a repointed checkout may skip the image build (the image of the head is already in the registry), and that a same-URL history replacement needs no step on a remote build. A local build still fails there at git pull: the section gives the manual step.

Upgrade notes

  • The fix reaches a box with the next bay up (or bay deploy), which renders rebuild.sh again. No step is needed, and the upgrade recreates no container: rebuild.sh is a plain file that no container hash or handler reads, and the receiver config and image map are unchanged. On a box with a local build that uses a token, bay up writes its GIT_ASKPASS helper file, with the content a full deploy already wrote.
  • A box that already lost a local checkout this way clones it on the next push.
  • When you change a repo URL: give the box read access to the new repo and create its push webhook before bay up, then push or write a trigger file. Remove the old repo's hook and deploy key afterwards. The old local checkout folder stays; remove it by hand. See the checklist in docs/build-pipeline.md.