Skip to content

v2.19.0

Choose a tag to compare

@AltanS AltanS released this 11 Nov 14:59
· 32 commits to main since this release

This release adds three things. Hosts alert when a reboot stays pending. The Headscale admin API is closed to the internet, and a box in another region registers through the control host over SSH. bay host upgrade upgrades the OS packages of one box, and bay deploy --tags tailnet_identity rotates the sidecar key alone.

Added

  • Reboot alert: new alert host.reboot_pending (level warn, on by default). A daily timer, bay-reboot-alert.timer, checks /var/run/reboot-required. unattended-upgrades installs a kernel update but never reboots, so a host could run for months on an old kernel with no word to anyone. When the file is older than the threshold, the alert names the host, the days pending, the first five packages from /var/run/reboot-required.pkgs, and the running kernel against the newest installed one.
  • Reboot alert: new outbound_monitor variables: reboot_alert_enabled (default true), reboot_alert_threshold_days (default 7) and reboot_alert_repeat_days (default 7). The first alert comes when the file reaches the threshold. After that, one reminder per repeat interval while it stays pending. The age counts from the first time the probe saw the file, so a later package that rewrites the file does not restart the clock. The state (first_seen and last_alert) is kept in /var/lib/bay-outbound-check/reboot-alert.state, and it is cleared when the file is gone. Set reboot_alert_enabled: false to stop and mask the timer.
  • Reboot alert, docs: docs/alerting.md has a new section "Pending reboot alert".
  • Headscale admin API: new variable headscale_admin_api_public (default false). A second Traefik router on the Headscale host name takes PathPrefix(/api) and PathPrefix(/swagger) and sends them through vpn-chain, the chain of every access: vpn service. The REST API and the Swagger page answer only to vpn_allowed_ips (loopback and the tailnet range). From the internet they answer 403. The router has priority 1000, above the main Headscale router, whose priority is the length of its rule. Under traefik_split_entrypoints it also binds websecure_tailnet. The tailscale protocol paths stay public: /ts2021, /key, /register/..., /auth/..., /oidc/callback, /verify, /derp, /bootstrap-dns, /machine/ping-response, /apple, /windows, /health, /version. A test checks this list against Headscale 0.29.7. true keeps the old behaviour and adds no label.
  • Headscale admin API: bay gateway apikey list shows the ID, prefix, expiry and creation time of every API key. bay gateway apikey expire <prefix-or-id> expires one key. It takes the prefix as list prints it (hskey-api-<prefix>-***), the bare prefix or the ID. Both run the Headscale CLI in the container on the control host, like every other bay gateway verb.
  • Headscale admin API: new variable headscale_register_via_api (default false, role tailscale_register). true keeps the old API registration path for a fleet whose deploy machine cannot reach the control host over SSH.
  • Host upgrade: bay host upgrade <env> --box <box> [--reboot] [--json] [--log <file>] upgrades the OS packages of one box. It runs apt-get dist-upgrade (keeping local config files) in a detached systemd unit, bay-host-upgrade, so a dropped SSH session does not stop it, then dpkg --configure -a (with one repair pass when apt fails, see below). It checks Docker, every container that ran before (healthy where it has a healthcheck), nftables and the Bay tables, Docker's chains, one published port, CrowdSec, the bouncer and tailscale. A container that is restarting, or whose restart count grew during the wait, fails the run (container_restarts), and its last log lines are printed: after a cold boot, Traefik looped until tailscaled had its address. Where tailscale is installed, tailscale ip -4 must give an address. A failed check fails the run with its name. --reboot reboots when a reboot is needed and waits for the box. Without it, the command says that a reboot is pending. -- --check changes nothing on the box. It writes no receipt and no lock record. See docs/host-upgrade.md.
  • Host upgrade: the upgrade refuses to start while a bay up holds a claim on the box env (env_busy, exit 43), while the box holds the deploy lock (box_locked, exit 43), or while an earlier upgrade unit still runs (upgrade_running, exit 43). It holds the deploy lock while it runs and touches it at each poll round.
  • Host upgrade: when apt-get dist-upgrade exits with an error, the run does not fail yet. It runs one repair pass: it waits up to host_upgrade_crowdsec_wait (60 s) until cscli lapi status answers (when CrowdSec is installed), runs dpkg --configure -a, then apt-get -f install -y and one more dist-upgrade with the same flags, in the same detached unit and the same log. The apt step fails only when that pass fails or dpkg --audit still lists a package. The result has apt_repair, apt_first_rc, apt_rc and dpkg_audit. A live trial needed it: the postinst of crowdsec-firewall-bouncer-nftables started the bouncer while CrowdSec restarted, and apt exited 100 on an upgrade of 78 packages.
  • Host upgrade: once the apt unit started, the firewall repair and the checks always run, also when the apt step failed. The result always has checks and containers.after. The run still fails, and failed_task names the apt step. The deploy lock is released on every path.
  • Host upgrade: --reboot never reboots when the apt step failed or dpkg --audit is not empty. reboot_note in the result says why.
  • bay deploy, bay provision and bay host upgrade with --log <file> and no --json end with a short summary on the terminal: the result word (ok or FAILED), the PLAY RECAP counts per host (ok, changed, failed, and unreachable when not 0) and the log path. bay host upgrade adds its key facts: packages upgraded, a repair pass, a pending or done reboot, the checks. Before, the terminal showed only the fleet: line, also for a failed run. bay up does not change.
  • Host upgrade: an upgrade of the nftables package flushes the whole ruleset (ExecStop=nft flush ruleset), Docker's rules included. When the package was upgraded, when nftables.service started again, or when a Docker chain is missing, the upgrade loads /etc/nftables.conf again, then restarts Docker, then the CrowdSec bouncer.

Changed

  • Headscale admin API: tailscale_register no longer calls the public REST API for a box outside the control region. It delegates to the control host, the first host of the inventory group named by headscale_control_region, and runs docker exec headscale headscale ... there as root: create the user, list users as JSON, make a 5-minute pre-auth key as JSON. The box then runs tailscale up with the key. No API key is needed. The deploy stops with a message that says what to do when the control host is not in the inventory, does not answer over SSH, or does not run Headscale. In check mode nothing runs, as before.
  • Headscale admin API: on the API path (headscale_register_via_api: true), a 403 from Traefik stops the deploy with a message that names headscale_admin_api_public.
  • Headscale admin API: bay gateway apikey says that Bay does not need the key. The next steps of bay region add no longer name headscale_api_key; they say to check SSH access to the control host. Docs: docs/tailnet-ingress.md has a new section "The Headscale admin API is tailnet-only". docs/multi-region.md has "Registration and the admin API" and new troubleshooting lines. docs/access-gateways.md and the headscale role defaults header say the same.
  • Headscale admin API, limit: a deploy machine with no SSH to the control host cannot enrol a box of another region, unless the fleet opts back in with headscale_register_via_api: true, headscale_admin_api_public: true and headscale_api_key in the vault.
  • Host upgrade: deploy.yml and provision.yml start with a guard: they stop before any role touches the box while a host upgrade runs on it (a fresh lock of kind host_upgrade, or the unit bay-host-upgrade active). Before, a deploy died on the apt lock half way.
  • Host upgrade: the deploy_stack lock message says when a host upgrade holds the lock.
  • Tailnet identity: bay deploy <env> --tags tailnet_identity now rotates the sidecar's Headscale API key alone. When the run does not include deploy_stack, the role reconciles only the tailnet-identity container on the control host (container_lifecycle_only, no orphan removal, no receipt), as the headscale role does for its container. The env file is in the sidecar's config hash, so a new key recreates it and an unchanged key is a no-op. No other container is touched. The role then waits until /healthz answers, and with the api source it checks the key against Headscale from inside the sidecar and fails the run unless the answer is 200. Before, a rotation needed --tags tailnet_identity,deploy_stack, which reconciles every container on the box. Steps: docs/tailnet-ingress.md, "Rotating the sidecar's API key".
  • bay deploy and bay provision take --log <file>, with the meaning it has for bay up. Progress and the Ansible output go to the file, and the terminal gets only the end summary. It works before or after the environment name. All of bay up, bay plan, bay remove, bay host upgrade, bay deploy and bay provision now use one implementation of the redirect (bay_cli.commands.output), and one help text.

Fixed

  • bay deploy and bay up printed "This deploy changed no containers" after a run that had recreated containers. The reconciler runs once per pass, and a run can have several passes on one host: the narrow headscale, tailnet-identity and webhook passes, then deploy_stack. Each pass wrote the same report file, so the last pass replaced the others. Each pass now writes its own file (<host>.<pass>.json) and the summary merges them all. When a report file is unreadable, the summary prints no "changed" or "no containers" line and shows every service in one list.
  • bay deploy -- --check pushed the local fleet commits before the dry run. A dry run (--check or -C after --) now pushes nothing. It still fetches, fast-forwards a fleet that is behind, and refuses a diverged fleet, an unreachable remote and uncommitted files, as a real run does. It prints one note that the local fleet commits were not pushed because this is a dry run. bay provision follows the same rule.
  • bay gateway apikey printed help text around the key, and the key twice, when stdout was not a terminal. bay gateway apikey | bay vault set ... was not safe. Now stdout carries only the key and one newline, once, in every mode (also with --json). The header, the hints, the fleet line and every error go to stderr. bay gateway key and bay gateway enroll follow the same rule: on a pipe, stdout is the pre-auth key and the join command with a <key> placeholder goes to stderr. bay --json gateway enroll keeps its JSON document. On a terminal nothing changes: the key shows in the alternate screen. docs/tailnet-ingress.md rotates the sidecar key with the pipe.

Upgrade notes

  • Reboot alert: the probe is part of the outbound_monitor role, and that role runs from provision.yml only. A deploy does not install it. On each existing host, run bay provision <env> --tags outbound_monitor. A host that is not re-provisioned has no check and sends no alert.
  • Reboot alert: expect one alert per host that has had a pending reboot for 7 days or more, on the first run after the install.
  • Reboot alert: Bay does not reboot hosts. Reboot at a quiet time, one box at a time.
  • Headscale admin API: run bay deploy <env> --tags headscale against the fleet that runs Headscale. The headscale role reconciles the headscale container alone. The new router labels change its spec, so the container is recreated once and every node reconnects once. A plain bay up of the control box env does the same and shows it as a headscale recreate step. A fleet that sets headscale_admin_api_public: true before the deploy keeps its labels, and its container is not recreated.
  • Headscale admin API: check your own API clients first. A script or tool that calls https://<headscale_domain>/api/... from outside the tailnet gets 403 after the deploy. Move it onto the tailnet, or set headscale_admin_api_public: true. The tailnet-identity sidecar is not affected: it calls http://headscale:8080 on the docker network.
  • Headscale admin API: boxes that are already on the tailnet are not touched. The new registration path runs only when a box joins. From then on, a deploy of a remote box (also with --limit) runs tasks on the control host, so the deploy machine needs SSH and sudo there.
  • Headscale admin API: when the deploy is live and every remote box has joined, headscale_api_key has no Bay user left. Find its prefix (the 12 characters after hskey-api- in the key, or bay gateway apikey list), run bay gateway apikey expire <prefix>, and remove the key from the vault with bay vault edit <env>. Keep it if the fleet sets headscale_register_via_api: true, or if tailnet_identity_api_key or another client holds the same key.
  • Tailnet identity: to rotate the key: bay gateway apikey, bay vault set <env> tailnet_identity_api_key, bay deploy <env> --tags tailnet_identity, verify, then expire the old key with bay gateway apikey expire <id> (bay gateway apikey list shows the ID).
  • Fresh framework checkout: bay provision needs the roles in vendor/roles, and they are not in git. Run bay self update or make install in the checkout first. Both fill vendor/roles.
  • Host upgrade: nothing to run for the guard. The guard is part of every deploy and provision from 2.19.0. A machine on an older Bay does not know the guard, so upgrade the machines that deploy before you run bay host upgrade.