Repository navigation
v2.19.0
This release adds three things. Hosts alert when a reboot stays pending. The Headscale admin API is closed to the internet, and a box in another region registers through the control host over SSH. bay host upgrade upgrades the OS packages of one box, and bay deploy --tags tailnet_identity rotates the sidecar key alone.
Added
- Reboot alert: new alert
host.reboot_pending(levelwarn, on by default). A daily timer,bay-reboot-alert.timer, checks/var/run/reboot-required.unattended-upgradesinstalls a kernel update but never reboots, so a host could run for months on an old kernel with no word to anyone. When the file is older than the threshold, the alert names the host, the days pending, the first five packages from/var/run/reboot-required.pkgs, and the running kernel against the newest installed one. - Reboot alert: new
outbound_monitorvariables:reboot_alert_enabled(defaulttrue),reboot_alert_threshold_days(default7) andreboot_alert_repeat_days(default7). The first alert comes when the file reaches the threshold. After that, one reminder per repeat interval while it stays pending. The age counts from the first time the probe saw the file, so a later package that rewrites the file does not restart the clock. The state (first_seenandlast_alert) is kept in/var/lib/bay-outbound-check/reboot-alert.state, and it is cleared when the file is gone. Setreboot_alert_enabled: falseto stop and mask the timer. - Reboot alert, docs:
docs/alerting.mdhas a new section "Pending reboot alert". - Headscale admin API: new variable
headscale_admin_api_public(defaultfalse). A second Traefik router on the Headscale host name takesPathPrefix(/api)andPathPrefix(/swagger)and sends them throughvpn-chain, the chain of everyaccess: vpnservice. The REST API and the Swagger page answer only tovpn_allowed_ips(loopback and the tailnet range). From the internet they answer 403. The router has priority 1000, above the main Headscale router, whose priority is the length of its rule. Undertraefik_split_entrypointsit also bindswebsecure_tailnet. The tailscale protocol paths stay public:/ts2021,/key,/register/...,/auth/...,/oidc/callback,/verify,/derp,/bootstrap-dns,/machine/ping-response,/apple,/windows,/health,/version. A test checks this list against Headscale 0.29.7.truekeeps the old behaviour and adds no label. - Headscale admin API:
bay gateway apikey listshows the ID, prefix, expiry and creation time of every API key.bay gateway apikey expire <prefix-or-id>expires one key. It takes the prefix aslistprints it (hskey-api-<prefix>-***), the bare prefix or the ID. Both run the Headscale CLI in the container on the control host, like every otherbay gatewayverb. - Headscale admin API: new variable
headscale_register_via_api(defaultfalse, roletailscale_register).truekeeps the old API registration path for a fleet whose deploy machine cannot reach the control host over SSH. - Host upgrade:
bay host upgrade <env> --box <box> [--reboot] [--json] [--log <file>]upgrades the OS packages of one box. It runsapt-get dist-upgrade(keeping local config files) in a detached systemd unit,bay-host-upgrade, so a dropped SSH session does not stop it, thendpkg --configure -a(with one repair pass when apt fails, see below). It checks Docker, every container that ran before (healthy where it has a healthcheck), nftables and the Bay tables, Docker's chains, one published port, CrowdSec, the bouncer and tailscale. A container that isrestarting, or whose restart count grew during the wait, fails the run (container_restarts), and its last log lines are printed: after a cold boot, Traefik looped until tailscaled had its address. Where tailscale is installed,tailscale ip -4must give an address. A failed check fails the run with its name.--rebootreboots when a reboot is needed and waits for the box. Without it, the command says that a reboot is pending.-- --checkchanges nothing on the box. It writes no receipt and no lock record. Seedocs/host-upgrade.md. - Host upgrade: the upgrade refuses to start while a
bay upholds a claim on the box env (env_busy, exit 43), while the box holds the deploy lock (box_locked, exit 43), or while an earlier upgrade unit still runs (upgrade_running, exit 43). It holds the deploy lock while it runs and touches it at each poll round. - Host upgrade: when
apt-get dist-upgradeexits with an error, the run does not fail yet. It runs one repair pass: it waits up tohost_upgrade_crowdsec_wait(60 s) untilcscli lapi statusanswers (when CrowdSec is installed), runsdpkg --configure -a, thenapt-get -f install -yand one moredist-upgradewith the same flags, in the same detached unit and the same log. The apt step fails only when that pass fails ordpkg --auditstill lists a package. The result hasapt_repair,apt_first_rc,apt_rcanddpkg_audit. A live trial needed it: the postinst ofcrowdsec-firewall-bouncer-nftablesstarted the bouncer while CrowdSec restarted, and apt exited 100 on an upgrade of 78 packages. - Host upgrade: once the apt unit started, the firewall repair and the checks always run, also when the apt step failed. The result always has
checksandcontainers.after. The run still fails, andfailed_tasknames the apt step. The deploy lock is released on every path. - Host upgrade:
--rebootnever reboots when the apt step failed ordpkg --auditis not empty.reboot_notein the result says why. bay deploy,bay provisionandbay host upgradewith--log <file>and no--jsonend with a short summary on the terminal: the result word (okorFAILED), the PLAY RECAP counts per host (ok, changed, failed, and unreachable when not 0) and the log path.bay host upgradeadds its key facts: packages upgraded, a repair pass, a pending or done reboot, the checks. Before, the terminal showed only thefleet:line, also for a failed run.bay updoes not change.- Host upgrade: an upgrade of the
nftablespackage flushes the whole ruleset (ExecStop=nft flush ruleset), Docker's rules included. When the package was upgraded, whennftables.servicestarted again, or when a Docker chain is missing, the upgrade loads/etc/nftables.confagain, then restarts Docker, then the CrowdSec bouncer.
Changed
- Headscale admin API:
tailscale_registerno longer calls the public REST API for a box outside the control region. It delegates to the control host, the first host of the inventory group named byheadscale_control_region, and runsdocker exec headscale headscale ...there as root: create the user, list users as JSON, make a 5-minute pre-auth key as JSON. The box then runstailscale upwith the key. No API key is needed. The deploy stops with a message that says what to do when the control host is not in the inventory, does not answer over SSH, or does not run Headscale. In check mode nothing runs, as before. - Headscale admin API: on the API path (
headscale_register_via_api: true), a 403 from Traefik stops the deploy with a message that namesheadscale_admin_api_public. - Headscale admin API:
bay gateway apikeysays that Bay does not need the key. The next steps ofbay region addno longer nameheadscale_api_key; they say to check SSH access to the control host. Docs:docs/tailnet-ingress.mdhas a new section "The Headscale admin API is tailnet-only".docs/multi-region.mdhas "Registration and the admin API" and new troubleshooting lines.docs/access-gateways.mdand theheadscalerole defaults header say the same. - Headscale admin API, limit: a deploy machine with no SSH to the control host cannot enrol a box of another region, unless the fleet opts back in with
headscale_register_via_api: true,headscale_admin_api_public: trueandheadscale_api_keyin the vault. - Host upgrade:
deploy.ymlandprovision.ymlstart with a guard: they stop before any role touches the box while a host upgrade runs on it (a fresh lock of kindhost_upgrade, or the unitbay-host-upgradeactive). Before, a deploy died on the apt lock half way. - Host upgrade: the
deploy_stacklock message says when a host upgrade holds the lock. - Tailnet identity:
bay deploy <env> --tags tailnet_identitynow rotates the sidecar's Headscale API key alone. When the run does not includedeploy_stack, the role reconciles only thetailnet-identitycontainer on the control host (container_lifecycle_only, no orphan removal, no receipt), as the headscale role does for its container. The env file is in the sidecar's config hash, so a new key recreates it and an unchanged key is a no-op. No other container is touched. The role then waits until/healthzanswers, and with theapisource it checks the key against Headscale from inside the sidecar and fails the run unless the answer is200. Before, a rotation needed--tags tailnet_identity,deploy_stack, which reconciles every container on the box. Steps:docs/tailnet-ingress.md, "Rotating the sidecar's API key". bay deployandbay provisiontake--log <file>, with the meaning it has forbay up. Progress and the Ansible output go to the file, and the terminal gets only the end summary. It works before or after the environment name. All ofbay up,bay plan,bay remove,bay host upgrade,bay deployandbay provisionnow use one implementation of the redirect (bay_cli.commands.output), and one help text.
Fixed
bay deployandbay upprinted "This deploy changed no containers" after a run that had recreated containers. The reconciler runs once per pass, and a run can have several passes on one host: the narrow headscale, tailnet-identity and webhook passes, thendeploy_stack. Each pass wrote the same report file, so the last pass replaced the others. Each pass now writes its own file (<host>.<pass>.json) and the summary merges them all. When a report file is unreadable, the summary prints no "changed" or "no containers" line and shows every service in one list.bay deploy -- --checkpushed the local fleet commits before the dry run. A dry run (--checkor-Cafter--) now pushes nothing. It still fetches, fast-forwards a fleet that is behind, and refuses a diverged fleet, an unreachable remote and uncommitted files, as a real run does. It prints one note that the local fleet commits were not pushed because this is a dry run.bay provisionfollows the same rule.bay gateway apikeyprinted help text around the key, and the key twice, when stdout was not a terminal.bay gateway apikey | bay vault set ...was not safe. Now stdout carries only the key and one newline, once, in every mode (also with--json). The header, the hints, the fleet line and every error go to stderr.bay gateway keyandbay gateway enrollfollow the same rule: on a pipe, stdout is the pre-auth key and the join command with a<key>placeholder goes to stderr.bay --json gateway enrollkeeps its JSON document. On a terminal nothing changes: the key shows in the alternate screen.docs/tailnet-ingress.mdrotates the sidecar key with the pipe.
Upgrade notes
- Reboot alert: the probe is part of the
outbound_monitorrole, and that role runs fromprovision.ymlonly. A deploy does not install it. On each existing host, runbay provision <env> --tags outbound_monitor. A host that is not re-provisioned has no check and sends no alert. - Reboot alert: expect one alert per host that has had a pending reboot for 7 days or more, on the first run after the install.
- Reboot alert: Bay does not reboot hosts. Reboot at a quiet time, one box at a time.
- Headscale admin API: run
bay deploy <env> --tags headscaleagainst the fleet that runs Headscale. The headscale role reconciles theheadscalecontainer alone. The new router labels change its spec, so the container is recreated once and every node reconnects once. A plainbay upof the control box env does the same and shows it as aheadscalerecreate step. A fleet that setsheadscale_admin_api_public: truebefore the deploy keeps its labels, and its container is not recreated. - Headscale admin API: check your own API clients first. A script or tool that calls
https://<headscale_domain>/api/...from outside the tailnet gets 403 after the deploy. Move it onto the tailnet, or setheadscale_admin_api_public: true. The tailnet-identity sidecar is not affected: it callshttp://headscale:8080on the docker network. - Headscale admin API: boxes that are already on the tailnet are not touched. The new registration path runs only when a box joins. From then on, a deploy of a remote box (also with
--limit) runs tasks on the control host, so the deploy machine needs SSH and sudo there. - Headscale admin API: when the deploy is live and every remote box has joined,
headscale_api_keyhas no Bay user left. Find its prefix (the 12 characters afterhskey-api-in the key, orbay gateway apikey list), runbay gateway apikey expire <prefix>, and remove the key from the vault withbay vault edit <env>. Keep it if the fleet setsheadscale_register_via_api: true, or iftailnet_identity_api_keyor another client holds the same key. - Tailnet identity: to rotate the key:
bay gateway apikey,bay vault set <env> tailnet_identity_api_key,bay deploy <env> --tags tailnet_identity, verify, then expire the old key withbay gateway apikey expire <id>(bay gateway apikey listshows the ID). - Fresh framework checkout:
bay provisionneeds the roles invendor/roles, and they are not in git. Runbay self updateormake installin the checkout first. Both fillvendor/roles. - Host upgrade: nothing to run for the guard. The guard is part of every deploy and provision from 2.19.0. A machine on an older Bay does not know the guard, so upgrade the machines that deploy before you run
bay host upgrade.