Skip to content

Release v0.1.1

Choose a tag to compare

@github-actions github-actions released this 27 Oct 00:35
6d173e0

Docker Image

ghcr.io/v31-dev/sage:0.1.1

What's Changed

Fixes

Maintenance

  • Backend test suite, CI and dev tooling by @sidxdev in #77
  • bump Python to 3.14 and some other dependencies by @sidxdev in #78
  • ui test suite and CI by @sidxdev in #79

Other Changes

  • npm audit fixes by @sidxdev in #80
  • Worker monitoring: Telegraf replaces Glances, and status comes from the workers' reports by @sidxdev in #81
  • Remove CLOUDFLARE_ACCOUNT_ID & TS_WORKER_TAG by @sidxdev in #82

Full Changelog: v0.1.0...v0.1.1

Developer Notes

Important

Workers are no longer found by their Tailscale tag: add each one from the Workers page. The workers already in Sage stay, and a machine that carries the tag but was never added is ignored (see the Workers section of the README). Update as for v0.1.0: take a backup, pause Sage until it shows it is safe to stop, then run the newer image against the same volume. On the first start each worker is synced once: Glances is replaced by Telegraf, and Traefik, Vector and cloudflared restart, so expect a short interruption on every worker.

Workers are added by hand, and report for themselves

Sage no longer goes looking for workers, and a worker's state now comes from what the worker reports instead of from SSH polling and Tailscale's own flag.

Adding and removing workers

  • Add Worker on the Workers page takes the hostname or Tailscale IP of a machine on your tailnet (Docker installed, Tailscale SSH enabled). Sage checks that it answers a ping, then sets it up over SSH. The worker shows as adding meanwhile.
  • TS_WORKER_TAG is no longer read. The tag stays in your tailnet policy, where it lets the manager reach the workers. The Tailscale settings page no longer shows it.
  • Several Sages can share a tailnet. Each worker records the Sage that manages it, and a machine another Sage manages is refused with that Sage's domain in the message: delete it from there first. Each Sage needs its own domain.
  • Deleting a worker:
    • an online worker that still has containers is refused: delete the containers first;
    • an online worker without containers has Sage's services stopped and Sage's files removed from it, so the machine is clean again (the VM itself is not deleted);
    • an offline worker is removed from Sage together with its containers. Sage can't clean it up, so delete the VM, or what is left on it, yourself. The dialog says so.
  • A worker is shown as online, not ready, offline, adding, rebooting or removing. Restart and Delete show their state at once, and a second restart or delete of the same worker is refused.

Worker monitoring

  • Every worker runs a Telegraf container (replacing Glances) that pushes the host's and the containers' metrics and the containers' state to the manager every 10 seconds, on port 9001 like Vector's logs.
  • Application and container status follow those reports, so a change shows within seconds instead of at the next minute, and the metrics charts are no longer two minutes behind.
  • A worker is online when it answers tailscale ping, and offline after two misses in a row (about a minute). Tailscale's own online flag could be wrong for up to 15 minutes in both directions. While the manager's own node is cut off from the tailnet, workers keep the state they had.
  • When a report shows a container of the worker's stack down, Traefik not answering, or a worker silent for a minute, Sage runs docker compose up -d for the stack and notifies you, at most once a minute per worker. This also repairs a Traefik that fails to start because the worker booted before its Tailscale address existed.
  • The worker page shows the state of its stack (Traefik, Vector, Telegraf, cloudflared), and the Workers list refreshes every 10 seconds.
  • Telegraf and Vector are limited to 128 MB each on a worker. The recommended minimum machine, for the manager and for each worker, is 1 GB of memory and 1 core.

Cloudflare

  • CLOUDFLARE_ACCOUNT_ID is no longer read: Sage takes the account from the active zone of DOMAIN. The zone must be active (the registrar lists Cloudflare's nameservers) and the token must see only one active zone of that name, or startup stops with a message that says which.

Fixes

  • Deploying an application from a Docker image always pulls the image, so a tag that was pushed again (latest) deploys the new build instead of the copy already on the worker.
  • Metrics charts no longer come back empty when a reading lands at the moment a minute turns.
  • Creating a project or an application without a label answers 400 instead of failing with a server error, and a delete that the database refuses answers 409 instead of 500.
  • A Git repository URL with a trailing newline is refused.
  • One bad timestamp from the backend no longer crashes the page that shows it.
  • 15 vulnerable dependencies in the UI (4 critical, 8 high) are upgraded.

Other changes

  • Secrets on workers are written readable by root only (0600): a worker's .env, an application's .env files and build secrets. The backup and restore scripts are 0700. The workers' traefik/certs/key.pem is rewritten only when the certificate changes, so on an existing worker it stays 0644 until then. Resync Traefik rewrites it as 0600 at once.
  • Commands Sage runs on a worker are stopped by the worker at their deadline, their output reaches the task log as it arrives, and the pooled SSH connections are closed when Sage stops.
  • The image runs on Python 3.14 (from 3.12), and the UI is built on Node 24.
  • Settings → Upgrade shows GitHub alert blocks (Note, Tip, Important, Warning, Caution) as highlighted blocks.

Things to know

  • The env parser follows Docker Compose: a # with no space before it is part of an unquoted value in the project env, the application env, build args and build secrets (PASSWORD=pa#ss is no longer cut at the #), and an apostrophe inside a value (MSG=don't stop) no longer opens a quote. An application that relied on the old behaviour changes on its next deploy.
  • TS_WORKER_TAG and CLOUDFLARE_ACCOUNT_ID can be removed from the env. Leaving them set is harmless, but v0.1.0 refuses to start without them, so keep them if you may go back.
  • METRICS_PORT is now INGEST_PORT (default 9001, so nothing changes unless you set it). Port 61208 (Glances) is no longer used, and a tailnet policy that lists it for the manager's access to workers can drop it. Workers reach the manager on 9001 as before.
  • The backup format is unchanged and v0.1.0 backups restore. A backup made before this release has no instance id, though: restored into a manager that already ran this version, it gives the manager a new one, and its workers (which hold the old one) are refused as managed by another Sage until you delete /opt/sage/revisions/owner on each.

Technical notes

Config

  • Required env: TS_AUTH_KEY, TS_NAME, DOMAIN, CLOUDFLARE_API_TOKEN, ENCRYPTION_KEY (checked in app/entrypoint.sh). No longer read: TS_WORKER_TAG, CLOUDFLARE_ACCOUNT_ID, METRICS_PORT (now INGEST_PORT, ENV INGEST_PORT=9001 in the image).
  • app/entrypoint.sh starts tailscaled without --outbound-http-proxy-listen, and TAILNET_HTTP_PROXY (127.0.0.2:1055) is gone: it carried the Glances requests.
  • app/config.py holds the locations, endpoints and timings: REPORT_INTERVAL_SECONDS 10, REPORT_KEEP_ALIVE_SECONDS 30, REPORT_MAX_BYTES 8 MiB, REPORT_STALE_SECONDS 30, REPORT_SETTLE_SECONDS 15, REPORT_STARTUP_GRACE_SECONDS 60, OFFLINE_AFTER_MISSED_PINGS 2, PING_TIMEOUT_SECONDS 3, WORKER_REBOOT_LIMIT_SECONDS 300, WORKER_STACK_SILENCE_SECONDS 60, WORKER_STACK_REPAIR_COOLDOWN_SECONDS 60, COMMAND_KILL_AFTER_SECONDS 5, COMMAND_WAIT_GRACE_SECONDS 15. Modules read them as config.NAME.
  • Times are stored as naive UTC (utcnow() in utils/common.py).

Startup (app/main.py)

  • The port-9001 server is api_ingest.py (was api_vector.py) on INGEST_PORT, with the access log off and keep-alive REPORT_KEEP_ALIVE_SECONDS. It serves POST /api/telegraf/report and the Vector routes under /api/vector.
  • On the first stop signal, after the running tasks finish, Tailscale.close() closes every pooled SSH connection.
  • Database.reconcile_schema() (startup, and after a restore) creates the Instance table and, when it is empty, one row.

Workers (app/services/manager/workers.py, decisions.py)

  • Worker rows are created only by add_worker. sync_workers looks at known workers only: Tailscale.peers() filtered by hostname, pinged together (Tailscale.online_states: tailscale ping -c 1 --timeout=3s, offline after OFFLINE_AFTER_MISSED_PINGS misses, states untouched while Tailscale.connected() is false). plan_worker_sync (a pure function) decides: offline (stopped answering, or gone from the tailnet), or setup (came back, new IP, forced sync, startup stamp check). repair_worker_stacks follows.
  • Manager.worker_status is computed on each call and stored nowhere: removing (a queued or running remove_worker), adding (add_worker), rebooting (a queued reboot_worker, or a reboot mark until the worker reports a Telegraf that started after the request, at most WORKER_REBOOT_LIMIT_SECONDS), offline, not ready (the report says Traefik does not answer), online.
  • The owner stamp is <worker home>/revisions/owner = <instance id> <domain> (Instance.id is secrets.token_hex(8), kept in the database and so in platform backups). setup_worker reads the stamps first, refuses another instance's (worker_claim_refusal), and writes its own before changing anything. A forced sync deletes revisions after that check and writes the claim again. add_worker needs the machine to answer a ping.
  • remove_worker is async. It deletes the worker's containers (forced), then for a worker that answers a ping, has no containers left and carries this instance's stamp runs docker compose down --remove-orphans && rm -rf /opt/sage. A failure there, or an offline worker, is a warning notification. Then the row, the worker's reports and metrics, and the *.int.<DOMAIN> record for its IP are removed.
  • Worker stack (app/templates/worker/docker-compose.yml): traefik, vector, telegraf, cloudflared (WORKER_STACK_SERVICES); glances is gone. telegraf is telegraf:1.40.2-alpine on the host network, read-only, all capabilities dropped, the Docker socket read-only, running as telegraf:${DOCKER_GID} with the socket's group read by stat -c %g /var/run/docker.sock and written to the worker .env. telegraf/telegraf.conf is rendered from IP, HOSTNAME, TS_IP and INTERVAL, and telegraf is restarted when the infra files change. vector and telegraf have mem_limit and memswap_limit of 128m. The infra stamp also covers manager_ip and report_interval.
  • Tailscale.exec_command runs timeout -k 5 <deadline> "${SHELL:-/bin/sh}" -c '<command>' (the worker enforces the deadline) and gives up deadline + 15 s; exit 124 or 137 near the deadline is reported as a timeout. Output is logged as it arrives. A command is retried only when opening its channel failed, never once it runs.
  • Tailscale.sync_file(..., mode=) uploads to .<name>.<hex>.tmp beside the target with that mode and renames it over. 0600: the worker .env, traefik/certs/key.pem, application .env files and build secrets; 0700: the backup and restore scripts.

Reports (app/services/reports.py, app/routes/telegraf.py)

  • POST /api/telegraf/report takes a gzip JSON batch (chunked) from Telegraf's outputs.http; the sender is the X-Node-Name header. Answers: 204; 404 for a hostname that is not a worker; 400 for a malformed or bad gzip body; 413 above REPORT_MAX_BYTES once decompressed.
  • Reports keeps the newest report of each worker in memory. A report is current for REPORT_STALE_SECONDS, and decides a status only if it arrived REPORT_SETTLE_SECONDS after the last write to the rows it is compared with. A worker that has not reported is not called not ready for REPORT_STARTUP_GRACE_SECONDS after the manager starts.
  • A change in a report queues the status sync of the applications involved (request_status_sync_for_containers); the minute cron (sync_application_status) stays as a safety net. There is no SSH fallback. The decisions are reconcile_container and rollup_status in decisions.py.
  • Metrics.ingest folds the reports into per-minute peaks, and Metrics.flush_workers (the minute cron) writes the completed minutes; queries add the minute still in memory. The per-worker Glances collection is gone.

API

  • Added: POST /api/workers with {"host": "<hostname or Tailscale IP>"} (404 when no such machine is in the tailnet, 409 when it is already added or being added), POST /api/telegraf/report (port INGEST_PORT).
  • GET /api/workers and GET /api/workers/{hostname} carry status; GET /api/workers/{hostname} also carries infra (the stack's containers from the current report, or null).
  • DELETE /api/workers/{hostname}: 400 for an online worker with containers, 409 while it is being removed. POST /api/workers/{hostname}/reboot: 409 unless the status is online or not ready.
  • Removed: worker_tag from GET /api.

Cloudflare (app/services/cloudflare.py)

  • find_zone(domain, token) calls GET /zones?name=<domain>&status=active and raises for no zone, several zones, or a zone without an account. Cloudflare.account_id comes from zone["account"]["id"] and is used for the tunnel calls; the settings check uses the same lookup. The token permissions are unchanged. The records Sage creates are unchanged.

Image and CI

  • Base python:3.14-slim (from python:3.12-slim); UI build on node:24-alpine (from node:lts-alpine). fastapi==0.143.0 with starlette==1.7.0 and pydantic<3.0.0 (from fastapi==0.115.10 and pydantic<2.0.0).
  • New .github/workflows/ci.yml on pull requests: a backend job (ruff, isort, pytest with coverage) and a UI job (Prettier, build, type-check, Vitest with coverage), on ubuntu-26.04 with the actions pinned by commit SHA.
  • ui/vite.config.ts puts vue, the charts and the other libraries in their own chunks.