Repository navigation
Release v0.1.1
Docker Image
ghcr.io/v31-dev/sage:0.1.1What's Changed
Fixes
Maintenance
- Backend test suite, CI and dev tooling by @sidxdev in #77
- bump Python to 3.14 and some other dependencies by @sidxdev in #78
- ui test suite and CI by @sidxdev in #79
Other Changes
- npm audit fixes by @sidxdev in #80
- Worker monitoring: Telegraf replaces Glances, and status comes from the workers' reports by @sidxdev in #81
- Remove CLOUDFLARE_ACCOUNT_ID & TS_WORKER_TAG by @sidxdev in #82
Full Changelog: v0.1.0...v0.1.1
Developer Notes
Important
Workers are no longer found by their Tailscale tag: add each one from the Workers page. The workers already in Sage stay, and a machine that carries the tag but was never added is ignored (see the Workers section of the README). Update as for v0.1.0: take a backup, pause Sage until it shows it is safe to stop, then run the newer image against the same volume. On the first start each worker is synced once: Glances is replaced by Telegraf, and Traefik, Vector and cloudflared restart, so expect a short interruption on every worker.
Workers are added by hand, and report for themselves
Sage no longer goes looking for workers, and a worker's state now comes from what the worker reports instead of from SSH polling and Tailscale's own flag.
Adding and removing workers
- Add Worker on the Workers page takes the hostname or Tailscale IP of a machine on your tailnet (Docker installed, Tailscale SSH enabled). Sage checks that it answers a ping, then sets it up over SSH. The worker shows as adding meanwhile.
TS_WORKER_TAGis no longer read. The tag stays in your tailnet policy, where it lets the manager reach the workers. The Tailscale settings page no longer shows it.- Several Sages can share a tailnet. Each worker records the Sage that manages it, and a machine another Sage manages is refused with that Sage's domain in the message: delete it from there first. Each Sage needs its own domain.
- Deleting a worker:
- an online worker that still has containers is refused: delete the containers first;
- an online worker without containers has Sage's services stopped and Sage's files removed from it, so the machine is clean again (the VM itself is not deleted);
- an offline worker is removed from Sage together with its containers. Sage can't clean it up, so delete the VM, or what is left on it, yourself. The dialog says so.
- A worker is shown as online, not ready, offline, adding, rebooting or removing. Restart and Delete show their state at once, and a second restart or delete of the same worker is refused.
Worker monitoring
- Every worker runs a Telegraf container (replacing Glances) that pushes the host's and the containers' metrics and the containers' state to the manager every 10 seconds, on port 9001 like Vector's logs.
- Application and container status follow those reports, so a change shows within seconds instead of at the next minute, and the metrics charts are no longer two minutes behind.
- A worker is online when it answers
tailscale ping, and offline after two misses in a row (about a minute). Tailscale's own online flag could be wrong for up to 15 minutes in both directions. While the manager's own node is cut off from the tailnet, workers keep the state they had. - When a report shows a container of the worker's stack down, Traefik not answering, or a worker silent for a minute, Sage runs
docker compose up -dfor the stack and notifies you, at most once a minute per worker. This also repairs a Traefik that fails to start because the worker booted before its Tailscale address existed. - The worker page shows the state of its stack (Traefik, Vector, Telegraf, cloudflared), and the Workers list refreshes every 10 seconds.
- Telegraf and Vector are limited to 128 MB each on a worker. The recommended minimum machine, for the manager and for each worker, is 1 GB of memory and 1 core.
Cloudflare
CLOUDFLARE_ACCOUNT_IDis no longer read: Sage takes the account from the active zone ofDOMAIN. The zone must be active (the registrar lists Cloudflare's nameservers) and the token must see only one active zone of that name, or startup stops with a message that says which.
Fixes
- Deploying an application from a Docker image always pulls the image, so a tag that was pushed again (
latest) deploys the new build instead of the copy already on the worker. - Metrics charts no longer come back empty when a reading lands at the moment a minute turns.
- Creating a project or an application without a label answers 400 instead of failing with a server error, and a delete that the database refuses answers 409 instead of 500.
- A Git repository URL with a trailing newline is refused.
- One bad timestamp from the backend no longer crashes the page that shows it.
- 15 vulnerable dependencies in the UI (4 critical, 8 high) are upgraded.
Other changes
- Secrets on workers are written readable by root only (
0600): a worker's.env, an application's.envfiles and build secrets. The backup and restore scripts are0700. The workers'traefik/certs/key.pemis rewritten only when the certificate changes, so on an existing worker it stays0644until then. Resync Traefik rewrites it as0600at once. - Commands Sage runs on a worker are stopped by the worker at their deadline, their output reaches the task log as it arrives, and the pooled SSH connections are closed when Sage stops.
- The image runs on Python 3.14 (from 3.12), and the UI is built on Node 24.
- Settings → Upgrade shows GitHub alert blocks (Note, Tip, Important, Warning, Caution) as highlighted blocks.
Things to know
- The env parser follows Docker Compose: a
#with no space before it is part of an unquoted value in the project env, the application env, build args and build secrets (PASSWORD=pa#ssis no longer cut at the#), and an apostrophe inside a value (MSG=don't stop) no longer opens a quote. An application that relied on the old behaviour changes on its next deploy. TS_WORKER_TAGandCLOUDFLARE_ACCOUNT_IDcan be removed from the env. Leaving them set is harmless, but v0.1.0 refuses to start without them, so keep them if you may go back.METRICS_PORTis nowINGEST_PORT(default 9001, so nothing changes unless you set it). Port 61208 (Glances) is no longer used, and a tailnet policy that lists it for the manager's access to workers can drop it. Workers reach the manager on 9001 as before.- The backup format is unchanged and v0.1.0 backups restore. A backup made before this release has no instance id, though: restored into a manager that already ran this version, it gives the manager a new one, and its workers (which hold the old one) are refused as managed by another Sage until you delete
/opt/sage/revisions/owneron each.
Technical notes
Config
- Required env:
TS_AUTH_KEY,TS_NAME,DOMAIN,CLOUDFLARE_API_TOKEN,ENCRYPTION_KEY(checked inapp/entrypoint.sh). No longer read:TS_WORKER_TAG,CLOUDFLARE_ACCOUNT_ID,METRICS_PORT(nowINGEST_PORT,ENV INGEST_PORT=9001in the image). app/entrypoint.shstartstailscaledwithout--outbound-http-proxy-listen, andTAILNET_HTTP_PROXY(127.0.0.2:1055) is gone: it carried the Glances requests.app/config.pyholds the locations, endpoints and timings:REPORT_INTERVAL_SECONDS10,REPORT_KEEP_ALIVE_SECONDS30,REPORT_MAX_BYTES8 MiB,REPORT_STALE_SECONDS30,REPORT_SETTLE_SECONDS15,REPORT_STARTUP_GRACE_SECONDS60,OFFLINE_AFTER_MISSED_PINGS2,PING_TIMEOUT_SECONDS3,WORKER_REBOOT_LIMIT_SECONDS300,WORKER_STACK_SILENCE_SECONDS60,WORKER_STACK_REPAIR_COOLDOWN_SECONDS60,COMMAND_KILL_AFTER_SECONDS5,COMMAND_WAIT_GRACE_SECONDS15. Modules read them asconfig.NAME.- Times are stored as naive UTC (
utcnow()inutils/common.py).
Startup (app/main.py)
- The port-9001 server is
api_ingest.py(wasapi_vector.py) onINGEST_PORT, with the access log off and keep-aliveREPORT_KEEP_ALIVE_SECONDS. It servesPOST /api/telegraf/reportand the Vector routes under/api/vector. - On the first stop signal, after the running tasks finish,
Tailscale.close()closes every pooled SSH connection. Database.reconcile_schema()(startup, and after a restore) creates theInstancetable and, when it is empty, one row.
Workers (app/services/manager/workers.py, decisions.py)
- Worker rows are created only by
add_worker.sync_workerslooks at known workers only:Tailscale.peers()filtered by hostname, pinged together (Tailscale.online_states:tailscale ping -c 1 --timeout=3s, offline afterOFFLINE_AFTER_MISSED_PINGSmisses, states untouched whileTailscale.connected()is false).plan_worker_sync(a pure function) decides: offline (stopped answering, or gone from the tailnet), or setup (came back, new IP, forced sync, startup stamp check).repair_worker_stacksfollows. Manager.worker_statusis computed on each call and stored nowhere: removing (a queued or runningremove_worker), adding (add_worker), rebooting (a queuedreboot_worker, or a reboot mark until the worker reports a Telegraf that started after the request, at mostWORKER_REBOOT_LIMIT_SECONDS), offline, not ready (the report says Traefik does not answer), online.- The owner stamp is
<worker home>/revisions/owner=<instance id> <domain>(Instance.idissecrets.token_hex(8), kept in the database and so in platform backups).setup_workerreads the stamps first, refuses another instance's (worker_claim_refusal), and writes its own before changing anything. A forced sync deletesrevisionsafter that check and writes the claim again.add_workerneeds the machine to answer a ping. remove_workeris async. It deletes the worker's containers (forced), then for a worker that answers a ping, has no containers left and carries this instance's stamp runsdocker compose down --remove-orphans && rm -rf /opt/sage. A failure there, or an offline worker, is a warning notification. Then the row, the worker's reports and metrics, and the*.int.<DOMAIN>record for its IP are removed.- Worker stack (
app/templates/worker/docker-compose.yml):traefik,vector,telegraf,cloudflared(WORKER_STACK_SERVICES);glancesis gone.telegrafistelegraf:1.40.2-alpineon the host network, read-only, all capabilities dropped, the Docker socket read-only, running astelegraf:${DOCKER_GID}with the socket's group read bystat -c %g /var/run/docker.sockand written to the worker.env.telegraf/telegraf.confis rendered fromIP,HOSTNAME,TS_IPandINTERVAL, andtelegrafis restarted when the infra files change.vectorandtelegrafhavemem_limitandmemswap_limitof128m. The infra stamp also coversmanager_ipandreport_interval. Tailscale.exec_commandrunstimeout -k 5 <deadline> "${SHELL:-/bin/sh}" -c '<command>'(the worker enforces the deadline) and gives updeadline + 15s; exit 124 or 137 near the deadline is reported as a timeout. Output is logged as it arrives. A command is retried only when opening its channel failed, never once it runs.Tailscale.sync_file(..., mode=)uploads to.<name>.<hex>.tmpbeside the target with that mode and renames it over.0600: the worker.env,traefik/certs/key.pem, application.envfiles and build secrets;0700: the backup and restore scripts.
Reports (app/services/reports.py, app/routes/telegraf.py)
POST /api/telegraf/reporttakes a gzip JSON batch (chunked) from Telegraf'soutputs.http; the sender is theX-Node-Nameheader. Answers: 204; 404 for a hostname that is not a worker; 400 for a malformed or bad gzip body; 413 aboveREPORT_MAX_BYTESonce decompressed.Reportskeeps the newest report of each worker in memory. A report is current forREPORT_STALE_SECONDS, and decides a status only if it arrivedREPORT_SETTLE_SECONDSafter the last write to the rows it is compared with. A worker that has not reported is not called not ready forREPORT_STARTUP_GRACE_SECONDSafter the manager starts.- A change in a report queues the status sync of the applications involved (
request_status_sync_for_containers); the minute cron (sync_application_status) stays as a safety net. There is no SSH fallback. The decisions arereconcile_containerandrollup_statusindecisions.py. Metrics.ingestfolds the reports into per-minute peaks, andMetrics.flush_workers(the minute cron) writes the completed minutes; queries add the minute still in memory. The per-worker Glances collection is gone.
API
- Added:
POST /api/workerswith{"host": "<hostname or Tailscale IP>"}(404 when no such machine is in the tailnet, 409 when it is already added or being added),POST /api/telegraf/report(portINGEST_PORT). GET /api/workersandGET /api/workers/{hostname}carrystatus;GET /api/workers/{hostname}also carriesinfra(the stack's containers from the current report, ornull).DELETE /api/workers/{hostname}: 400 for an online worker with containers, 409 while it is being removed.POST /api/workers/{hostname}/reboot: 409 unless the status is online or not ready.- Removed:
worker_tagfromGET /api.
Cloudflare (app/services/cloudflare.py)
find_zone(domain, token)callsGET /zones?name=<domain>&status=activeand raises for no zone, several zones, or a zone without an account.Cloudflare.account_idcomes fromzone["account"]["id"]and is used for the tunnel calls; the settings check uses the same lookup. The token permissions are unchanged. The records Sage creates are unchanged.
Image and CI
- Base
python:3.14-slim(frompython:3.12-slim); UI build onnode:24-alpine(fromnode:lts-alpine).fastapi==0.143.0withstarlette==1.7.0andpydantic<3.0.0(fromfastapi==0.115.10andpydantic<2.0.0). - New
.github/workflows/ci.ymlon pull requests: a backend job (ruff, isort, pytest with coverage) and a UI job (Prettier, build, type-check, Vitest with coverage), onubuntu-26.04with the actions pinned by commit SHA. ui/vite.config.tsputsvue, the charts and the other libraries in their own chunks.