Skip to content

Release v0.1.0

Choose a tag to compare

@github-actions github-actions released this 20 Oct 04:37
ff75e7f

Docker Image

ghcr.io/v31-dev/sage:0.1.0

What's Changed

Other Changes

Full Changelog: v0.0.27...v0.1.0

Developer Notes

Important

This release changes how Sage is deployed: the manager is now a single container with its own Tailscale node and new environment variables, so it needs a new compose file or run command (see the Manager section of the README). The in-app upgrade can't do this, so update by hand. Take a backup first, but keep it only for going back to the previous version: this version can't restore backups made by earlier ones. The database itself has no breaking changes.

The manager is now a single container that joins your tailnet as its own node. It needs no host Tailscale, TUN device, Docker socket or published ports, only a persistent volume and its environment. Cloudflare is part of the platform again: Sage sets up the tunnel, the DNS record and the certificates itself on start.

Setup

  • The manager is configured by environment variables: TS_AUTH_KEY, TS_NAME, TS_WORKER_TAG, DOMAIN, CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN and ENCRYPTION_KEY. A missing one stops the container with a message that names it.
  • The UI is served only at https://<TS_NAME>.core.<DOMAIN>, with its own Let's Encrypt certificate. Plain http://<TS_NAME> redirects to it, and any other address redirects too. Workers' internal domains (*.int.<DOMAIN>) use a second certificate. Both renew automatically, and the manager's is swapped in without a restart.
  • A first start can take a few minutes: Sage sets up Cloudflare, then requests the certificate, and logs each step. If that keeps failing it retries for about 20 minutes and exits with the reason.
  • An invalid or expired TS_AUTH_KEY stops the container at once with a [CRITICAL] line. So does an ENCRYPTION_KEY that isn't the key the data was written with, and the key can't be changed after the first boot.
  • TS_NAME, TS_WORKER_TAG and the Cloudflare details change with an env edit and a restart: workers are brought in line on start.
  • S3 (backups) and Discord notifications stay optional and are set in the UI.
  • Sage has no login of its own. The tailnet policy is the only gate, and the README now has an example policy.
  • The image is smaller (profiling tools are development-only) and the manager uses less memory (about 90 MB, from about 150).

Settings

  • One page per section under a Settings group in the sidebar: Tailscale, Cloudflare, S3, Notifications and Upgrade.
  • Status icons show whether each section works, with the reason when it doesn't. Sections are checked every 5 minutes. The Workers item in the sidebar shows whether every worker is online.
  • S3 and Notifications have a Test button (S3 writes and deletes a small marker object), a Reset button once edited, and ask before you leave with unsaved changes.
  • The Tailscale and Cloudflare pages show what the env set, link to the matching dashboards and show both certificates' expiry.
  • Secret fields have a copy button, and read-only fields show a lock.

Backups

  • Platform and volume backups share one format: AES-256 with openssl, then an integrity check (HMAC-SHA256). It is checked before anything is decrypted, so a corrupted file or another key is refused up front. Settings → S3 shows copyable commands that open either kind with openssl.
  • Download a backup from the Backups page (platform) or a volume's Restore dialog (volume). The browser fetches it from S3 through a link that works for 60 seconds.
  • Backups already in your S3 bucket are listed when you save the S3 settings, at startup and from Discover Backups. Entries whose file is gone are removed from the list.
  • Platform backups run every 12 hours (was 6). History is kept for 14 days (was 7): logs, metrics, notifications, tasks and the backups in S3.
  • Restoring a platform backup blocks Sage while it runs: an overlay shows what is happening, changes are refused, and the restore starts once running tasks have finished. Afterwards the page reloads, the restored settings are in use (S3 keeps the values it had), workers are resynced and backups are discovered again. Tasks that were waiting are cancelled.
  • A failed or rejected restore leaves the database as it was.
  • Backup routes say "S3 is not configured" instead of failing when S3 is off.

Stopping and upgrading

  • Sage no longer restarts or upgrades itself. Settings → Upgrade shows the release notes and the manual steps: take a backup, pause Sage until it shows it is safe to stop, then run the newer image against the same volume.
  • Pause holds queued work and refuses changes until you resume. Sage always starts unpaused.
  • On a stop signal Sage lets running tasks (a deploy, a backup) finish before it exits. Queued tasks are recorded as cancelled. Give the container a stop grace period of a few minutes. A second signal stops it at once.
  • ghcr.io/v31-dev/sage:latest now follows the newest release.

Other changes

  • Actions show their state at once. An application and its containers show deploying, stopping, backup or restoring as soon as you click, a settings section shows Applying after Save or a resync, and the restore overlay appears immediately. The sidebar icons and the pause banner refresh every 30 seconds.
  • Statuses look the same everywhere: applications, containers, workers, domains, tasks and settings use one set of badges and icons, in light and dark themes.
  • Worker sync no longer sends a notification when nothing changed.
  • Placeholders in forms are examples ("e.g. …"), defaults are prefilled and optional fields say so.
  • Dates are shown in readable form throughout.
  • The manager reaches workers' SSH and Glances metrics through its own Tailscale node, and workers ship their logs to port 9001 of the manager.

Things to know

  • Earlier backups can't be restored by this version. Platform backups are now encrypted as a whole and volume archives carry the integrity check.
  • Resources of a previous domain, account or TS_NAME stay in Cloudflare until you delete them.

Technical notes

Config

  • Required env: TS_AUTH_KEY, TS_NAME, TS_WORKER_TAG, DOMAIN, CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN, ENCRYPTION_KEY (checked in app/entrypoint.sh).
  • No longer read: HOSTNAME (now TS_NAME), WORKER_TAILSCALE_TAG (now TS_WORKER_TAG), ADMIN_EMAIL, S3_*, DISCORD_WEBHOOK. Setting rows for undefined keys are dropped on every Settings.load_all (startup and after a restore).
  • Ports: PORT (default 80) redirects to the manager address, HTTPS_PORT (default 443) serves the UI and API, METRICS_PORT (9001) takes worker logs. ENV=development serves plain HTTP on PORT (9000 in the dev compose) behind the dev Traefik and skips the hostname check.

Startup (app/entrypoint.sh, app/main.py)

  • entrypoint: tailscaled in userspace networking, wait for BackendState, tailscale up with TS_AUTH_KEY only when it is NeedsLogin (failure is fatal), wait up to 30 s for a tailnet IPv4 (fatal), tailscale set hostname TS_NAME, then exec the app. Fatal paths stop tailscaled first and print a [CRITICAL] line in Sage's log format.
  • main.py: Manager() (the settings rows decrypting is the ENCRYPTION_KEY check; InvalidToken exits), Manager.setup_domain() (Cloudflare connect, then the manager certificate; retries after 60, 240 and 960 s; a rate limit exits; a still-valid certificate lets startup continue), then HTTPS on 443 with uvicorn ssl (Certs.tls_context reloaded in place) plus redirect_app on PORT.
  • Queued after the servers are up (scheduler _AT_STARTUP): refresh_latest_release, renew_certificates, discover_backups, check_settings, check_worker_stamps.
  • api.py: in production every request whose Host is not the manager hostname gets a redirect to it (only_manager_address).

Certificates (app/services/certs.py)

  • Two Certificate objects: manager = [<TS_NAME>.core.<DOMAIN>] (never leaves the manager) and workers = [*.int.<DOMAIN>] (pushed to workers). ACME DNS-01 through Cloudflare TXT records (comment sage-acme-challenge), no contact email, renewal inside 30 days, daily at 03:00. is_valid() requires every name in the SANs, so a domain change makes both stale.
  • A Certificate keeps its parsed PEM in memory (load() at construction and after issue() writes), so validity and expiry checks and the settings status never read the disk.

Cloudflare (app/services/cloudflare.py)

  • Plain httpx client (the SDK is gone); network errors, 429 and 5xx are retried after 1 and 2 s; CloudflareError carries the API messages.
  • Records: tunnel <DOMAIN>-sage, proxied CNAME *.<DOMAIN> (comment is the tunnel name), A <TS_NAME>.core.<DOMAIN> (ttl 60, comment sage-manager), A *.int.<DOMAIN> per worker (comment sage-worker-<hostname>).

Workers (app/services/manager/workers.py)

  • setup_worker is the single idempotent heal. Stamps in <worker home>/revisions: infra (hash of worker templates, version, domain, tunnel id), certs (hash of the workers certificate) and one routing hash per application. check_stamps at startup converges workers that stayed online, which is how a changed env reaches them. A forced sync deletes the stamps first and ends with docker compose restart traefik and restart vector.
  • The infra stamp includes the version, so every new release restarts traefik, vector and cloudflared on each worker once.
  • SSH goes through tailscale nc <host> 22 as the asyncssh proxy command; Glances requests use TAILNET_HTTP_PROXY (127.0.0.2:1055, started by the entrypoint).
  • Notification only for a new worker, a changed infra stamp or repaired items; a converged worker is a log line.
  • Manager.worker_summary ({total, online}) is refreshed at startup, after each sync_workers run, after remove_worker and after a restore, so GET /api reads no database.

Task queue (app/utils/queue.py)

  • QueueState: running, paused, restoring, stopping (in memory; a restart comes back running). add_task(restore=True) is refused unless running and sets restoring, where only that task starts. The restore task ends it with end_restore(restored, follow_ups): when it took effect the queued tasks are recorded as cancelled and the follow-ups (sync_workers force, discover_s3_backups when S3 is configured) are queued with priority; otherwise the queue just resumes.
  • quiet tasks (scheduled housekeeping) write no completed or cancelled row and never make a settings section applying. The common lane is one thread (low-priority work), so its common:<section> checks run one at a time.
  • stop() (first SIGTERM, main.py Server.handle_exit) cancels queued and later-added tasks and the server exits once running tasks finish; a second signal exits at once.
  • api.py refuses changes with 503 unless running: every method except GET, HEAD and OPTIONS, apart from settings pause and resume and the read-only POST routes in _READ_ONLY_POSTS (get_available_workers, metrics, platform_backup_status, download, test).

API

  • Added: POST /api/settings/{s3|notifications}/test, POST /api/settings/pause and resume, POST /api/backups/discover, POST /api/backups/{id}/download (also under a volume), discovery_in_progress in POST /api/backups/platform_backup_status.
  • GET /api: hostname renamed name, plus worker_tag, state, restoring_from, workers ({total, online}) and settings (the status of each settings section; null when it can't be built). All of it is read from memory.
  • Removed: POST /api/settings/restart and /api/settings/upgrade (they needed the Docker socket), the Cloudflare setting, GET /api/settings/status (now part of GET /api) and the manager upgrade script.

Backups (app/utils/archive.py, templates/worker/application/backup.sh and restore.sh)

  • One format for platform and volume backups: openssl enc -aes-256-cbc -pbkdf2 -salt, then a 32-byte trailer, HMAC-SHA256 over the ciphertext. The HMAC key is SHA-256 of "sage-archive-mac:" + ENCRYPTION_KEY + newline. The passphrase reaches openssl through the environment (SAGE_BACKUP_PASSPHRASE on the manager), never argv. The MAC is checked before decrypting; a wrong key, a damaged file and an unencrypted file all fail it.
  • Keys: <path>/backups/platform/<ts>.db.enc and <path>/backups/applications/<project>/<app>/<worker>/<volume>/<ts>.tar.gz.enc, timestamp format %Y%m%d_%H%M%S.
  • discover_s3_backups lists both prefixes and makes the Backup table match (adds missing records, volume ones only for an application that exists, drops records whose object is gone). The database phase runs in the lane's thread in one transaction (insert_many, batches of 100). It runs at startup, after saving S3, from POST /api/backups/discover and after a restore.
  • Presigned links (app/services/s3.py): every call states its validity. Browser downloads use BROWSER_LINK_SECONDS (60), created per request after a HEAD check, with response-content-disposition set; the file never passes through the manager. The links given to workers (volume backup upload, volume restore download) use WORKER_LINK_SECONDS (1200) and reach curl through a --config file, not its command line.
  • Cleanup (daily 04:00) deletes S3 keys older than RETENTION_DAYS (14) under both prefixes; the platform backup cron is 0 */12 * * *.
  • Restoring a platform backup: download, check and decrypt, integrity check of the SQLite file, safety copy through the SQLite backup API, restore into the live database, reconcile schema, Settings.load_all, the manager's S3 settings written back (a restore never changes them; notification settings come from the backup), then S3 and Notifications are reloaded and the worker summary refreshed. Any failure rolls back to the safety copy.

UI

  • The app store (ui/src/stores/app.ts) owns the manager's status (GET /api) and its one poll: every 30 s and on focus, every 2 s while restoring; a refresh keeps a single timer chain when calls overlap. It also owns the maps from a status to an indicator (StateIndicator, tones in lib/tone.ts).
  • Actions set the store at once and the next poll confirms: applicationDeployStatus with the statuses of the containers touched, updateContainerStatus for a container delete, setSettingApplying, setState. The application page polls every 10 s, as does the worker page.
  • The Upgrade page renders the release notes with raw HTML allowed (DOMPurify-sanitized), so the collapsible block works.

Image and CI

  • Multi-stage Dockerfile; tailscale and tailscaled are copied from tailscale/tailscale:v1.102.5. ENTRYPOINT is /app/entrypoint.sh, which must stay executable. gdb, py-spy and memray are development-only (app/requirements-dev.txt).
  • Dependencies dropped: gunicorn, fastapi[standard] extras, uvicorn[standard] extras, docker, cloudflare, aioboto3.
  • publish.yml also tags :latest; release-pr.yml and publish.yml no longer touch sample.env. VERSION is bumped by the release PR.