Skip to content

Release v0.1.3

Latest

Choose a tag to compare

@github-actions github-actions released this 02 Nov 01:35
3c2ad82

Docker Image

ghcr.io/v31-dev/sage:0.1.3

What's Changed

Features

Fixes

  • fix tailscale ping; other optimizations by @sidxdev in #86

Full Changelog: v0.1.2...v0.1.3

Developer Notes

Important

Update as usual: take a backup, pause Sage until it shows it is safe to stop, then run the newer image against the same volume. No environment variable, compose or tailnet change is needed. On the first start each worker is synced once, and Traefik, cloudflared, Vector and Telegraf are recreated with their new log limits, so expect a short interruption on every worker.

Some log history does not carry over. The Requests page starts empty, tasks from before the update show no log lines, and the manager's old log and the stored lines of Traefik, cloudflared, Telegraf and Vector are deleted by the first daily cleanup. Application logs are kept.

Live updates

The UI no longer polls. Each browser tab keeps one connection to the manager, and what a background task, the manager's own checks or another tab changes appears without a reload.

  • Status, tasks, backups, notifications and logs all update this way.
  • A dot on the manager's icon in the sidebar shows the connection (connected, on a slower fallback, or not connected) and pulses with each heartbeat, every 10 seconds.

Logs

Each kind of log now has its own store and its own place in the UI.

  • Application logs show the latest 1000 lines, from five minutes back by default, and follow new lines while you stay at the bottom. Filter by worker and by a From/To window, and search for text anywhere in a line.
  • Markers in an application's logs show what Sage did to each container: deployed, stopped, stopped and started again for a backup, a volume restored, a worker rebooted, a container removed.
  • Requests are Traefik's access lines in columns, filtered by worker, method, status, route, host, path and time. Routed only hides requests for hosts no application serves, which are mostly internet scanners.
  • Tasks can be filtered by name, executor, status, scope (app:test* works) and time, and show when each was queued, started and finished. A task's log dialog follows new lines, and a task with several containers can be narrowed to one container's part.
  • The Logs page is gone. The lines of the manager's tasks are on the Tasks page, and the rest of its output is in docker logs.
  • Only Traefik's and the applications' lines are shipped. Anything else run on a worker, such as a one-off docker run, stays there.
  • Log storage is bounded. Each application's logs, and the task logs, are limited to 256 MiB, and all logs together to 2 GiB or 10% of the data volume, whichever is smaller, next to the 14-day retention. On a worker, each container keeps at most five Docker log files of 10 MB. An application's container gets this limit at its next deploy.

Workers

  • A worker is online when its SSH server answers through Tailscale, the path Sage's commands take, instead of when it answers tailscale ping. It still goes offline after two misses in a row, about a minute.
  • After the manager starts, a worker shows as checking until the first sync has looked at it.
  • The Workers page shows each worker's latest minute of metrics from the pushed list.

Fixes

  • Logs a worker pushes that the manager can't store are now refused, so Vector sends them again. They used to be lost.
  • A Cloudflare token that is rejected, or a domain it can't use, now stops startup at once with the reason. Sage retries only failures that can clear by themselves.

Technical notes

Config

  • No change to the env vars, sample.env, docker-compose.yml, the Dockerfile or app/entrypoint.sh.
  • Added to app/config.py: UPDATES_COALESCE_SECONDS 3, UPDATES_PING_INTERVAL_SECONDS 10, UPDATES_PING_TIMEOUT_SECONDS 10, LOG_SHARD_MAX_BYTES 256 MiB, LOG_TOTAL_MAX_BYTES 2 GiB, LOG_TOTAL_MAX_DISK_SHARE 0.10, LOG_TRIM_TO_SHARE 0.9, LOG_KEEP_NEWEST_ROWS 1000, LOG_SIZE_CHECK_SECONDS 30, LOG_TRIM_BATCH_ROWS 10000, LOG_SHARD_VACUUM_RATIO 1.25, LIST_MAX_ROWS 1000, LOG_INTAKE_MAX_BYTES 8 MiB, REPORT_UNKNOWN_LIMIT 100, WORKER_METRICS_STALE_SECONDS 180, SSH_BANNER_TIMEOUT_SECONDS 3, SSH_CLOSE_WAIT_SECONDS 5.
  • Renamed: OFFLINE_AFTER_MISSED_PINGS is OFFLINE_AFTER_MISSED_CHECKS, still 2.

Startup (app/main.py, app/api.py)

  • The UI and API server runs app_asgi (build_asgi_app): socketio.ASGIApp in front of FastAPI, inside OnlyManagerAddress outside development. That class replaces the only_manager_address middleware: another host name gets the redirect, and a WebSocket is closed with code 1008. redirect_app is app_redirect.
  • fastapi_middleware is gone from both apps, with the request id and the X-Task-ID header. uvicorn's access log runs instead, and AccessLogFilter leaves out the routes marked openapi_extra=NO_ACCESS_LOG and SOCKET_IO_PATH. The ingest server no longer sets access_log=False, which would silence the logger every server shares.
  • Manager starts Logs().start_capture() and calls load_workers(), which replaces refresh_worker_summary(). refresh_worker_paths joins _AT_STARTUP.

Pushed updates (app/services/updates.py, app/routes/updates.py, ui/src/lib/updates.ts)

  • Socket.IO at /api/socket.io. Client events: subscribe, unsubscribe. Server events: state {topic, data} and topic_error {topic, status}.
  • Topics, each the JSON of its REST route: info, workers, worker:<hostname>, application:<project>/<name>, tasks, backup-status, backups, release. Feeds of rows: notifications, task-history, task-logs:<task> or task-logs:<task>/<part>, application-logs:<project>-<app>, requests.
  • A change waits UPDATES_COALESCE_SECONDS, then the topics that depend on it are computed on UPDATES_EXECUTOR (2 threads). The server pings every UPDATES_PING_INTERVAL_SECONDS and drops a connection after UPDATES_PING_TIMEOUT_SECONDS.
  • same_origin accepts a page only if its Origin is the scheme and host the request arrived with. A client that sends no Origin is accepted.

Endpoints

  • Added: GET /api/requests/ (worker, host, method, status, path, min_duration_ms, routed, from_ts, to_ts, after_id, limit), GET /api/projects/{project}/applications/{application}/logs (hostname, search, from_ts, to_ts, after_id, limit), GET /api/tasks/history, GET /api/tasks/history/filters, GET /api/tasks/{task_id}/parts, GET /api/tasks/{task_id}/logs (sub_task_id, after_id). limit is at most LIST_MAX_ROWS.
  • Removed: GET /api/workers/{hostname}/logs/{container}, workers from GET /api, completed and limit from GET /api/tasks, and * as a hostname.
  • POST to GET: .../applications/{application}/get_available_workers, .../applications/{application}/metrics, /api/backups/platform_backup_status, and /{backup}/download, also under a volume. _READ_ONLY_POSTS is gone, so the settings tests, which are POSTs, answer 503 while Sage is paused.
  • GET /api/tasks carries queued_at and started_at. GET /api/workers is Manager.worker_list: hostname, ip, online, status, path, container_count, infra, metrics. GET /api/workers/{hostname} adds metrics. online is null until a sync has looked at the worker, and status can be checking. Deleting a container on such a worker fails with a message to try again.
  • POST /api/vector/logs answers 204, 400 without X-Node-Name, 413 above LOG_INTAKE_MAX_BYTES, and 503 when a write fails. Vector sends a batch again on a 5xx, so a file already written gets it twice.

Logs (app/services/logs/, replacing services/logs.py)

  • Files under logs/ in the data volume: <project>-<app>.db for each application, requests.db (Traefik's access lines in columns: ts, worker, host, method, path, status, duration_ms, router, service, client, size) and tasks.db (task_id, sub_task_id, ts, level, logger, message).
  • A new application shard indexes with the FTS5 trigram tokenizer. A shard from an earlier release keeps its tokenizer and is scanned when searched.
  • stored_time keeps every time as UTC with nine digits of fraction and Z, so that times compare as text.
  • Markers are lines in the stream sage (MARKER_STREAM), written by mark and Logs.write_marker when an action ends.
  • Logs.trim_shard and Logs.trim_total run after a write, at most every LOG_SIZE_CHECK_SECONDS, and in Logs.cleanup.
  • Logs.cleanup drops every file that is neither in Logs.FILES nor an application's shard. This removes sage.db, traefik.db, cloudflared.db, telegraf.db and vector.db.
  • The manager stores only the lines of its queued tasks. The lines of a quiet task are held by TaskLines and written only if it fails. Everything goes to stdout, with task or task/part from TaskIdFilter.

Database (app/services/db/)

  • Worker.online is dropped. Task.started_at is added (null for a task cancelled before it started), Task.created_at is when the task was queued, and Task.finished_at is indexed.
  • Database._sync_columns now also drops a column no model has, at startup and after a platform restore.

Workers (app/services/manager/workers.py, decisions.py, app/services/tailscale.py)

  • vector.yml: the shipped transform keeps traefik and containers with the sage.deployed_at label, set since v0.0.22, and reduces each event to container_name, timestamp, stream and message. batch.timeout_secs is 10 (was 30) and batch.max_bytes is 4194304.
  • Every Compose template sets logging: json-file, max-size: 10m, max-file: "5". The stack gets it at the first sync, because the infra stamp covers the version and the templates.
  • Tailscale.reachable reads the SSH banner through tailscale nc <hostname> 22, and online_states uses it. tailscale ping now only finds the path, in Tailscale.refresh_paths.
  • online is held in memory: Manager.worker_list is derived from WorkerInputs by worker_entry, and replaces worker_summary and worker_reboots.
  • services/reports.py is services/worker_reports.py, and Reports is WorkerReports.

Task queue (app/utils/queue.py, app/services/manager/queue.py)

  • QueuedTask has queued_at and started_at. admit and next_to_start are pure functions, and TaskQueue takes a notify hook, called after each change.
  • LANES replaces PLATFORM_EXECUTOR, COMMON_EXECUTOR, APP_EXECUTOR and METRICS_EXECUTOR.
  • New job: refresh_worker_paths, every 60 s and at startup, common scope, quiet. Metrics.flush_workers now runs through Manager.flush_worker_metrics.
  • container_part records a container's Event and sets sub_task_id for its deploy, stop, delete, backup or restore. Routes no longer pass a task_id to add_task.

Cloudflare and Tailscale

  • Sage creates no new resources: the tunnel, its ingress rules, the DNS records and the certificates are unchanged.
  • Manager.setup_domain retries only what is_transient accepts (a network error, HTTP 5xx or 429), after 1, 4 and 16 minutes. Anything else exits at once.

Backups

  • The format, the S3 keys, backup.sh and restore.sh are unchanged.

UI

  • ui/src/services/api/ replaces services/api.ts, with a module for each resource. lib/updates.ts holds the connection, and useFeedList (lib/feedList.ts) is the one list contract.
  • Added: FilterBar.vue, TaskLogs.vue, SocketStatus.vue. Removed: pages/Logs.vue with its /logs route, LogViewer.vue, RestoreLogsDialog.vue.

Image and CI

  • app/requirements.txt adds websockets==15.0.1, python-socketio==5.17.0 and python-engineio==4.14.0. ui/package.json adds socket.io-client ^4.8.4.
  • ci.yml skips draft PRs and runs on ready_for_review, and the backend job adds autopep8 --diff --recursive --exit-code ..
  • All three workflows pin their actions by commit SHA.
  • release-pr.yml ends the PR body with ## Developer Notes.