Repository navigation
Docker Image
ghcr.io/v31-dev/sage:0.1.3What's Changed
Features
Fixes
Full Changelog: v0.1.2...v0.1.3
Developer Notes
Important
Update as usual: take a backup, pause Sage until it shows it is safe to stop, then run the newer image against the same volume. No environment variable, compose or tailnet change is needed. On the first start each worker is synced once, and Traefik, cloudflared, Vector and Telegraf are recreated with their new log limits, so expect a short interruption on every worker.
Some log history does not carry over. The Requests page starts empty, tasks from before the update show no log lines, and the manager's old log and the stored lines of Traefik, cloudflared, Telegraf and Vector are deleted by the first daily cleanup. Application logs are kept.
Live updates
The UI no longer polls. Each browser tab keeps one connection to the manager, and what a background task, the manager's own checks or another tab changes appears without a reload.
- Status, tasks, backups, notifications and logs all update this way.
- A dot on the manager's icon in the sidebar shows the connection (connected, on a slower fallback, or not connected) and pulses with each heartbeat, every 10 seconds.
Logs
Each kind of log now has its own store and its own place in the UI.
- Application logs show the latest 1000 lines, from five minutes back by default, and follow new lines while you stay at the bottom. Filter by worker and by a From/To window, and search for text anywhere in a line.
- Markers in an application's logs show what Sage did to each container: deployed, stopped, stopped and started again for a backup, a volume restored, a worker rebooted, a container removed.
- Requests are Traefik's access lines in columns, filtered by worker, method, status, route, host, path and time.
Routed onlyhides requests for hosts no application serves, which are mostly internet scanners. - Tasks can be filtered by name, executor, status, scope (
app:test*works) and time, and show when each was queued, started and finished. A task's log dialog follows new lines, and a task with several containers can be narrowed to one container's part. - The Logs page is gone. The lines of the manager's tasks are on the Tasks page, and the rest of its output is in
docker logs. - Only Traefik's and the applications' lines are shipped. Anything else run on a worker, such as a one-off
docker run, stays there. - Log storage is bounded. Each application's logs, and the task logs, are limited to 256 MiB, and all logs together to 2 GiB or 10% of the data volume, whichever is smaller, next to the 14-day retention. On a worker, each container keeps at most five Docker log files of 10 MB. An application's container gets this limit at its next deploy.
Workers
- A worker is online when its SSH server answers through Tailscale, the path Sage's commands take, instead of when it answers
tailscale ping. It still goes offline after two misses in a row, about a minute. - After the manager starts, a worker shows as checking until the first sync has looked at it.
- The Workers page shows each worker's latest minute of metrics from the pushed list.
Fixes
- Logs a worker pushes that the manager can't store are now refused, so Vector sends them again. They used to be lost.
- A Cloudflare token that is rejected, or a domain it can't use, now stops startup at once with the reason. Sage retries only failures that can clear by themselves.
Technical notes
Config
- No change to the env vars,
sample.env,docker-compose.yml, theDockerfileorapp/entrypoint.sh. - Added to
app/config.py:UPDATES_COALESCE_SECONDS3,UPDATES_PING_INTERVAL_SECONDS10,UPDATES_PING_TIMEOUT_SECONDS10,LOG_SHARD_MAX_BYTES256 MiB,LOG_TOTAL_MAX_BYTES2 GiB,LOG_TOTAL_MAX_DISK_SHARE0.10,LOG_TRIM_TO_SHARE0.9,LOG_KEEP_NEWEST_ROWS1000,LOG_SIZE_CHECK_SECONDS30,LOG_TRIM_BATCH_ROWS10000,LOG_SHARD_VACUUM_RATIO1.25,LIST_MAX_ROWS1000,LOG_INTAKE_MAX_BYTES8 MiB,REPORT_UNKNOWN_LIMIT100,WORKER_METRICS_STALE_SECONDS180,SSH_BANNER_TIMEOUT_SECONDS3,SSH_CLOSE_WAIT_SECONDS5. - Renamed:
OFFLINE_AFTER_MISSED_PINGSisOFFLINE_AFTER_MISSED_CHECKS, still 2.
Startup (app/main.py, app/api.py)
- The UI and API server runs
app_asgi(build_asgi_app):socketio.ASGIAppin front of FastAPI, insideOnlyManagerAddressoutside development. That class replaces theonly_manager_addressmiddleware: another host name gets the redirect, and a WebSocket is closed with code 1008.redirect_appisapp_redirect. fastapi_middlewareis gone from both apps, with the request id and theX-Task-IDheader. uvicorn's access log runs instead, andAccessLogFilterleaves out the routes markedopenapi_extra=NO_ACCESS_LOGandSOCKET_IO_PATH. The ingest server no longer setsaccess_log=False, which would silence the logger every server shares.ManagerstartsLogs().start_capture()and callsload_workers(), which replacesrefresh_worker_summary().refresh_worker_pathsjoins_AT_STARTUP.
Pushed updates (app/services/updates.py, app/routes/updates.py, ui/src/lib/updates.ts)
- Socket.IO at
/api/socket.io. Client events:subscribe,unsubscribe. Server events:state{topic, data}andtopic_error{topic, status}. - Topics, each the JSON of its REST route:
info,workers,worker:<hostname>,application:<project>/<name>,tasks,backup-status,backups,release. Feeds of rows:notifications,task-history,task-logs:<task>ortask-logs:<task>/<part>,application-logs:<project>-<app>,requests. - A change waits
UPDATES_COALESCE_SECONDS, then the topics that depend on it are computed onUPDATES_EXECUTOR(2 threads). The server pings everyUPDATES_PING_INTERVAL_SECONDSand drops a connection afterUPDATES_PING_TIMEOUT_SECONDS. same_originaccepts a page only if itsOriginis the scheme and host the request arrived with. A client that sends noOriginis accepted.
Endpoints
- Added:
GET /api/requests/(worker,host,method,status,path,min_duration_ms,routed,from_ts,to_ts,after_id,limit),GET /api/projects/{project}/applications/{application}/logs(hostname,search,from_ts,to_ts,after_id,limit),GET /api/tasks/history,GET /api/tasks/history/filters,GET /api/tasks/{task_id}/parts,GET /api/tasks/{task_id}/logs(sub_task_id,after_id).limitis at mostLIST_MAX_ROWS. - Removed:
GET /api/workers/{hostname}/logs/{container},workersfromGET /api,completedandlimitfromGET /api/tasks, and*as a hostname. - POST to GET:
.../applications/{application}/get_available_workers,.../applications/{application}/metrics,/api/backups/platform_backup_status, and/{backup}/download, also under a volume._READ_ONLY_POSTSis gone, so the settings tests, which are POSTs, answer 503 while Sage is paused. GET /api/taskscarriesqueued_atandstarted_at.GET /api/workersisManager.worker_list:hostname,ip,online,status,path,container_count,infra,metrics.GET /api/workers/{hostname}addsmetrics.onlineisnulluntil a sync has looked at the worker, andstatuscan bechecking. Deleting a container on such a worker fails with a message to try again.POST /api/vector/logsanswers 204, 400 withoutX-Node-Name, 413 aboveLOG_INTAKE_MAX_BYTES, and 503 when a write fails. Vector sends a batch again on a 5xx, so a file already written gets it twice.
Logs (app/services/logs/, replacing services/logs.py)
- Files under
logs/in the data volume:<project>-<app>.dbfor each application,requests.db(Traefik's access lines in columns:ts,worker,host,method,path,status,duration_ms,router,service,client,size) andtasks.db(task_id,sub_task_id,ts,level,logger,message). - A new application shard indexes with the FTS5
trigramtokenizer. A shard from an earlier release keeps its tokenizer and is scanned when searched. stored_timekeeps every time as UTC with nine digits of fraction andZ, so that times compare as text.- Markers are lines in the stream
sage(MARKER_STREAM), written bymarkandLogs.write_markerwhen an action ends. Logs.trim_shardandLogs.trim_totalrun after a write, at most everyLOG_SIZE_CHECK_SECONDS, and inLogs.cleanup.Logs.cleanupdrops every file that is neither inLogs.FILESnor an application's shard. This removessage.db,traefik.db,cloudflared.db,telegraf.dbandvector.db.- The manager stores only the lines of its queued tasks. The lines of a quiet task are held by
TaskLinesand written only if it fails. Everything goes to stdout, withtaskortask/partfromTaskIdFilter.
Database (app/services/db/)
Worker.onlineis dropped.Task.started_atis added (null for a task cancelled before it started),Task.created_atis when the task was queued, andTask.finished_atis indexed.Database._sync_columnsnow also drops a column no model has, at startup and after a platform restore.
Workers (app/services/manager/workers.py, decisions.py, app/services/tailscale.py)
vector.yml: theshippedtransform keepstraefikand containers with thesage.deployed_atlabel, set since v0.0.22, and reduces each event tocontainer_name,timestamp,streamandmessage.batch.timeout_secsis 10 (was 30) andbatch.max_bytesis 4194304.- Every Compose template sets
logging:json-file,max-size: 10m,max-file: "5". The stack gets it at the first sync, because the infra stamp covers the version and the templates. Tailscale.reachablereads the SSH banner throughtailscale nc <hostname> 22, andonline_statesuses it.tailscale pingnow only finds the path, inTailscale.refresh_paths.onlineis held in memory:Manager.worker_listis derived fromWorkerInputsbyworker_entry, and replacesworker_summaryandworker_reboots.services/reports.pyisservices/worker_reports.py, andReportsisWorkerReports.
Task queue (app/utils/queue.py, app/services/manager/queue.py)
QueuedTaskhasqueued_atandstarted_at.admitandnext_to_startare pure functions, andTaskQueuetakes anotifyhook, called after each change.LANESreplacesPLATFORM_EXECUTOR,COMMON_EXECUTOR,APP_EXECUTORandMETRICS_EXECUTOR.- New job:
refresh_worker_paths, every 60 s and at startup,commonscope, quiet.Metrics.flush_workersnow runs throughManager.flush_worker_metrics. container_partrecords a container'sEventand setssub_task_idfor its deploy, stop, delete, backup or restore. Routes no longer pass atask_idtoadd_task.
Cloudflare and Tailscale
- Sage creates no new resources: the tunnel, its ingress rules, the DNS records and the certificates are unchanged.
Manager.setup_domainretries only whatis_transientaccepts (a network error, HTTP 5xx or 429), after 1, 4 and 16 minutes. Anything else exits at once.
Backups
- The format, the S3 keys,
backup.shandrestore.share unchanged.
UI
ui/src/services/api/replacesservices/api.ts, with a module for each resource.lib/updates.tsholds the connection, anduseFeedList(lib/feedList.ts) is the one list contract.- Added:
FilterBar.vue,TaskLogs.vue,SocketStatus.vue. Removed:pages/Logs.vuewith its/logsroute,LogViewer.vue,RestoreLogsDialog.vue.
Image and CI
app/requirements.txtaddswebsockets==15.0.1,python-socketio==5.17.0andpython-engineio==4.14.0.ui/package.jsonaddssocket.io-client^4.8.4.ci.ymlskips draft PRs and runs onready_for_review, and the backend job addsautopep8 --diff --recursive --exit-code ..- All three workflows pin their actions by commit SHA.
release-pr.ymlends the PR body with## Developer Notes.