Skip to content

Scheduling and Backups

Savvas Leousis edited this page Sep 26, 2026 · 1 revision

Scheduling and Backups

Stonks has a built-in scheduler, so you don't write cron entries for the daily loop. It follows the exchange calendar, catches up missed runs, alerts when a job misses its deadline, and can ping an external monitor.

flowchart LR
  CAL[Exchange calendar<br/>NYSE close] --> I[close + 30 min<br/>ingest_prices]
  I --> T[close + 45 min<br/>tick]
  T --> R[close + 90 min<br/>report]
  H[every 4 hours<br/>health]
  I & T -. start / ok / fail .-> P[Dead-man monitor]
  W[Watchdog] -->|deadline missed| A[Alert]
Loading

Commands

uv run python -m stonks.scheduling run          # run until Ctrl+C or SIGTERM
uv run python -m stonks.scheduling next         # each job's next fire time
uv run python -m stonks.scheduling runs         # recent runs and their status
uv run python -m stonks.scheduling run-now tick --as-of 2026-09-25
uv run python -m stonks.scheduling check        # deadline check once; exit 1 on a miss
uv run python -m stonks.scheduling metrics      # Prometheus text metrics

Run exactly one scheduler. A second run finds the lock taken and exits with code 2.

Default jobs

Job When Deadline
ingest_prices NYSE close + 30 min, trading days 60 min
tick NYSE close + 45 min, trading days 60 min
report NYSE close + 90 min, trading days none
health every 4 hours none

These are the only four actions. Listing any [[scheduler.jobs]] in the TOML replaces the whole list:

[[scheduler.jobs]]
name = "tick"
action = "tick"
deadline_minutes = 60
ping_url_env = "STONKS_PING_TICK"      # the URL itself lives in .env
trigger = { type = "session", calendar = "XNYS", anchor = "close", offset_minutes = 45 }

[[scheduler.jobs]]
name = "crypto_tick"
action = "tick"
params = { tickers = ["BTC-USD.CC", "ETH-USD.CC"] }
trigger = { type = "daily", at = "00:30", timezone = "UTC" }

Triggers: session (minutes after a calendar's open or close), daily (a local time in a time zone) and interval (every N minutes). The tick and price ingest skip days on which no instrument in the universe trades; crypto trades every day.

Where jobs run

DuckDB allows one writer per lake file, so [scheduler].backend decides where jobs run:

Backend Jobs run Use when
api Through the running API (STONKS_API_URL, STONKS_API_TOKEN) stonks serve is running, as in the Compose stack
local In the scheduler process Nothing else holds the lake
in_process Inside the API process Built, but stonks serve does not start it yet

auto (the default) picks api when an API URL is set, else local.

Runs, catch-up, alerts

  • Every fire is one row in scheduled_runs, keyed by job and date, so a fire never runs twice.
  • On start, catch_up = "latest" (default) runs only the most recent missed fire; none and all exist too.
  • A failed run is not retried. It alerts; rerun it with run-now.
  • The watchdog alerts once per job that has no successful run by its deadline.
  • A job with ping_url_env pings a healthchecks.io-style URL on start, success and failure. The default jobs set none, so add it in your job list.

Other recurring jobs

The scheduler has no action for these yet, so run them from cron or a timer:

uv run python -m stonks.ops backup                # nightly
uv run python -m stonks.notify deliver            # every minute or two, if you use notifications
uv run python -m stonks.connections sync --due    # if you use broker connections

Backups

uv run python -m stonks.ops backup            # back up, verify, prune
uv run python -m stonks.ops list
uv run python -m stonks.ops verify stonks-20260926T220000Z
uv run python -m stonks.ops restore stonks-20260926T220000Z --data-dir /srv/stonks/data-restored
uv run python -m stonks.ops prune
  • A backup is one folder with the state DB, the lake, Parquet bars (if used), artifacts and a manifest.json of hashes and row counts. Each store is a consistent copy even while Stonks runs.
  • It goes to backups/ next to the lake, or --dest. Retention keeps the newest backup of each of the last 7 days, 4 weeks and 12 months.
  • If the lake is in use (stonks serve holds it), stop the server for the backup.
  • restore refuses to overwrite data unless you pass --force, which moves old files aside instead of deleting them. It runs the migrations, so an older backup comes up on the current schema.

On the server, the deploy scripts also send an encrypted off-site copy every night with restic and test a restore every month. See Deploy.

Runbook: docs/runbooks/restore.md in the repo.

Clone this wiki locally