Skip to content

Operations

Savvas Leousis edited this page Sep 26, 2026 · 2 revisions

Operations

Run Stonks unattended: the daily loop, health checks, alerts and P&L. Scheduling and backups have their own page: Scheduling and Backups. Server setup: Deploy.

flowchart LR
  S[Scheduler] --> I[ingest prices<br/>quality checks]
  I --> T[tick]
  T --> R[report]
  S --> H[health]
  I & T & H -->|problem| A[Alert<br/>log, alerts table, webhook]
Loading

The daily loop by hand

uv run stonks ingest prices --tickers AAPL.US,MSFT.US
uv run stonks tick
uv run stonks health --notify
uv run stonks report
  • All four are safe to rerun.
  • Times are UTC. tick defaults --as-of to today's UTC date, so run it after the day's bars land.
  • Run uv run stonks db init once after every upgrade, before the first tick.

Health

stonks health exits 0 when healthy, 1 when not. --notify also sends an alert. Settings under [production.health].

Check Fails when
freshness:<ticker> Latest daily bar older than max_bar_age_days (4)
stuck_ticks A tick running longer than stuck_tick_minutes (60)
stuck_ingest_runs An ingest running longer than stuck_ingest_minutes (180)
ingest_failures An ingest ended in error in the last ingest_failure_lookback_hours (24)

Tickers come from --tickers or [production].universe. The API serves the same report at GET /api/health/report, and GET /api/health is the liveness probe.

Metrics

python -m stonks.scheduling metrics prints Prometheus text: tick counts and duration, orders and rejections, job queue, each scheduled job's last success and next run, and the scheduler heartbeat. Add --data-age for bar age buckets (it opens the lake, so not while stonks serve runs). The API has no metrics route yet.

A useful alert: time() - stonks_scheduled_job_last_success_timestamp_seconds{job="tick"} > 26 * 3600 on weekdays.

Data quality

Every price ingest checks its bars. Bad rows go to quarantined_bars; odd ones are warnings. An alert goes out when a run quarantines a bar or many tickers carry warnings. Look at ingest_runs.quality_json for the counts. Runbook: docs/runbooks/data-stale.md.

P&L

uv run stonks pnl                        # real portfolio, from the start
uv run stonks pnl --since 2026-09-01     # show only recent rows
uv run stonks pnl --strategy <id>        # a shadow strategy's virtual P&L

Columns: date, value, change, daily, cumulative, drawdown. Cumulative and drawdown are always measured from the start.

HTML report

uv run stonks report                                  # writes data/reports/report.html
uv run stonks report --strategy <id> --since 2026-08-01 --out my.html
uv run stonks report --backtest momentum --start 2024-01-01 --end 2026-09-01 --tickers AAPL.US

Equity curve, drawdown, positions, orders and fills, survival verdicts, shadow P&L and drift, in one file. --backtest writes a tear sheet instead.

Alerts

Configure under [notify]. Put the webhook URL in .env because it usually contains a token.

[notify]
backends = ["log", "store", "webhook"]
min_level = "warning"
STONKS_NOTIFY_WEBHOOK_URL=https://hooks.slack.com/services/...
From Level When
tick error The tick raised
tick warning Status partial, or the broker rejected orders
health --notify error Any check failed
ingest warning Bars quarantined, or 5 or more tickers with warnings
scheduler error A job failed or missed its deadline
  • store keeps every alert in the alerts table, which the console shows. Secrets are scrubbed first.
  • The webhook gets JSON with a one-line text field, so Slack and Mattermost show it as is. A failing webhook is logged and never breaks the tick.
  • Per-user notifications (Web Push, email) are separate; see Accounts and Modes.

Logs are JSON lines (structlog), one event per line. jq reads them well.

Runbooks

In the repo under docs/runbooks/: data-stale.md, restore.md and deploy-failed.md.

Clone this wiki locally