Operating toard

한국어로 보기

Operating toard

Use this page for the recurring checks and safe decisions after deployment. The repository's deployment guide is the canonical reference for platform-specific settings.

Check service health first

curl -fsS http://localhost:3000/api/health
curl -fsS http://localhost:3000/api/ready
docker compose ps -a

/api/health is the process liveness check. /api/ready also checks database connectivity, so it is the endpoint to use before sending users to a newly deployed server. In a successful Compose installation, app and postgres run and migrate exits with status 0.

For Kubernetes, use /api/health as the liveness probe and /api/ready as the readiness probe; only ready pods should receive traffic. See the deployment guide.

Keep pricing current

toard calculates cost from LiteLLM pricing data. In self-hosted deployments, the built-in scheduler runs sync-pricing daily using the organization timezone. The Admin → System page shows its state and permits a manual sync.

• Check whether PRICING_AUTO_SYNC=off has intentionally disabled automatic sync.
• When using an external scheduler instead, prevent duplicate work by disabling the built-in scheduler.
• Treat a $0 cost warning as a signal to check the sync state; missing pricing data results in zero calculated cost.

External calls to /api/cron/* must be protected with CRON_SECRET in production. See the scheduler reference for Vercel and GitHub Actions details.

Update the server deliberately

The optional Compose updater is for Compose deployments only. It updates TOARD_TAG, pulls images, runs migrations, restarts the app, then checks health, readiness, and version. It needs Docker-socket and deployment-directory access, so keep its API private to the Compose network.

The first version has no automatic container rollback. If an update fails, return TOARD_TAG to the previous value and follow the documented Compose pull and restart procedure. Helm and Kubernetes deployments use their own image-release workflow, not the Compose updater.

Keep deployment settings, including AUTH_SECRET, in the deployment .env; do not paste their values into tickets, logs, or this Wiki. Full updater and rollback notes are in docs/DEPLOY.md.

Manage shim collection and updates

The Rust shim collects usage from local session files and sends normalized events to /api/v1/events. For users whose Desktop or IDE use does not pass through the wrapper, install the recommended scheduled collection:

toard-shim daemon install
toard-shim daemon status
toard-shim doctor

The shim checks for a release in the background on a six-hour throttle. It verifies SHA256SUMS before an atomic replacement. Run an immediate check with:

toard-shim update
toard-shim version

Set TOARD_SHIM_AUTO_UPDATE=0 only when your release process requires it. Air-gapped or mirrored environments can set TOARD_SHIM_RELEASE_BASE. The shim README is the source of truth.

Treat ingest tokens as per-user credentials

Tokens authenticate event ingestion and determine the server-side user attribution. Issue a new token before revoking an old one, then revoke the specific token for an offboarded user, a lost device, or suspected exposure. Tokens can also expire; periodic rotation is recommended.

Keep more than one administrator, and do not share an invite link and an ingest token through the same channel. See Privacy and security for the security boundary.

Back up what is needed to recover

Use the backup and restore process approved for your Postgres and deployment platform. At minimum, include the Postgres data and the deployment configuration required to reconnect the application. If content collection is enabled, retain the content KEK separately from the database backup: without it, existing encrypted content cannot be decrypted.

Test a restore in a non-production environment. Do not make production database changes directly as part of routine recovery, and never run docker compose down -v unless the data deletion has been explicitly approved.

Use ClickHouse only when needed

Postgres is the default backend. ClickHouse is opt-in for events and analytics; accounts, teams, authentication, and tokens remain in Postgres. Start with the documented STORAGE_BACKEND=clickhouse deployment configuration rather than treating ClickHouse as a complete replacement for Postgres.

The raw query path remains the default. Before enabling CLICKHOUSE_READ_ROLLUP=1, verify that duplicate rows are zero, the Postgres outbox is stably drained, raw-versus-rollup differences are zero, /api/ready returns 200, and ClickHouse is healthy. Follow the exact rollup runbook for the full validation and rollback path.

Non-negotiable safety boundaries

• Do not expose tokens, passwords, connection strings, KEKs, or secrets in output or documentation.
• Do not run destructive SQL against a production database.
• Plan schema changes as expand then contract so old and new application instances can coexist during a rollout.
• Do not delete Compose volumes without explicit approval.