Architecture

Architecture

Two paths and three stores. That's the whole system.

Data path (ingest)

POST /v0/events  →  Gatherer (goroutine + channel, batches on max(N rows, 5s))  →  ClickHouse

Each row is validated against its datasource schema; bad rows are quarantined, not dropped silently.

Query path

GET /v0/pipes/{name}.json  →  parse {{Type(param, default)}} template  →  bind params as
ClickHouse query parameters (never string-interpolated)  →  ClickHouse HTTP  →  formatted result

The three stores

Store	Job
ClickHouse	event data, materialized views, query execution, pipe_stats
Redis	metadata registry (datasource/pipe defs, tokens, deploy state) and hot cache. AOF-persisted, system of record. No Postgres (ADR 0001).
git	.datasource / .pipe files — the real source of truth. The registry is rebuildable with tinyraven deploy.

The keystone: Datasource

Datasource (internal/model/model.go) is the one type every subsystem shares — the schema contract. Deploy reads a .datasource file into it and builds the ClickHouse table; ingest validates rows against it; the API registry serves it; pipes query the table it defines. It sits exactly on the seam between the control plane (deploy, registry) and the data plane (ingest, query) — which is why it's the most-connected node in the codebase.

Branching

One ClickHouse database per git branch (tr_{branch}). tinyraven deploy detects the current branch and targets the matching DB. Breaking migrations use shadow table → MV backfill → atomic EXCHANGE TABLES.