History and ClickHouse

History and ClickHouse (per-guild analytics)

Per-guild and per-user questions are high cardinality and must never hit Prometheus (invariant 2/7). The analytical path is fully separate: hooks emit per-guild events to an EventSink, in parallel to the operational counters, sharing no labels or storage.

Enable with both enable_per_guild=True and clickhouse_dsn=.... With the flag off (the default), the sink is a NullSink and there is zero overhead.

Event shape

The hooks emit (see core/instrumentation.py):

{"ts": "<iso8601>", "event": "interaction|app_command",
 "guild_id": "123", "type": "application_command", "command": "ping",
 "duration_ms": 12.5}

duration_ms is the precise command duration (interaction receipt to completion). interaction events carry type; app_command events carry command + duration_ms.

EventSink contract (history/sink.py)

• EventSink.record(event) is async, non-blocking, and never raises into the caller.
• BatchingSink: a bounded asyncio.Queue (default 10k) + a background flusher. record does a put_nowait; on overflow it increments dropped and returns (drop-and-count, never blocks the bot, invariant 3). The worker collects up to batch_size (default 100) or waits flush_interval (default 5s), then calls _flush. A wake event makes aclose() drain promptly without losing the in-flight item.
• NullSink: discards everything.

ClickHouseSink (history/clickhouse.py)

Subclass of BatchingSink using clickhouse-connect's async client (the clickhouse extra). On first connect it runs:

CREATE TABLE IF NOT EXISTS argus_events (
  ts String,
  event LowCardinality(String),
  guild_id String,
  type LowCardinality(String),
  command String,
  duration_ms Float64
) ENGINE = MergeTree() ORDER BY (guild_id, ts)

ts is stored as the ISO-8601 string the hooks emit and parsed in queries with parseDateTimeBestEffort, which keeps inserts trivial and timezone-safe. Inserts are batched (client.insert). The client is created lazily, so importing the module does not require the optional dependency; tests inject a fake client via client_factory. For very high write concurrency you can also enable ClickHouse server-side async inserts; it is complementary to the client-side batching here.

Queries (history/query.py)

AnalyticsQuery(client, table="argus_events"), all parameterised:

Method	Returns
interaction_volume(guild_id, since_days=30)	[(day, count)]
top_commands(guild_id, limit=10)	[(command, count)]
command_stats(guild_id, limit=50)	[(command, count, avg_ms)]
avg_duration(guild_id)	float overall avg ms

These back /api/analytics/*, which is mounted only when analytics is enabled and fails closed without a dashboard_auth_token.

Testing against a real server

The clickhouse CI job runs the integration-marked round-trip test against a ClickHouse service container (CLICKHOUSE_DSN). Locally:

docker run -d --rm -p 8123:8123 clickhouse/clickhouse-server:24.8
pip install -e ".[clickhouse]" --group dev
CLICKHOUSE_DSN=http://localhost:8123 pytest -m integration -v