Performance improvements

[michalkucharczyk]

Performance: TimescaleDB migration + parallel batch writer

Motivation

We ran jip-3-spammer against v0.3.0 with 300 nodes at realistic
rates (~258K events/s total) and hit performance problems pretty quickly — the single
batch writer couldn't keep up, events started dropping, and the write buffer filled up
within seconds of the nodes connecting.

Why

The original PostgreSQL schema uses a single flat events table with 6+ indexes. At 3M events/s from 1024 nodes, every INSERT must update all indexes, all writes hit the same table, and aggregate queries (dashboards, stats) scan the entire table. This doesn't scale.

Database: PostgreSQL → TimescaleDB

TimescaleDB is a PostgreSQL extension — same Postgres, just with time-series superpowers.

Hypertable with automatic chunking
The events table is split into 1-hour chunks automatically. Queries like "events in the last hour" scan 1-2 chunks instead of the whole table. Old data is dropped per-chunk (DROP TABLE, instant) instead of DELETE + vacuum.

32 hash partitions on node_id
Each 1-hour chunk is further split into 32 sub-chunks by hashing node_id. This spreads writes from 1024 nodes across 32 parallel physical tables — 32x less lock contention on indexes and WAL. Queries filtering by node_id only scan 1/32 of each chunk. This is a DB-internal detail, transparent to application code.

Continuous aggregates (pre-computed rollups)
Instead of running COUNT(*) over billions of rows on every API request:

• event_stats_1m — per-minute counts per node/type, refreshed every 2 min
• event_stats_1h — per-hour counts, built from the 1m aggregate (not raw events)

These are incrementally maintained by TimescaleDB — only changed chunks get re-aggregated.

Data retention pyramid

Tier	Resolution	Retention
Raw events	Full JSONB payload	7 days
1-min aggregates	Counts per node/type	30 days
1-hour aggregates	Counts per node/type	365 days

After 7 days raw event data is gone, but you still know how many events each node sent — per-minute for 30 days, per-hour for a year.

Compression after 2 hours
Columnar compression grouped by (node_id, event_type), ordered by timestamp DESC. Typical 10-20x compression ratio. Queries can skip irrelevant segments without decompressing.

Other schema changes

• event_type: INTEGER → SMALLINT (130 types fit in 2 bytes, saves ~500GB/day at full throughput)
• id BIGSERIAL PRIMARY KEY → event_id BIGINT (no PK — hypertables don't support it, and ON CONFLICT dedup is too expensive at 3M/s)
• Indexes reduced from 6+ to 2 (each index costs write throughput)
• No per-row triggers (catastrophic at high throughput) — app-level batch stats instead
• Docker image: postgres:18-alpine → timescale/timescaledb:latest-pg16

Ingestion: Work-stealing batch writer pool

Single writer replaced with 32 parallel workers sharing an Arc<Mutex<Receiver>>:

• Each worker drains events into a local batch (up to 16k events) then flushes via COPY BINARY
• Node stats aggregated per-worker and flushed every 5s (additive, concurrent-safe)
• Channel buffer: 5M events to absorb bursts from 1024 nodes
• Removed single-row store_event() — everything is batched

Server improvements

• Per-event logging downgraded from debug! to trace! to reduce log noise
• wait_for_connections() watch channel for deterministic test synchronization
• Partition health check removed (TimescaleDB handles this)

Tests

• 13 new data-driven API tests: insert real events, query endpoints, validate results
• Event encoding added for WorkPackageReceived, Authorized, Refined, GuaranteeBuilt
• All test setups updated for TimescaleDB

Bug fixes

• da_stats INTEGER overflow: Status fields (num_shards, num_preimages, preimages_size) cast to BIGINT instead of INTEGER