feat: durable market-data time series via TimescaleDB

[emrebulutlar]

Problem

Everything on the market-data plane truncated on a market gateway restart: chart candles (all 6 intervals, 500-candle in-memory rings), the recent-trades tape, and the 24h ticker. Worse, the ticker's "24h" figures were since-boot accumulations (open = first trade after boot, volume never rolls off).

Design

The database is the source of truth; memory exists only for performance.

• Raw aggregated trades -> trades hypertable (TimescaleDB, new marketdata DB on the system PG16)
• Candles derived in-database: candles_1m continuous aggregate from trades, then hierarchical caggs 5m/15m/1h/4h/1d, all with real-time aggregation (materialized_only = false) so the current bucket is included at query time
• REST /api/candles, WS on-connect CANDLE_HISTORY, trade tape: DB-first with the in-memory rings as fallback; live per-trade CANDLE_UPDATE stays in-memory
• True rolling-24h ticker recomputed from the 1m cagg every 5s and merged with live trades
• Boot hydration seeds the rings (incl. the in-flight bucket) before egress polling starts

Uptime discipline: the egress thread never touches JDBC (lock-free queue -> dedicated writer thread, batched inserts, exponential backoff, batch retention during outages, drop-and-count on a full queue). Netty event loops only receive completed futures; reads run on a 2-thread pool with a 1.5s time-box and a 2s JSON cache (absorbs WS connect/resync storms). If Postgres is down or unconfigured the gateway behaves exactly as before this PR.

Consumer contract

Unchanged. Same WS message shapes, same /api/candles response, time still epoch seconds, doubles throughout (display-grade; money authority is the OMS plane). trading-ui needs no changes. One improvement: candle tradeCount now sums the aggregated trade_count instead of counting batch entries, so in-memory and DB-derived candles are identical.

Testing

• 257 unit tests green; new: bucket-alignment proof (time_bucket origin ≡ epoch flooring for all six widths), TradeWriter overflow/retention/poison-batch, read fallback + cache, ticker baseline merge, ring seeding
• TimescaleIntegrationTest (gated on MARKET_PG_TEST_URL, CI runs a timescale/timescaledb:latest-pg16 service): schema idempotency, identical trade stream -> identical candles from DB caggs vs in-memory rings across all six intervals, 24h baseline, policy registration
• New /metrics: market_pg_up, market_pg_trades_written_total, market_pg_trades_dropped_total, market_pg_write_errors_total, market_pg_batch_flushes_total, market_pg_queue_depth, market_pg_read_fallbacks_total

Rollout

Deploys dark: without MARKET_PG_PASSWORD in the environment the persistence layer is off. Provisioning runbook: deploy/timescale/provision-timescaledb.sh (installs the extension into the system PG16, creates marketdata + marketdata_test, writes the admin-service drop-in). Rollback = remove the drop-in and restart.

Known accepted limitations: trades during gateway downtime are absent (no cluster replay backfill this phase); leader-seam redelivery can double-insert (no trade IDs in the aggregated stream — same as today's in-memory double-count).

🤖 Generated with Claude Code