Skip to content

global-monitor: add clickhouse writer#3449

Merged
snormore merged 5 commits intomainfrom
snor/global-monitor-clickhouse
Apr 6, 2026
Merged

global-monitor: add clickhouse writer#3449
snormore merged 5 commits intomainfrom
snor/global-monitor-clickhouse

Conversation

@snormore
Copy link
Copy Markdown
Contributor

@snormore snormore commented Apr 4, 2026

Summary of Changes

  • Add ClickHouse as a second telemetry backend alongside the existing InfluxDB writer, with both independently toggleable via environment variables
  • Include goose-based migration setup with an initial migration creating 3 tables: solana_validator_icmp_probe, solana_validator_tpuquic_probe, doublezero_user_icmp_probe
  • Introduce a ProbeWriter interface so planners can be tested without a real ClickHouse connection

Diff Breakdown

Category Files Lines (+/-) Net
Core logic 7 +831 / -19 +812
Scaffolding 3 +274 / -0 +274
Tests 5 +510 / -12 +498
Config/build 2 +92 / -72 +20

~⅓ tests, ~½ core logic, remainder is migration scaffolding and dependency updates.

Key files (click to expand)
  • telemetry/global-monitor/internal/clickhouse/writer.go — new buffered ClickHouse writer with typed row structs per table, batch insert flush, and ProbeWriter interface
  • telemetry/global-monitor/db/clickhouse/migrations/20260404000001_init.sql — initial migration creating 3 MergeTree tables with monthly partitioning and 90-day TTL
  • telemetry/global-monitor/internal/gm/planner_dz_icmp.go — add recordClickHouseRow() for DZ user ICMP probes including Solana cross-reference fields
  • telemetry/global-monitor/internal/gm/planner_sol_tpuquic.go — add recordClickHouseRow() for Solana validator TPUQUIC probes
  • telemetry/global-monitor/internal/gm/planner_sol_icmp.go — add recordClickHouseRow() for Solana validator ICMP probes
  • telemetry/global-monitor/internal/clickhouse/migrations.go — goose migration runner adapted for slog
  • telemetry/global-monitor/cmd/global-monitor/main.go — wire ClickHouse config via env vars, run migrations on startup, pass writer to runner
  • telemetry/global-monitor/internal/gm/runner.go — add ClickHouseWriter to config, flush buffered rows at end of each tick

Testing Verification

  • Unit tests added for all 3 planners verifying ClickHouse row recording: success case (all fields populated), failure case (probe_ok=false with fail reason), not-ready case (no row written), and nil writer safety
  • Writer buffer tests: correct accumulation per table, concurrent append safety with 300 goroutines
  • All existing InfluxDB recording tests continue to pass unchanged

@snormore snormore changed the title global-monitor: add clickhouse writer with goose migrations global-monitor: add clickhouse writer Apr 4, 2026
@snormore snormore force-pushed the snor/global-monitor-clickhouse branch from d5f3512 to f424451 Compare April 4, 2026 17:16
@snormore snormore requested review from nikw9944 and packethog April 4, 2026 21:15
@snormore snormore marked this pull request as ready for review April 4, 2026 21:31
@snormore snormore enabled auto-merge (squash) April 5, 2026 18:28
snormore added 5 commits April 6, 2026 14:52
Add ClickHouse as a telemetry backend alongside the existing InfluxDB
writer. Either backend can be independently enabled/disabled via
environment variables.
- Use CLICKHOUSE_USER/CLICKHOUSE_PASS env vars to match codebase convention
- Remove unnecessary doc comments
- Requeue failed rows on flush error to prevent data loss
- Use goose.NewProvider instead of global state functions
@snormore snormore force-pushed the snor/global-monitor-clickhouse branch from a4ea9ed to 4a74ef1 Compare April 6, 2026 18:52
@snormore snormore merged commit 836b2f3 into main Apr 6, 2026
33 checks passed
@snormore snormore deleted the snor/global-monitor-clickhouse branch April 6, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants