Luminle

Illuminate your database costs

Open source query cost attribution for ClickHouse. Tag your queries with SQL comments and see which teams and services are driving your database spend.

---

Status: v1 — ClickHouse only. PostgreSQL planned for v2.

---

The Problem

Your database bill is growing, but you don't know why.

• Which team is running the most expensive queries?
• Is that new microservice costing you thousands per day?
• Which endpoints are driving 80% of your database spend?

Most monitoring tools show you performance (slow queries, CPU). Cloud providers show you total bills. Neither tells you which teams or services are responsible for which costs.

What Luminle Does

Luminle polls system.query_log in ClickHouse, extracts cost attribution tags from SQL comments, calculates per-query costs (compute time + I/O bytes), and stores them locally. Query the results with the CLI.

$ luminle top teams

Top 4 by TEAM:

RANK  TEAM           QUERIES  TOTAL COST   AVG COST     MAX COST     TOTAL DATA
----  ----           -------  ----------   --------     --------     ----------
1     analytics      145      $0.012450    $0.000086    $0.001234    2.3 GB
2     backend        89       $0.007820    $0.000088    $0.000956    1.8 GB
3     data-science   34       $0.005600    $0.000165    $0.002100    890.5 MB
4     platform       12       $0.001200    $0.000100    $0.000450    234.1 MB

Total: 280 queries, $0.027070

Quick Start

Prerequisites: Go 1.21+, Docker

git clone https://github.com/tjb/luminle
cd luminle
make dev

make dev builds the binary, starts a ClickHouse test container, loads sample data, runs the collector, seeds tagged queries, and prints cost results.

To keep collecting after make dev:

# Terminal 1 — run collector
make start

# Terminal 2 — seed more queries anytime
make seed

# Query results
./luminle top teams
./luminle top services
./luminle top queries
./luminle query --team analytics

Connect to your own ClickHouse:

./luminle start \
  --clickhouse your-host:9000 \
  --clickhouse-user default \
  --clickhouse-password "" \
  --hourly-cost 0.15 \
  --io-cost 0.20

Tagging Your Queries

Add SQL comments to attribute costs:

/* team=analytics service=dashboard */
SELECT * FROM events WHERE date = today();

/* team=backend service=api endpoint=/users */
SELECT * FROM users WHERE id = 123;

Supported formats: tag=value, tag="value", tag='value', tag:value

Luminle extracts all tags automatically and stores them alongside cost data.

How It Works

┌─────────────┐
│  Your App   │
└──────┬──────┘
       │ queries with tags
       │ /* team=analytics */
       ▼
┌─────────────────┐         ┌──────────────┐
│   ClickHouse    │◄────────┤   Luminle    │
│                 │  polls  │              │
│ system.query_log│  every  │  Collector   │
└─────────────────┘  10s    └──────┬───────┘
                                   │ calculates costs
                                   │ extracts tags
                                   ▼
                          ┌─────────────────┐
                          │   SQLite        │
                          │ ~/.luminle/     │
                          └────────┬────────┘
                                   │
                                   ▼
                          ┌─────────────────┐
                          │   CLI Commands  │
                          │  top, query,    │
                          │  report         │
                          └─────────────────┘

Cost model:

cost = (duration_ms / 3_600_000) × hourly_rate
     + (bytes_read  / 1_000_000_000) × io_rate_per_gb

CLI Reference

Start collector

luminle start --clickhouse localhost:9000
luminle start --clickhouse localhost:9000 --hourly-cost 100.0 --io-cost 10.0

Query costs

luminle query --limit 20
luminle query --team analytics --since 24h
luminle query --service user-api --min-cost 0.01

Top consumers

luminle top teams
luminle top services
luminle top queries --limit 10
luminle top users

Reports

luminle report --since 24h
luminle report --since 2024-01-01 --until 2024-01-31

Makefile Targets

make build    compile the luminle binary
make test     run all tests
make up       start ClickHouse container (idempotent)
make down     stop ClickHouse container
make tables   create test schema if not present (idempotent)
make seed     run tagged test queries + flush query_log
make start    run collector in foreground (use with make seed in another terminal)
make dev      full demo: build → tables → start → seed → wait → show results
make reset    tear down everything and start clean

Why Luminle?

Solution	Strengths	Limitations
ClickHouse native	Built-in query_log	No cost model, no team attribution
PMM / Datadog DBM	Deep query performance insights	No cost attribution
CloudZero / Vantage	Comprehensive cloud cost visibility	No query-level database granularity
Custom scripts	Tailored to your needs	Hard to maintain, no cross-database view
Luminle	Query-level costs + team attribution	v1: ClickHouse only

Roadmap

v1 — Current

• ClickHouse system.query_log collector
• Cost model: compute time + I/O bytes
• Tag extraction from SQL comments
• SQLite local storage
• CLI: start, top, query, report, init, version
• High-water mark incremental collection
• Tested: cost model, tag extractor, SQLite repository

v2 — Planned

• PostgreSQL collector (pg_stat_statements)
• lastCollectedAt persistence across restarts
• YAML config file (once there are 2+ databases to configure)
• ClickHouse collector integration tests (Docker in CI)

Troubleshooting

Costs showing $0.0000

Your queries ran too fast or touched too little data. Use inflated test pricing:

./luminle start --clickhouse localhost:9000 --hourly-cost 100.0 --io-cost 10.0

Queries not appearing

ClickHouse buffers query_log for ~7.5 seconds. Either wait, or:

docker exec luminle-clickhouse clickhouse-client \
  --user luminle --password luminle123 \
  --query="SYSTEM FLUSH LOGS"

Start fresh

make reset

License

MIT — see LICENSE.

---

Built with ☕ by developers tired of database bill surprises.