Ingestion and Scaling

Ingestion and Scaling

The central constraint: ingestion must never block CI, and the platform must scale from a single team to thousands of runs/hour without a rewrite.

Bottleneck #1 — write-path throughput

CI jobs finish in bursts (a monorepo push fans out dozens of parallel jobs). If ingestion did synchronous work — resolve identity, score, cluster, RCA — the reporter would block the CI job.

Solution — async decomposition:

reporter ──OTLP──► api: validate + enqueue ──► 202 ACK (fast)
                                    │
                                    ▼
                         durable queue (Postgres SKIP LOCKED)
                                    │
                                    ▼
                      worker pool: normalize → identity → score → cluster → RCA

The api does the minimum: authenticate, validate, enqueue, respond 202. p99 ingestion latency is independent of how backed-up processing is. The reporter also fails open — if the endpoint is down, it buffers to disk and never fails the test process.

Why Postgres-backed queue (for now)

MVP uses SELECT … FOR UPDATE SKIP LOCKED as the queue:

• Single dependency — no Redis/Kafka to operate. Aligns with the one-command-deploy promise.
• Transactional with the rest of the write, so enqueue is atomic.
• Good to meaningful scale (thousands of jobs/hour).

The queue sits behind an interface, so Redis Streams / a real broker slots in at M5 without touching producers/consumers.

Delivery guarantees

• At-least-once delivery + idempotency key per run → safe re-delivery, no double-counting.
• Retry with backoff; dead-letter after N attempts.
• Kill a worker mid-load → jobs resume on restart, zero data loss.
• Backpressure: when queue depth is high, signal producers / shed non-critical work (e.g. defer RCA).

Bottleneck #2 — high-cardinality span storage

Full trace ingestion (M2) explodes row counts: steps + network + browser spans per test. Postgres JSONB is fine early, but span-level analytics at volume want a columnar store.

Solution — storage seam: the span store lives behind a repository interface. At M5, ClickHouse / TimescaleDB slots in for span-level, high-cardinality data while the relational core (identities, scores, runs) stays in Postgres. Core code is unchanged behind the seam.

Bottleneck #3 — dashboard query cost

Trend charts over millions of executions can't scan raw rows.

Solution — rollups: materialized views / rollup tables (daily_test_stats, flaky_trends, suite_daily) refreshed incrementally. Trend endpoints query O(rollup), not O(executions). Raw executions get short retention; rollups get long retention.

Bottleneck #4 — LLM cost

Naive per-failure RCA is unbounded spend.

Solution: signature clustering + known-issue fast path + per-project token budgets. New signatures only, cached per cluster. See AI RCA Architecture.

Multi-tenancy at scale

• org_id / project_id on every table + optional Postgres RLS; cross-tenant leakage covered by negative tests (M5).
• Per-tenant quotas and noisy-neighbor protection on ingestion.
• Tenant-scoped object-store prefixes.

Horizontal scaling model

Component	Scales by
api (ingestion)	stateless → replicas behind a load balancer
worker	add consumers; queue distributes via SKIP LOCKED
Postgres	read replicas for query API; partition executions by time
Span store	ClickHouse sharding (M5)
Object store	S3/managed, effectively unbounded

Reliability targets (M5)

• SLOs on ingestion availability + processing lag.
• Load/soak tests in CI to catch throughput regressions.
• Backup/restore + disaster-recovery runbook.

Related: Architecture · Data Model · Roadmap.