Architecture

Architecture

Flakemetry is a layered, event-driven platform. The guiding constraint: ingestion must never block CI. The write-path returns 202 Accepted instantly and all heavy work happens asynchronously in workers.

Layered overview

┌──────────────────────────────────────────────────────────────────────┐
│  INGESTION EDGE                                                        │
│  ┌───────────────────┐  ┌──────────────────┐  ┌────────────────────┐  │
│  │ @flakemetry/       │  │ OTLP endpoint    │  │ GitHub Action      │  │
│  │ playwright-reporter│  │ (OpenTelemetry)  │  │ (upload + PR bot)  │  │
│  └─────────┬─────────┘  └────────┬─────────┘  └─────────┬──────────┘  │
└────────────┼─────────────────────┼──────────────────────┼─────────────┘
             │  OTLP/HTTP (protobuf/json), zstd, idempotency-key         
             ▼                     ▼                        ▼            
┌──────────────────────────────────────────────────────────────────────┐
│  INGESTION API  (apps/api — Fastify)                                  │
│  • project-token auth  • schema validation (zod)  • rate-limit        │
│  • enqueue → 202 fast ACK (never block CI)                            │
└────────────────────────────────┬─────────────────────────────────────┘
                                  ▼  (durable queue: Postgres SKIP LOCKED)
┌──────────────────────────────────────────────────────────────────────┐
│  PROCESSING WORKERS  (apps/worker)                                    │
│  1. Normalize spans → runs + test executions                         │
│  2. Test Identity Resolution (fingerprint + fuzzy match)             │
│  3. Flaky Scoring (incremental Bayesian update)                      │
│  4. Failure Signature clustering (stack normalize + embeddings)     │
│  5. Trigger AI RCA (async, budget-gated)                            │
└────────────────────────────────┬─────────────────────────────────────┘
                                  ▼                                       
┌──────────────────────────────────────────────────────────────────────┐
│  STORAGE                                                              │
│  PostgreSQL (relational + JSONB + pgvector)                          │
│  Object store (S3/MinIO)  ── traces, videos, screenshots, HAR       │
│  [scale-out] ClickHouse/Timescale ── span-level, high-cardinality   │
└────────────────────────────────┬─────────────────────────────────────┘
                                  ▼                                       
┌──────────────────────────────────────────────────────────────────────┐
│  QUERY / ANALYTICS API  (tRPC + REST for CI)                         │
└────────────────────────────────┬─────────────────────────────────────┘
                                  ▼                                       
┌──────────────────────────────────────────────────────────────────────┐
│  PRESENTATION  (apps/web — Next.js)                                   │
│  Runs · Test detail (history sparkline) · Flaky board · RCA · Trends │
└──────────────────────────────────────────────────────────────────────┘

Data flow (happy path)

1. Emit — reporter builds OTel spans per the OTel Test Conventions and exports via OTLP/HTTP with an idempotency key per run.
2. Accept — apps/api authenticates the project token, validates against @flakemetry/contracts, enqueues the batch, and returns 202 with a receipt id. CI moves on.
3. Process — apps/worker dequeues (FOR UPDATE SKIP LOCKED), normalizes spans into run + test_execution rows, resolves Test Identity Engine, updates the Flaky Scoring, and clusters the failure signature.
4. Explain — new failure signatures trigger AI RCA Architecture asynchronously, budget-gated; known signatures reuse a cached RCA with zero LLM cost.
5. Serve — the query API reads from rollups/materialized views; the Next.js dashboard renders runs, test history, the flaky board, and RCA.
6. Notify — the GitHub Action posts a single sticky PR comment with new/known flakes and RCA links.

Monorepo layout

flakemetry/
├─ apps/
│  ├─ web/            Next.js dashboard
│  ├─ api/            Fastify ingestion + tRPC query
│  └─ worker/         processing (identity, scoring, clustering, RCA)
├─ packages/
│  ├─ contracts/      zod schemas + shared types (single source of truth)
│  ├─ db/             Prisma schema + migrations + client
│  ├─ core/           domain logic: flaky algo, test identity (pure, tested)
│  ├─ reporter/       @flakemetry/playwright-reporter (npm published)
│  ├─ sdk/            OTel instrumentation + ingest client
│  ├─ ai/             LLMProvider abstraction + RCA
│  └─ cli/            @flakemetry/cli
├─ .github/actions/flakemetry/   composite action
├─ docker-compose.yml · turbo.json · pnpm-workspace.yaml · .changeset/
└─ docs/adr/          architecture decision records

Why monorepo: the ingestion contract is shared across reporter → api → worker → web. A schema change must be atomic across all consumers. packages/contracts (zod) is the single source of truth; packages/core holds pure, fully unit-tested domain logic (identity + scoring) with no I/O. See ADR-0001.

Key architectural principles

• Never block CI. 202 + queue decouples ingestion latency from processing. This is bottleneck #1, solved by design from day one. See Ingestion and Scaling.
• Multi-tenant from day one. org_id/project_id on every table, even in single-tenant self-host. Retrofitting tenancy = rewrite.
• Storage seams. The span store sits behind a repository interface so a columnar backend (ClickHouse/Timescale) can slot in at scale without touching core.
• Provider-agnostic AI. LLMProvider abstracts hosted (Claude) vs local (Ollama) so self-host stays fully functional and private.
• Dogfood OTel. The platform instruments itself with OpenTelemetry.