AI RCA Architecture

AI RCA Architecture

Automated root-cause analysis turns a raw failure into "here's the likely cause and what to do". The design principle: cheap classification first, LLM only when it adds value, always off the write-path, always budget-gated.

The anti-pattern we avoid

Running an LLM on every failing log.

That is slow, expensive, non-deterministic, and mostly redundant — the same failure recurs thousands of times. Flakemetry spends LLM tokens only on genuinely new failure signatures.

Stage 1 — Signature clustering (no LLM)

failure
  → normalize stack/error
        strip line numbers, hex addresses, timestamps, absolute paths, UUIDs
  → normalized_hash + embedding (pgvector)
  → nearest-neighbor match
        if within threshold → attach to existing cluster  (occurrence_count++)
        else                → new cluster

Normalization collapses superficially-different failures ("the same bug") into one cluster. This is fast, deterministic, and free.

Stage 2 — Known-issue fast path (no LLM)

If the failure maps to an existing cluster:

• Label instantly: "known issue #N" / "known flake".
• Reuse the cluster's cached RCA — zero LLM cost.
• The PR bot de-noises the build: "this failure is a known flake, tracked in #N."

Two identical failures cost one LLM call, not two.

Stage 3 — LLM RCA (new signatures only, budget-gated)

Only new clusters reach the model. The prompt is assembled from high-signal context:

context = normalized error + stack template
        + failing git diff summary (files changed in this run)
        + trace summary (from OTel spans: which step/network call failed)
        + top-K similar past failures (retrieval over clustered history)   [M3]

Output is structured (validated against a zod schema before persistence):

{
  "likely_cause": "…",
  "confidence": 0.0,
  "suggested_action": "…",
  "similar_past": [{ "signature_id": "…", "resolution": "…" }]
}

The result is cached on cluster_id, so the whole cluster benefits from one analysis.

Budget & cost control

• Per-project daily token budget; when exceeded, RCA is queued/skipped, never blocking.
• token_cost recorded on every rca_report for observability and (later) usage metering.
• RCA runs entirely asynchronously — it never delays the ingestion 202 or run processing.

Provider abstraction

LLMProvider decouples the pipeline from any vendor:

Provider	Use case
Claude (default)	Hosted, highest-quality structured RCA
Ollama (local)	Self-host / privacy mode — no data leaves the network

Swapping is config-only. Self-hosters with strict data policies run fully local.

Privacy: PII / secret scrubbing

Before any text is stored or sent to an LLM, a scrubbing pass removes secrets and PII (tokens, emails, connection strings, keys) from error messages and stack traces. Self-host + Ollama gives a fully air-gapped path for regulated environments.

Evaluation loop (M3)

• Dashboard thumbs-up/down on each RCA feeds a labelled eval set.
• Prompt templates are versioned; changes are measured against the eval set before rollout.
• Retrieval quality (are the "similar past" failures actually similar?) is tracked.

Delivery

• M1 — basic RCA: normalization + fast path + single-shot structured LLM analysis, budget-gated.
• M3 — clustering at scale, retrieval-augmented RCA with historical context and calibrated confidence, known-issue linking, eval harness.

Related: Data Model (error_signature, rca_report) · Architecture (async processing) · Ingestion and Scaling.