Test Identity Engine

Test Identity Engine

The hardest and most differentiating piece of Flakemetry. It answers a deceptively simple question: "is this the same test I saw last week?" — correctly, across file moves, renames, and parameterization.

Why it matters

A naive file::title key breaks the moment a test is refactored:

auth.spec.ts › "login should work"
   ── file moved ──►  auth/login.spec.ts › "login should work"
   ── renamed   ──►  auth/login.spec.ts › "logs in with valid creds"

With a naive key, each change creates a new identity → all history is lost → the flaky model resets to zero. A test that has been flaky for months suddenly looks brand new. Identity stability is a precondition for every downstream analytic.

Multi-level fingerprint (confidence-descending)

The resolver tries increasingly permissive matches and records how confident it is:

L1 (exact)    hash(normalized_path + suite + title + params_hash)
L2 (moved)    hash(suite + title + params_hash)          -- file path changed
L3 (renamed)  fuzzy(title) within same file + neighbor context
L4 (param)    base_title identity + separate params_hash bucket

• L1 — the common case, cheapest, highest confidence.
• L2 — same test, file relocated. Match on suite+title+params.
• L3 — same location, title edited. Uses fuzzy similarity plus the surrounding tests ("neighbors") as context to avoid false merges.
• L4 — data-driven / parameterized tests. A stable base identity aggregates the family; each parameter set gets its own bucket so [user=admin] and [user=guest] don't collide but still roll up.

Resolution algorithm

resolve(candidate):
  if exact = find(L1): return exact
  if moved = find(L2): stitch(moved, candidate); return moved   # alias += old fp
  if renamed = find(L3) with confidence >= τ: stitch; return renamed
  if param  = find(L4): attach to base bucket; return base
  return createNew(candidate)

stitch() appends the previous fingerprint to aliases[] and re-points history, so the flaky score and execution history carry over seamlessly.

Path & parameter normalization

• Paths are normalized to workspace-relative, POSIX separators, lowercased where the FS is case-insensitive — so the same test isn't two identities on macOS vs Linux CI.
• Parameters are hashed into a canonical params_hash (sorted keys, stable serialization) so ordering/whitespace differences don't fragment identity.

Guardrails against wrong merges

• Confidence threshold τ on fuzzy matches; below it, prefer a new identity over a bad merge.
• Manual merge/split override in the dashboard for the rare ambiguous case.
• Every match records its level (L1–L4) and confidence, so identity decisions are auditable.

Delivery

• M1 ships L1 + L2 (exact + moved) — enough to survive the most common refactor (file relocation), with pure, property-tested logic in @flakemetry/core.
• M3 adds L3 + L4 (rename + parameterized) plus a backfill migration that re-stitches historical aliases.

Testing approach

Pure functions, no I/O → property-based tests:

• Identity is stable under path change (move a file → same identity).
• Identity is distinct under real content change (different assertions → different identity).
• Fixture suites simulating real refactor sequences (move, then rename, then parameterize).

This is the piece reviewers notice first — it signals the design accounts for how a real test suite evolves, not just the happy path.

Related: Flaky Scoring consumes identity to attribute history; Data Model stores aliases[].