Glossary

Glossary

Test identity — the stable fingerprint of a test that survives file moves, renames, and parameterization, so history isn't lost on refactor. See Test Identity Engine.

Fingerprint — a hash derived from a test's path/suite/title/params used to compute identity. Multiple levels (L1–L4) trade exactness for resilience.

Flaky test — a test that produces different verdicts without a code change. The strongest evidence is same commit, different result. See Flaky Scoring.

Flaky score — a [0,1] Beta-Binomial, time-decayed measure of how untrustworthy a test is, always accompanied by reason codes.

Reason codes — human-readable explanations attached to a flaky score (e.g. "passed on rerun 4/5 times").

Quarantine — marking a flaky test so CI runs it non-blocking, with an audit trail; auto-reverts when the test stabilizes.

same_sha_variance — different results on the identical commit_sha; the highest-confidence flakiness signal.

fail_isolation — whether a test fails alone (likely a real per-test flake) or alongside many others (likely infra/environment).

Error signature — a normalized fingerprint of a failure (stack/error with line numbers, hex, timestamps stripped) used to dedup and cluster failures.

Cluster — a group of error signatures that represent "the same bug"; enables known-issue detection without an LLM.

RCA (Root-Cause Analysis) — an AI-generated structured explanation of a failure: likely cause, confidence, suggested action, similar past failures. See AI RCA Architecture.

Known-issue fast path — labelling a failure instantly by matching an existing cluster, spending zero LLM tokens.

Run — one CI job / workflow invocation of a suite; the root span of a trace.

Test execution — one execution of one test within a run; retries are separate executions linked by retry_of.

OTLP — OpenTelemetry Protocol, the wire format reporters use to send test telemetry. See OTel Test Conventions.

Span — a timed unit of work in a trace. Flakemetry models test.run, test.case, test.step, and (M2) http / browser.action.

Rollup — a pre-aggregated table/materialized view (e.g. daily_test_stats) so dashboards don't scan raw executions.

Backpressure — signalling producers to slow down (or deferring non-critical work like RCA) when the processing queue is deep.

Open-core — OSS core + a proprietary commercial layer; only operational cost / enterprise features are paid. See Monetization and OSS Model.

LLMProvider — the abstraction that lets RCA run on hosted Claude or local Ollama by config alone.