OTel Test Conventions

OTel Test Conventions

The load-bearing spec of the whole platform. "Tests as traces" only works if the span and attribute model is precisely defined. Any reporter (Playwright, Jest, Vitest, pytest) emits to this contract; the api and worker consume it. See ADR-0002.

Span hierarchy

A test run is a trace. Each level is a span:

test.run                         (root span = one CI job / suite invocation)
 ├─ test.case                    (one test; retries share identity, differ by attempt)
 │   ├─ test.step                (a step / hook: beforeEach, action, assertion)
 │   │   ├─ http                 (network call made during the test)      [M2]
 │   │   └─ browser.action       (click/navigation/etc.)                  [M2]
 │   └─ test.case  …
 └─ test.case  …

M1 ingests test.run + test.case. M2 adds the child spans (test.step, http, browser.action) for full trace waterfalls.

Resource attributes (per run)

Set once on the OTel Resource:

Attribute	Example	Purpose
service.name	flakemetry-reporter	standard OTel
flakemetry.project	acme/web	tenant/project routing
ci.provider	github_actions	CI context
ci.run_id	1234567890	link back to CI
vcs.commit_sha	abc123…	enables same-sha flake signal
vcs.branch	main	branch analytics
vcs.pr_number	42	PR bot targeting

Span attributes (per test.case)

Attribute	Example	Purpose
test.identity.fingerprint	sha256:…	stable identity (see Test Identity Engine)
test.suite	auth/login	grouping
test.title	logs in with valid creds	display
test.params_hash	sha256:…	parameterized bucketing
test.status	pass|fail|skip|flaky	verdict
test.attempt	2	retry index
test.retry_of	<span_id>	retry linkage
test.duration_ms	1834	timing

Status & error mapping

• Test verdict → OTel span status (OK / ERROR).
• Failures attach an OTel exception event carrying exception.type, exception.message, exception.stacktrace.
• The worker derives the normalized AI RCA Architecture from the exception event.

Artifacts

Screenshots, videos, traces, and HAR files are referenced as span attributes (object-store keys). The Architecture (M2) uploads them and the query API serves signed URLs.

Transport

• OTLP/HTTP, protobuf or JSON.
• zstd compression for large batches.
• Idempotency key per run so re-delivery (CI retries, at-least-once queue) never double-counts.

Why standardize on OTel

• Reuses a mature ecosystem (SDKs, exporters, collectors) instead of a bespoke wire format.
• Enables correlation with application traces — a failing E2E test can link to the backend spans it triggered.
• Makes Flakemetry a first-class citizen of an existing observability stack, not a walled garden.

Conformance

All reporters (Playwright first; Jest/Vitest/pytest in M4) must pass a shared conformance suite asserting they emit spec-valid OTLP. This keeps every ingestion source interchangeable.

Related: Data Model (how spans map to rows) · Ingestion and Scaling.