Delta Comparison Policy

Delta Comparison Policy

This document defines deterministic semantics for Phase 3 delta analytics.

Run role semantics

• baseline_run_id: reference run used as the comparison anchor.
• current_run_id: run being evaluated against baseline.
• Delta formula:
  • absolute_delta = current_value - baseline_value
  • relative_delta_pct = ((current_value - baseline_value) / baseline_value) * 100 when baseline is non-zero.

Classification labels

• improvement: metric moved in a favorable direction.
• regression: metric moved in an unfavorable direction.
• neutral: no change (absolute_delta == 0).
• unknown: metric cannot be compared due to missing or incompatible data.

Direction policy table

Metric	Group	Better Direction	Notes
total_tests	reliability	higher	More executed tests means broader reliability signal.
passed	reliability	higher	More passing tests is better.
failed	reliability	lower	More failed tests is worse.
broken	reliability	lower	More broken tests is worse.
skipped	reliability	lower	More skipped tests is treated as worse coverage quality.
health_pct	reliability	higher	Higher health percentage is better.
wall_duration_ms	performance	lower	Lower wall-clock duration is better.
metrics_duration_ms	performance	lower	Lower aggregate metrics duration is better.
avg_case_ms	performance	lower	Lower average case duration is better.

Unknown/null reason codes

• missing_current_metric: current run has no value.
• missing_baseline_metric: baseline run has no value.
• zero_baseline_for_relative: absolute delta can be computed, relative percent cannot.
• incompatible_test_kind: run pair has different test_kind.

Stage-level deltas (added 2026-07-23)

GET /api/v1/analytics/delta also returns stage_deltas: one entry per stage name present on either run's stage_health, matched by name (no archive extraction — reuses the stage_health already loaded on each CompletedRunView). Classification uses the same higher_is_better rule as health_pct above, applied to each stage's own health_pct. A stage present on only one side (renamed/added/removed since the baseline ran) gets classification: "unknown" with the missing side's fields null, rather than being dropped — mirrors the metric-level missing_*_metric reasons in spirit, but stage deltas don't carry a reason field.

Per-test deltas (added 2026-07-23)

GET /api/v1/analytics/delta/cases returns the full per-test breakdown — the API/UI equivalent of testo diff's non---metrics-only output. Each test case is matched across the two runs by historyId → fullName → uuid (first present, in that order) and classified as:

• added: present in current, not in baseline.
• removed: present in baseline, not in current.
• regression: passed in baseline → failed/broken in current.
• fix: failed/broken in baseline → passed in current.
• status_change: any other status transition.
• (unchanged cases are computed but dropped from the response — only actual changes are returned.)

Unlike the run/stage-level deltas, this is computed on demand from each run's live artifact snapshot every request, with no caching — see CLI-UI Parity - Pyramid, Graphs, Deep Diff - 2026-07-23 for why (archives aren't large enough yet to justify the cache-invalidation work) and for why this reads from each run's snapshot directory rather than the separate ReportArchive table (testo report list/diff's id space, which has no link to run-history run ids).

---

Generated from docs/Specs & ADRs/Delta Comparison Policy.md. Edit it there; changes made in the wiki are overwritten.