test: tighten schema-v2 warm P95 budgets (Fixes #511)

[Karthik Nadig (karthiknadig)]

Summary

Tighten schema-v2 warm refresh and warm time-to-first P95 budgets now that cold and warm samples are independent. The new limits retain at least 2.5x observed hosted-runner variance while blocking multi-second warm-cache drift.

Changes

• tighten Linux warm refresh/time-to-first P95 budgets to 50ms/50% and 25ms/100%
• tighten Windows warm P95 budgets to 250ms/100% and 100ms/100%
• tighten macOS warm P95 budgets to 300ms/100% and 250ms/100%
• preserve all P50, startup, cold-refresh, coverage, and schema behavior
• prove the worst observed P95 values pass on every platform
• prove multi-second warm regressions fail on every platform
• document calibration provenance and historical schema migration behavior

Validation

• python -m unittest discover -s scripts/tests -p 'test_*.py' -v (33 passed)
• exact schema-v2 merged baselines pass under the new limits
• calibration uses three unchanged-code PR runs plus merged baseline ad7ca14

Fixes #511

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	80.815%	80.815%	+0.000pp
Functions	84.806%	84.806%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	1ms	+0ms	+0.0%	>5ms and >100%	➖
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Full refresh P50	49ms	59ms	-10ms	-16.9%	>25ms and >30%	✅
Full refresh P95	53ms	60ms	-7ms	-11.7%	>50ms and >50%	✅
Time to first environment P50	9ms	13ms	-4ms	-30.8%	>20ms and >100%	✅
Time to first environment P95	12ms	16ms	-4ms	-25.0%	>25ms and >100%	✅
Cold refresh P50	122ms	156ms	-34ms	-21.8%	>100ms and >50%	✅

Workload	PR	Baseline
Environments	5	5
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	78ms	79ms	-1ms	-1.3%	>100ms and >50%	✅
Server startup P95	570ms	652ms	-82ms	-12.6%	>750ms and >100%	✅
Full refresh P50	145ms	134ms	+11ms	+8.2%	>100ms and >50%	🔺
Full refresh P95	150ms	160ms	-10ms	-6.2%	>300ms and >100%	✅
Time to first environment P50	84ms	97ms	-13ms	-13.4%	>150ms and >50%	✅
Time to first environment P95	111ms	133ms	-22ms	-16.5%	>250ms and >100%	✅
Cold refresh P50	367ms	332ms	+35ms	+10.5%	>250ms and >50%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	78.207%	78.207%	+0.000pp
Functions	81.533%	81.533%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	8ms	7ms	+1ms	+14.3%	>10ms and >50%	🔺
Server startup P95	11ms	10ms	+1ms	+10.0%	>50ms and >100%	🔺
Full refresh P50	117ms	105ms	+12ms	+11.4%	>50ms and >30%	🔺
Full refresh P95	122ms	109ms	+13ms	+11.9%	>250ms and >100%	🔺
Time to first environment P50	17ms	16ms	+1ms	+6.2%	>25ms and >50%	🔺
Time to first environment P95	23ms	24ms	-1ms	-4.2%	>100ms and >100%	✅
Cold refresh P50	119ms	104ms	+15ms	+14.4%	>150ms and >50%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	2	2

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[Copilot]

Pull request overview

This PR recalibrates the schema-v2 warm performance regression budgets (P95 for full refresh and time-to-first environment) to better reflect post-#510 sampling behavior, preventing multi-second warm-cache drift while still tolerating observed hosted-runner variance.

Changes:

• Tighten per-platform warm P95 regression budgets in the snapshot comparator (Linux/Windows/macOS).
• Add unit tests that (1) validate worst observed warm P95 variance still passes and (2) enforce failure on multi-second warm regressions across all platforms.
• Update docs/QUALITY_SNAPSHOTS.md to document the new budgets and calibration provenance.

Show a summary per file

File	Description
scripts/tests/test_quality_snapshot.py	Adds targeted tests for schema-v2 warm P95 budget calibration and multi-second regression rejection.
scripts/quality_snapshot.py	Updates warm P95 regression budgets in PERFORMANCE_BUDGETS to the tightened thresholds.
docs/QUALITY_SNAPSHOTS.md	Aligns documented budget table and explains recalibration context/provenance for schema-v2 warm P95.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite