test: tighten post-fix macOS performance budgets (Fixes #507)

[Karthik Nadig (karthiknadig)]

Summary

Recalibrate macOS P95 regression budgets now that #506 removed the deterministic 30-second interpreter-probe tail. The new limits preserve 4-6x observed hosted-runner variance while blocking multi-second drift that the old baseline-era limits allowed.

Changes

• tighten startup P95 from 10,000ms to 750ms absolute headroom
• tighten full-refresh P95 from 5,000ms/25% to 1,000ms/50%
• tighten time-to-first P95 from 10,000ms to 750ms absolute headroom
• prove the worst observed post-fix run remains within budget
• prove 1.5s/2.5s/1.5s synthetic regressions now fail
• document calibration from three unchanged-content runs and the exact merged baseline

Validation

• python -m unittest discover -s scripts/tests -p 'test_*.py' -v (22 passed)
• worst observed macOS artifact passes against merged baseline f0c62d9

Fixes #507

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	1ms	+0ms	+0.0%	>5ms and >100%	➖
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Full refresh P50	58ms	55ms	+3ms	+5.5%	>25ms and >30%	🔺
Full refresh P95	344ms	311ms	+33ms	+10.6%	>1000ms and >100%	🔺
Time to first environment P50	12ms	14ms	-2ms	-14.3%	>20ms and >100%	✅
Time to first environment P95	18ms	17ms	+1ms	+5.9%	>250ms and >100%	🔺

Workload	PR	Baseline
Environments	5	5
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	80.815%	80.815%	+0.000pp
Functions	84.806%	84.806%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	85ms	57ms	+28ms	+49.1%	>100ms and >50%	🔺
Server startup P95	1036ms	621ms	+415ms	+66.8%	>750ms and >100%	🔺
Full refresh P50	172ms	91ms	+81ms	+89.0%	>100ms and >50%	🔺
Full refresh P95	2099ms	1343ms	+756ms	+56.3%	>1000ms and >50%	🔺
Time to first environment P50	115ms	81ms	+34ms	+42.0%	>150ms and >50%	🔺
Time to first environment P95	1094ms	649ms	+445ms	+68.6%	>750ms and >100%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	10ms	9ms	+1ms	+11.1%	>10ms and >50%	🔺
Server startup P95	14ms	12ms	+2ms	+16.7%	>50ms and >100%	🔺
Full refresh P50	177ms	153ms	+24ms	+15.7%	>50ms and >30%	🔺
Full refresh P95	5279ms	1308ms	+3971ms	+303.6%	>5000ms and >100%	🔺
Time to first environment P50	32ms	23ms	+9ms	+39.1%	>25ms and >50%	🔺
Time to first environment P95	328ms	36ms	+292ms	+811.1%	>500ms and >100%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	2	2

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	78.207%	78.207%	+0.000pp
Functions	81.533%	81.533%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[Copilot]

Pull request overview

This PR recalibrates macOS performance regression budgets used by PET’s quality snapshot comparator now that the deterministic 30s cold-refresh tail was removed in #506, ensuring future multi-second drift is caught while staying tolerant of hosted-runner variance.

Changes:

• Tighten macOS P95 regression budgets for server startup, full refresh, and time-to-first-env in the snapshot comparator.
• Add unit tests proving an observed post-fix run stays within the new budgets and that synthetic multi-second regressions now fail.
• Update quality snapshot documentation to reflect the new macOS budgets and recalibration rationale.

Show a summary per file

File	Description
scripts/quality_snapshot.py	Tightens macOS P95 regression budgets in the comparator’s platform budget table.
scripts/tests/test_quality_snapshot.py	Adds tests validating new macOS budgets accept observed variance and reject multi-second regressions.
docs/QUALITY_SNAPSHOTS.md	Updates documented macOS budget values and adds recalibration notes.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

docs/QUALITY_SNAPSHOTS.md:55

• “fixed by issue #504” is incorrect phrasing (issues don’t fix code). This should reference the PR/change that fixed the tail (e.g., #506), while optionally keeping issue #504 as the tracking issue.

The macOS cold-refresh tail fixed by issue #504 remains guarded by phase and locator distributions plus privacy-safe interpreter timeout counts.

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

docs/QUALITY_SNAPSHOTS.md:27

• The documentation still states that the budgets "reflect ... 11 consecutive main-branch baselines", but the macOS P95 budgets in the table are now calibrated from a different dataset (three unchanged-content PR runs + an exact merged baseline). Consider updating the wording to explicitly note this exception so readers don’t assume the macOS P95 numbers came from the 11-baseline set.

Each cell is `absolute / relative`. The budgets reflect observed GitHub-hosted runner variance from 11 consecutive main-branch baselines. Tighten them when a noisy path is fixed rather than normalizing a known regression into the baseline.

The macOS P95 budgets were recalibrated after PR #506 (tracking issue #504) using three unchanged-content pull-request runs and the exact merged baseline at `f0c62d9`. Their absolute headroom is four to six times the observed post-fix run-to-run range.

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

docs/QUALITY_SNAPSHOTS.md:26

• This paragraph cites issue #504 as the tracker for the macOS P95 budget recalibration, but this PR/issue context indicates the tightening work is tracked by #507 (while #504 tracks the original macOS tail). Updating the reference avoids confusing readers about which issue governs the budgets.

The macOS P95 budgets were recalibrated after PR #506 (tracking issue #504) using three unchanged-content pull-request runs and the exact merged baseline at `f0c62d9`. Their absolute headroom is four to six times the observed post-fix run-to-run range.

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite