Reduce snapshot CI sizing overhead - #219
Conversation
The practical options fall into four tiers. For this repository, I’d trial Depot or GitHub’s M2 runner first; a self-hosted Mac mini is potentially fastest but carries much more operational and security baggage.
1. GitHub-hosted XLarge M2GitHub’s current standard macOS runner is an M1 VM with 3 CPU cores and only 7 GB RAM. The XLarge runner moves to an M2 configuration with 5 CPU cores, 14 GB RAM, and GPU acceleration. The relevant label would be Current pricing is $0.102/minute. Larger-runner minutes are always billed: included Actions minutes do not apply, and public repositories do not make them free. GitHub also currently limits larger-runner management to organizations on Team or Enterprise Cloud, so a personally owned repository may need organizational/account changes. GitHub runner pricing At the recent 26.5-minute snapshot-job median:
Advantages:
Caveats:
My expectation would be a meaningful build improvement and a smaller—but still useful—capture improvement from the newer CPU/GPU. It is the cleanest controlled benchmark, not necessarily the fastest choice. 2. Depot macOS runnersDepot currently offers:
This is unusually attractive for Stuff because it combines newer Apple Silicon with substantially more RAM and storage while costing less per minute than GitHub XLarge. At the current snapshot-job duration:
The workflow change is essentially replacing Caveats:
Of the paid options, this is the one I’d be most interested in benchmarking for Stuff. 3. Cirrus RunnersCirrus sells managed, ephemeral GitHub Actions runners backed by:
Cirrus claims roughly 2× performance over GitHub’s M1 macOS runners, and its published customer examples report 30-minute iOS builds dropping to roughly 12 minutes. Those are vendor/customer claims, so I would treat them as reasons to benchmark rather than forecasts. Cirrus Runners The economics:
One important topology detail: one Cirrus concurrency slot would serialize the iOS unit and snapshot jobs if both used it. Options would be:
This is most attractive if the repository runs several long CI jobs every day. 4. A self-hosted Mac miniAn M4 or M4 Pro Mac mini running GitHub’s self-hosted runner could plausibly produce the largest improvement. The local snapshot suite already runs dramatically faster than GitHub’s M1 VM, and persistent package/build caches could reduce build setup further. Benefits:
But it is not merely “install the runner and forget it”:
Most importantly, GitHub recommends self-hosted runners only for private repositories because an untrusted fork PR can execute arbitrary code on the machine. GitHub self-hosted runner security guidance For this public repository, I would not attach a persistent personal Mac directly to unrestricted
5. MacStadium / OrkaMacStadium offers hosted bare-metal Macs and Orka, its Kubernetes-oriented macOS VM orchestration platform. Orka supports ephemeral Apple Silicon VMs and direct GitHub Actions integration. It starts conceptually around a multi-node fleet and scales to hundreds of nodes. Orka overview This gives you:
It also brings:
I would consider it only if this grows into CI infrastructure shared by several Apple-platform projects or developers. What faster hardware will and won’t fixThe snapshot suite cannot safely be parallelized: captures serialize around UIKit/global simulator state, and prior sharding experiments made it both slower and flaky. More cores therefore mainly accelerate:
The serial settle floors remain real wall time. Newer Apple Silicon can render and compare faster, but it cannot eliminate a deliberate 250 ms or 1-second settle window. That makes the likely ranking:
A sensible benchmark would route only the snapshot job to each candidate for 5–10 runs, retaining the standard runner as a control. Compare queue time, build time, 381-image execution time, pixel stability, and cost per successful run. |
…hot-time # Conflicts: # Shared/SnapshotKitTesting/AGENTS.md
## Summary - split the snapshot suite into two duration-balanced, serial jobs on isolated Xcode runners - expose `Snapshot Tests (iOS, 1/2)` and `Snapshot Tests (iOS, 2/2)` as the checks branch protection requires directly - add checked-in shard membership plus `./snapshot-shards` commands for validation and balancing from local, profile, or successful-CI timing reports - make `./test` and `./profile` emit the same versioned per-suite duration format ## Why PR #219 reduced work inside each capture, but snapshot execution still owns most of the CI critical path. The seeded longest-processing-time assignment measures 146.33s versus 146.24s of suite self-time, allowing separate runners to halve test execution without reintroducing unsafe same-Mac render-server contention. ## Design decisions - shard membership is explicit and checked so each run is reproducible and a new suite cannot silently disappear - rebalancing is dry-run by default; `--write` is required to change the configuration - CI history uses medians from complete successful `main` runs with both timing artifacts - each shard stays serial; concurrent shards remain prohibited on one developer Mac - branch protection requires both shard checks directly; there is no legacy aggregate compatibility job ## Review focus - branch protection must require both exact shard check names before merge - duplicated checkout/build work increases total runner use by an estimated 30% in exchange for roughly eight minutes off the critical path ## Testing - `python3 -m unittest discover -s Tools/Tests -p test_snapshot_shards.py` (10 tests passed) - `./snapshot-shards check` (42 suites exactly once: 20 + 22) - existing `.xcresult` report extraction and `./snapshot-shards balance --report` (146.332s / 146.243s) - `./test --snapshots --timing-report /private/tmp/snapshot-full-report.json` (42 tests passed in 5:19) - `./test --snapshots --snapshot-shard 1/2 --no-generate --no-build --timing-report /private/tmp/snapshot-shard-1-report.json` (20 tests passed in 2:35) - `./test --snapshots --snapshot-shard 2/2 --no-generate --no-build --timing-report /private/tmp/snapshot-shard-2-report.json` (22 tests passed in 2:50) - shard report union: 42 suites, zero overlap; observed self-time 147.426s / 146.248s - GitHub Actions: shard 1 passed in 18:29; shard 2 passed in 18:02 - `./swiftformat --lint` - workflow YAML parse, shell syntax checks, and `git diff --check`
1 participant
Summary
Performance
The safe result is below the 15% stretch target because DataSettings immediate measurement was reverted after a reproducible accessibility snapshot failure.
Validation