CI on Bazel: rewrite workflows for the post-#246 tree, start the merge-gate trial - #361
Merged
Conversation
…e-gate trial The old workflows ran swift build / swift test against the slim Package.swift stub, so CI would green-light without testing anything real, and release.yml would package the stub binary. Both now run Bazel: ci.yml is one macos-15 job (Xcode 26.2 to match the .bazelrc pin; the image bundles the iOS 26.2 sim runtime and iPhone 17 devicetype SimulatorTestDevices needs) doing bazel test //... plus the hermetic lint, with the Bazel disk cache under actions/cache. release.yml builds -c opt //previewsmcp/cli:previewsmcp. cache-warm.yml existed to keep SPM caches alive and goes away with them. CI runs as a non-required signal while we collect flake and wall-clock data; the ruleset making it the merge gate comes after a green track record. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… SDK First trial run failed in 3 minutes: rules_swift's test_discoverer is built in the exec configuration, which defaults its minimum OS to the SDK version (26.2), and dyld on the macos-15 runner refuses to load it. Hosts running macOS 26 never see this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Trial run 2: all four sim/integration targets died at Bazel's default 300s cap on the GHA runner (slower cores + per-run first-boot device creation), while passing locally in 20-180s. The suites bound their own tests with .timeLimit traits, so eternal delegates the per-test caps to them; PreviewsJITLinkTests ran 189s on the runner, close enough to 300 to bump to long. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hosted macos-15 measured 50-70 min warm against ~5-7 min for the same suite locally, so the trial pivots to a self-hosted Apple Silicon runner. The persistent host replaces three steps: DEVELOPER_DIR pins Xcode per-job instead of sudo xcode-select, ~/.cache/bazel-disk persists natively so actions/cache goes away, and Spotlight is disabled once at host setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Trial data from the three hosted-runner runs (all on macos-15):
Conclusion: hosted runners are out on both speed (83 min warm vs 5-7 min local) and capability (no hardware H.264). Pivoting this PR to a self-hosted Apple Silicon runner (Mac mini); the 🤖 Generated with Claude Code |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First-ever run on a fresh machine (the mini's first CI job) pays device first-boot plus a cold SwiftPM build of the example inside preview_start, minutes by design; the generic 60s callTool bound failed exactly the two iOS suites that used it while IOSAppServerTests survived only because callToolResult carries no timeout at all. 600 seconds is a bound, not a wait: warm runs stay ~17s, and the suite .timeLimit traits still cap the tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mini's first CI run failed the 2s budget under full build+test load while the agent kept answering runOnMain polls — the main thread was alive, the posted event's run-loop dequeue was just late. The probe passes over SSH on the same host in 1.3s, so dispatch itself works there; the budget was sized for an idle laptop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tbeat The schedule replaces what cache-warm.yml did for the hosted design: with the disk cache living on the runner there is nothing to keep alive, but a weekly run on a quiet repo still catches host/toolchain drift before a PR does, and is a standing flake probe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AGENTS.md still said CI was disabled and described the retired two-job hosted topology (build-and-test/ios-tests socket-dir split, cross-job daemon contention); setup-cache.md still documented the SPM actions/cache keys. Both now describe the single-job self-hosted reality. The parity spec's historical baseline reference stays as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… at 600s The three .disabled(if: CI != nil) traits stopped meaning anything the moment tests moved under Bazel: test envs are hermetic, nothing forwards CI, so the sim suites ran on the mini only by accident of env-scrubbing — and anyone adding --test_env=CI later would silently skip all three and revert the migration with no failing signal. The guards' original referent (GHA macos-15 under combined load wedging CoreSimulator, PR #141) is retired hardware for this suite, and an unprovisioned host still skips via the nil-device guard. callToolResult gains an optional bounded path reusing withTimeout + the stderr dump, and appServerEndToEnd's preview_start uses it at 600s — the last of the three iOS starts that could hang silently to its .timeLimit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…true Code review flagged that bazel test --test_output=errors dropped the old workflow's on-failure daemon/sim dumps. On the persistent runner that state survives the job but not the next run; printing it into the failing run's log keeps the weekly flake probe actionable (#350 specimen collection included). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Direction change from the merge-queue discussion: replace the VM verify+sign+push pipeline with GitHub-native gating. This PR is step 1 — make CI real again and run it as a non-required signal to collect flake and wall-clock data. The ruleset (require CI + restrict merges) comes after a green track record.
Why a rewrite
The old workflows predate the Bazel migration (#246): every build/test step ran
swift build/swift testagainst the slimPackage.swiftstub, so CI would go green while testing nothing real, andrelease.ymlwould package the stub binary.What
macos-15job — select Xcode 26.2 (matches the.bazelrc --xcode_versionpin; the image bundles the iOS 26.2 simulator runtime and iPhone 17 devicetype thatSimulatorTestDevicesprovisions from), disable Spotlight,actions/cacheon the Bazel disk cache keyed byMODULE.bazel.lock,bazel test //...,bazel run //tools/lint:check. 300-minute cap while the trial measures real wall clock (cold runs build LLVM).bazel build -c opt //previewsmcp/cli:previewsmcpand package frombazel-bin(opt config verified analysis-clean locally). Xcode select aligned to the pin.The CI workflow was
disabled_manuallyand is now re-enabled, so this PR runs the newci.ymlon itself — that run is the first trial data point. Repo is public, so macOS minutes are free.Known unknowns the trial answers: cold LLVM build time on the runner, whether the disk cache fits the 10GB Actions cache cap, sim-suite behavior on single-tenant runners (the fixed flake classes — #321/#269/#324/#336/#337/#345/#352 — were mostly local-contention phenomena; #353 and #350 remain open tails).
🤖 Generated with Claude Code