ci(ios-metal): enable Metal API validation in screenshot suite#5110
Merged
Conversation
The #5103 stencil pixel-format mismatch shipped in 7.0.243-246 because no CI path exercised Apple's Metal validation layer. The mistake was a five-line oversight (a render pass missing a Stencil8 attachment while binding a pipeline that declared one), exactly the class of bug Metal validation flags instantly at setRenderPipelineState:. Catch this class going forward by: 1. Forwarding MTL_DEBUG_LAYER + MTL_DEBUG_LAYER_ERROR_MODE through run-ios-ui-tests.sh -> simctl launch when set in the caller's environment. The plumbing is gated by env-var presence so non-Metal local runs are unaffected. 2. Setting both vars in the build-ios-metal job (assert mode) so any validation error crashes the app and the suite's missed CN1SS:SUITE:FINISHED marker fails the step. The GL job is left untouched because validation only fires under the Metal layer. Static audit before flipping: scanned every MTLRenderPassDescriptor and setRenderPipelineState: call site in CN1Metalcompat.m, CN1MetalPipelineCache.m, and METALView.m. The post-#5103-fix code is consistent (all sites that bind a pipeline attach a Stencil8 texture; the framebufferOnly assertion noted at METALView.m:145 was already addressed by metalLayer.framebufferOnly=NO on line 149), so the first CI run on this branch should pass cleanly. If it doesn't, validation has surfaced a sibling latent bug, which is exactly the point. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Compared 122 screenshots: 122 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
simctl launch on Xcode 26 does not accept a --setenv flag (the help output for `xcrun simctl help launch` documents none); the run on the previous commit failed five retries with "Invalid device: --setenv" because simctl interpreted --setenv as a positional device argument. Apple's documented mechanism is to export SIMCTL_CHILD_<NAME>=<value> in the shell that invokes simctl, which the launch helper unwraps into <NAME>=<value> for the child process. Swap the LAUNCH_ENV_ARGS array plumbing for two `export` statements guarded by the same MTL_DEBUG_LAYER / MTL_DEBUG_LAYER_ERROR_MODE checks. The simctl launch line goes back to its original form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Compared 109 screenshots: 109 matched. Benchmark Results
Build and Run Timing
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wire Apple's Metal API Validation into the existing
build-ios-metalscreenshot job so the class of bug that produced #5103 (a render pass binding a pipeline whosestencilAttachmentPixelFormatdoes not match the pass's stencil attachment) trips CI the moment it is committed, rather than four releases later in a user-built iOS app.The plumbing is intentionally narrow:
scripts/run-ios-ui-tests.shreadsMTL_DEBUG_LAYERandMTL_DEBUG_LAYER_ERROR_MODEfrom the caller's environment and forwards them to the launched simulator app viasimctl --setenv. When neither is set the launch line is byte-identical to before..github/workflows/scripts-ios.ymlsets both vars (assertmode) on the Metal job's test step. The GL job is untouched because Metal validation only fires under the Metal layer.The shape on the Metal job: any validation error crashes the app, the suite stops emitting
CN1SS:SUITE:FINISHED, the test step times out, the job goes red. Failure surface is loud; thesimctl-launch.logand the existinglog show --last 30mfallback both capture the assertion message for diagnosis.Why not just rely on the merged #5103 fix
Same bug, different vector. The #5103 fix patched one seed-pass site. This change is the structural guard that would have caught it (and will catch the next one) regardless of which file the mistake lives in.
Static audit done before flipping
Before enabling I scanned every
MTLRenderPassDescriptorcreation and everysetRenderPipelineState:call inCN1Metalcompat.m,CN1MetalPipelineCache.m, andMETALView.m:CN1Metalcompat.m:1407clearPassCN1Metalcompat.m:1430seedPass (post-#5103)CN1Metalcompat.m:1522BeginMutableImageDrawMETALView.m:323clearPassMETALView.m:354screen passThe pre-#5103 framebufferOnly validation assertion noted at
METALView.m:145was already fixed bymetalLayer.framebufferOnly = NOat line 149. So the first CI run on this branch should pass cleanly. If it doesn't, validation has surfaced a sibling latent bug, which is exactly the point — we'd address it in a follow-up or revert this one env-var block until that's done.Test plan
MTL_DEBUG_LAYER_ERROR_MODE=asserttriggering. If it does trigger, dig intosimctl-launch.log+ the fallbacklog showcapture in the uploadedios-ui-tests-metalartifact.bash -n scripts/run-ios-ui-tests.shclean; bash-3.2 array-expansion idiom verified empty + populated underset -u.🤖 Generated with Claude Code