Skip to content

ci(ios-metal): enable Metal API validation in screenshot suite#5110

Merged
shai-almog merged 2 commits into
masterfrom
ci-metal-validation
May 30, 2026
Merged

ci(ios-metal): enable Metal API validation in screenshot suite#5110
shai-almog merged 2 commits into
masterfrom
ci-metal-validation

Conversation

@shai-almog
Copy link
Copy Markdown
Collaborator

Summary

Wire Apple's Metal API Validation into the existing build-ios-metal screenshot job so the class of bug that produced #5103 (a render pass binding a pipeline whose stencilAttachmentPixelFormat does not match the pass's stencil attachment) trips CI the moment it is committed, rather than four releases later in a user-built iOS app.

The plumbing is intentionally narrow:

  • scripts/run-ios-ui-tests.sh reads MTL_DEBUG_LAYER and MTL_DEBUG_LAYER_ERROR_MODE from the caller's environment and forwards them to the launched simulator app via simctl --setenv. When neither is set the launch line is byte-identical to before.
  • .github/workflows/scripts-ios.yml sets both vars (assert mode) on the Metal job's test step. The GL job is untouched because Metal validation only fires under the Metal layer.

The shape on the Metal job: any validation error crashes the app, the suite stops emitting CN1SS:SUITE:FINISHED, the test step times out, the job goes red. Failure surface is loud; the simctl-launch.log and the existing log show --last 30m fallback both capture the assertion message for diagnosis.

Why not just rely on the merged #5103 fix

Same bug, different vector. The #5103 fix patched one seed-pass site. This change is the structural guard that would have caught it (and will catch the next one) regardless of which file the mistake lives in.

Static audit done before flipping

Before enabling I scanned every MTLRenderPassDescriptor creation and every setRenderPipelineState: call in CN1Metalcompat.m, CN1MetalPipelineCache.m, and METALView.m:

Site Binds a pipeline? Stencil attached?
CN1Metalcompat.m:1407 clearPass No (clear + endEncoding) n/a
CN1Metalcompat.m:1430 seedPass (post-#5103) Yes Yes (1449)
CN1Metalcompat.m:1522 BeginMutableImageDraw Yes (via activeEncoder) Yes (1544)
METALView.m:323 clearPass No (clear + endEncoding) n/a
METALView.m:354 screen pass Yes (via activeEncoder) Yes (371)

The pre-#5103 framebufferOnly validation assertion noted at METALView.m:145 was already fixed by metalLayer.framebufferOnly = NO at line 149. So the first CI run on this branch should pass cleanly. If it doesn't, validation has surfaced a sibling latent bug, which is exactly the point — we'd address it in a follow-up or revert this one env-var block until that's done.

Test plan

  • First Metal CI run on this branch completes the screenshot suite without MTL_DEBUG_LAYER_ERROR_MODE=assert triggering. If it does trigger, dig into simctl-launch.log + the fallback log show capture in the uploaded ios-ui-tests-metal artifact.
  • GL job is untouched and stays green.
  • Local sanity: bash -n scripts/run-ios-ui-tests.sh clean; bash-3.2 array-expansion idiom verified empty + populated under set -u.

🤖 Generated with Claude Code

The #5103 stencil pixel-format mismatch shipped in 7.0.243-246 because
no CI path exercised Apple's Metal validation layer. The mistake was a
five-line oversight (a render pass missing a Stencil8 attachment while
binding a pipeline that declared one), exactly the class of bug Metal
validation flags instantly at setRenderPipelineState:. Catch this
class going forward by:

1. Forwarding MTL_DEBUG_LAYER + MTL_DEBUG_LAYER_ERROR_MODE through
   run-ios-ui-tests.sh -> simctl launch when set in the caller's
   environment. The plumbing is gated by env-var presence so non-Metal
   local runs are unaffected.

2. Setting both vars in the build-ios-metal job (assert mode) so any
   validation error crashes the app and the suite's missed
   CN1SS:SUITE:FINISHED marker fails the step. The GL job is left
   untouched because validation only fires under the Metal layer.

Static audit before flipping: scanned every MTLRenderPassDescriptor
and setRenderPipelineState: call site in CN1Metalcompat.m,
CN1MetalPipelineCache.m, and METALView.m. The post-#5103-fix code is
consistent (all sites that bind a pipeline attach a Stencil8 texture;
the framebufferOnly assertion noted at METALView.m:145 was already
addressed by metalLayer.framebufferOnly=NO on line 149), so the first
CI run on this branch should pass cleanly. If it doesn't, validation
has surfaced a sibling latent bug, which is exactly the point.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@shai-almog
Copy link
Copy Markdown
Collaborator Author

shai-almog commented May 30, 2026

Compared 122 screenshots: 122 matched.
✅ Native iOS screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 262 seconds

Build and Run Timing

Metric Duration
Simulator Boot 67000 ms
Simulator Boot (Run) 1000 ms
App Install 12000 ms
App Launch 40000 ms
Test Execution 344000 ms

Detailed Performance Metrics

Metric Duration
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 native encode 944.000 ms
Base64 CN1 encode 1827.000 ms
Base64 encode ratio (CN1/native) 1.935x (93.5% slower)
Base64 native decode 417.000 ms
Base64 CN1 decode 1063.000 ms
Base64 decode ratio (CN1/native) 2.549x (154.9% slower)
Base64 SIMD encode 388.000 ms
Base64 encode ratio (SIMD/native) 0.411x (58.9% faster)
Base64 encode ratio (SIMD/CN1) 0.212x (78.8% faster)
Base64 SIMD decode 387.000 ms
Base64 decode ratio (SIMD/native) 0.928x (7.2% faster)
Base64 decode ratio (SIMD/CN1) 0.364x (63.6% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 75.000 ms
Image createMask (SIMD on) 11.000 ms
Image createMask ratio (SIMD on/off) 0.147x (85.3% faster)
Image applyMask (SIMD off) 168.000 ms
Image applyMask (SIMD on) 105.000 ms
Image applyMask ratio (SIMD on/off) 0.625x (37.5% faster)
Image modifyAlpha (SIMD off) 166.000 ms
Image modifyAlpha (SIMD on) 122.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.735x (26.5% faster)
Image modifyAlpha removeColor (SIMD off) 224.000 ms
Image modifyAlpha removeColor (SIMD on) 91.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.406x (59.4% faster)
Image PNG encode (SIMD off) 1266.000 ms
Image PNG encode (SIMD on) 1003.000 ms
Image PNG encode ratio (SIMD on/off) 0.792x (20.8% faster)
Image JPEG encode 673.000 ms

simctl launch on Xcode 26 does not accept a --setenv flag (the help
output for `xcrun simctl help launch` documents none); the run on the
previous commit failed five retries with "Invalid device: --setenv"
because simctl interpreted --setenv as a positional device argument.
Apple's documented mechanism is to export SIMCTL_CHILD_<NAME>=<value>
in the shell that invokes simctl, which the launch helper unwraps
into <NAME>=<value> for the child process.

Swap the LAUNCH_ENV_ARGS array plumbing for two `export` statements
guarded by the same MTL_DEBUG_LAYER / MTL_DEBUG_LAYER_ERROR_MODE
checks. The simctl launch line goes back to its original form.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@shai-almog
Copy link
Copy Markdown
Collaborator Author

shai-almog commented May 30, 2026

Compared 109 screenshots: 109 matched.
✅ Native iOS Metal screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 377 seconds

Build and Run Timing

Metric Duration
Simulator Boot 79000 ms
Simulator Boot (Run) 1000 ms
App Install 16000 ms
App Launch 13000 ms
Test Execution 1505000 ms

@shai-almog shai-almog merged commit f93ae8c into master May 30, 2026
12 checks passed
@shai-almog shai-almog deleted the ci-metal-validation branch May 30, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant