Immersive Presenter

Immersive Presenter

The immersive presenter is the last stage of the pipeline. It takes the pictures the original engine produced and draws them into the Vision Pro's compositor. It is written directly against CompositorServices (CompositorLayer / LayerRenderer) with Metal, and uses ARKit WorldTrackingProvider for head pose. RealityKit is not used. On every display frame the presenter does the following:

1. Leases the newest complete panorama from the runtime: eight camera views plus a HUD layer, ten texture layers in all.
2. Lays the eight views out as panels on one sphere of radius 4.25 m around the viewer, cross-fading each into the panels already drawn.
3. Draws Halo's own interface (HUD, menus) on a curved panel 3.1 m away, with a soft shadow surface behind it.
4. Turns menu look-and-pinch interactions into engine cursor clicks.
5. Feeds head yaw, pitch and roll back to the host.

When no world panorama is available (menus, loading), it draws the flat engine frame, or a loading card, on the same curved panel inside a dim "surround" that continues the picture's edges.

Producing the ten layers is the host's job and is documented in Panorama System. This page covers the Swift consumer side and the runtime pool that hands frames across. The app shell is described in visionOS App.

Source files

File	Role
EngineImmersive.swift	Layer configuration, EngineImmersiveSession (main actor), EngineImmersiveControl (thread-safe shared state), EngineImmersiveRenderer (render loop, encoding, menu input), and the Metal shader source
EngineImmersiveScreenGeometry.swift	Curved menu/HUD panel, HUD shadow, surround sphere, eight-panel panorama sphere, panel fades, menu ray hit test
EnginePanoramaTexture.swift	Panorama snapshot: zero-copy lease or three-slot copy pool, validation, sRGB views, transition hold
EnginePanoramaLayerArray.swift	Walks C fixed-size arrays (imported as tuples) by memory so the layer count can never fall behind the header
EngineImmersiveRenderTargets.swift	Sample count (1) and render pass construction, including the tracking-areas attachment
EngineImmersiveLoadingTexture.swift	CoreGraphics/CoreText loading card (1024x640)
EngineImmersiveOwnership.swift	Generation tokens for open/close races
EngineImmersiveTrace.swift	Bounded (128-entry) lifecycle timeline
EngineWorldCadence.swift	Submission cadence statistics for distinct world frames
EngineMenuInput.swift	Spatial-event to click state machine
EngineFrameTexture.swift	Immutable flat-frame texture snapshot
EngineHaptics.swift	pump() is called once per presenter frame
EngineLayerAlignment.swift	Optional per-layer rotation (see Layer Alignment)
EngineVisionRuntime.m	Runtime side of the zero-copy pool (acquire, carry, publish, lease)

The ten layers, as the presenter sees them

From panorama.h (HALO_PANORAMA_LAYERS = 10):

Layer	Content	Presenter panel (source index)
0	bearing -60°	0
1	bearing 0°, left eye (the centre)	1
2	bearing +60°	2
3	HUD / interface (flat, premultiplied)	drawn on the menu panel, not the sphere
4	bearing 0°, right eye (only when stereo)	panel 1, right eye
5	straight up (zenith)	6 (sky)
6	straight down (nadir)	7 (floor)
7	bearing +120°	3
8	bearing 180°	4
9	bearing -120°	5

EnginePanoramaTexture.Projections.ringLayers = [0, 1, 2, 7, 8, 9] gives the order around the ring, starting at the left of the forward view (EnginePanoramaTexture.swift:17). Only the centre bearing is rendered per eye. Snapshot.view(eye:sector:) returns layer 4 for the right eye of sector 1 when stereo is set, and the same texture for both eyes otherwise (L107-L110).

Compositor layer configuration

EngineImmersiveLayerConfiguration.makeConfiguration (EngineImmersive.swift:10-76):

Setting	Value	Why (from source comments)
colorFormat	.bgra8Unorm_srgb	Engine bytes are display-referred sRGB; sampling them through sRGB views and writing to an sRGB target puts the engine's byte on screen
depthFormat	.depth32Float
isFoveationEnabled	capabilities.supportsFoveation	It was disabled early because of stale peripheral tiles, back when the screen was a 192-triangle cylinder. With eight dense panels sampled for both eyes, the headset measured 43.5 Hz on a 90 Hz display, so it was short on fragment work. The rate map is applied per pass.
layout	.dedicated if supported, else .layered, else .shared
defaultDepthRange	(100, max(0.05, supportedMinimumNearPlaneDistance))	Reversed depth: far, near
trackingAreasFormat	.r8Uint only when allowed (below)

Tracking areas (eye-tracked menu hover)

Tracking areas are the only form of eye tracking available to an app. The app writes an 8-bit identifier per pixel; the system draws its hover glow where the eyes rest and names that area in a pinch event. The decision is recorded in trackingDecision and reported (L44-L74):

1. Off unless the environment has HALO_EYE_MENU=1. On xrOS 26.3 the request passed the validator but the layer never started, which left the game stuck behind its setup window.
2. Off if the previous launch attempted tracking areas and never presented a frame. The UserDefaults keys HaloTrackingAreasAttempted and HaloTrackingAreasSucceeded track this.
3. Off if .r8Uint is not offered.
4. Otherwise the configuration marks the attempt, sets .r8Uint (leaving the usage at its default, because an explicit usage aborted the scene in Build64), and validates with LayerRenderer.Properties(configuration:). If validation throws, the format and usage are reverted.

The first GPU-completed frame with tracking enabled sets HaloTrackingAreasSucceeded = true (L847-L849). The renderer also requires sampleCount == 1 to use tracking areas.

Session, control and renderer objects

Object	Isolation	Responsibility
EngineImmersiveSession (L187-L280)	@MainActor	active, opening, status, closeRequest; owns at most one EngineImmersiveControl; ownership tokens; loading state; diagnostic accessors
EngineImmersiveControl (L99-L185)	NSLock	Shared between the main actor, the render thread and GPU completion handlers: stopped, recenter generation, loading card state, frame counters, GPU time, last submission, world cadence
EngineImmersiveRenderer (L282-L1284)	Its own detached Thread	Pipelines, meshes, ARKit session, menu input, the render loop

Session start

EngineImmersiveSession.start(layer:) (L219-L265):

1. Stops any previous control and cancels pending engine pointer input (enginevision_pointer(-1, -1, 0, -1)).
2. Advances the ownership generation and creates a new EngineImmersiveControl carrying the latest loading state. Sets active = true.
3. In a Task, creates EngineImmersiveRenderer(layer:control:report:), then await renderer.startTracking(), which runs ARKitSession.run([WorldTrackingProvider]) and throws if world tracking is unsupported.
4. If the control was superseded or stopped meanwhile, stops ARKit and returns.
5. Thread.detachNewThread { renderer.run() }. When the loop exits, it records renderLoop.exit. Back on the main actor, if this control is still current, it sets active = false, clears the control and increments closeRequest. The setup window observes closeRequest and dismisses the space.
6. A renderer init failure sets the status to "Immersive renderer failed: ..." and requests close.

finish() advances the generation, stops the control, cancels pointer input and clears state. recenter() increments the control's recenter generation.

Session ownership

EngineImmersiveOwnership (EngineImmersiveOwnership.swift) is a monotonically increasing generation:

• beginOpen(active:opening:) returns a new token only when neither flag is set.
• owns(token) means the token is still the current generation.
• canDismiss(token, active:opening:) requires ownership and neither flag set.

Any asynchronous completion (an openImmersiveSpace result, a renderer exit) may therefore only act on the generation that scheduled it. ImmersiveOwnershipValidation.swift checks the interleavings: a failing startup A cannot clear B; a queued open cannot toggle closed; a layer that arrives before openImmersiveSpace returns invalidates the late open's token.

EngineImmersiveControl.stop() and pointer() are serialised by the same lock. A stopped (superseded) renderer can therefore neither enqueue nor clear the replacement's taps (L119-L133).

Renderer construction

EngineImmersiveRenderer.init (L371-L460) builds:

Resource	Details
queue	One MTLCommandQueue on layer.device
frame, panorama, loadingTexture	EngineFrameTexture, EnginePanoramaTexture, EngineImmersiveLoadingTexture
vertices/indices, hudVertices, hudShadowVertices	Curved panel meshes for an initial aspect of 1024/640
black	1x1 opaque black fallback texture
pipeline	curve_vertex + curve_fragment, no blending, maxVertexAmplificationCount = viewCount
panoramaPipeline	Same functions, source-alpha over blending (panels cross-fade)
alignedPipeline	EngineAlignedPipeline: built lazily in the background from the shader source plus EngineLayerAlignment.shaderSource, using a copy of the panorama descriptor (Layer Alignment)
backdropPipeline	backdrop_fragment, no blending
overlayPipeline	hud_fragment or hud_fragment_tracked, premultiplied over blending (one, oneMinusSourceAlpha)
rowsPipeline, bandsPipeline	Compute kernels hud_rows, hud_bands (tracking areas only)
depth	greater, write on (reversed depth)
panoramaDepth, overlayDepth	always. The panels all lie on one sphere, so their depths are equal to within rounding and a depth test could drop whichever arrived second. Draw order decides instead.

When tracking is enabled, colour attachment 1 is the r8Uint tracking texture with writeMask = [] for every pipeline except the overlay. Only the interface writes identifiers. The configuration string reported to diagnostics is layout=... foveation=... msaa=... views=... tracking=.... layer.onSpatialEvent is attached last, because its closure captures self.

Render targets

EngineImmersiveRenderTargets.sampleCount always returns 1 (L11-L14). Every surface is a texture-mapped panel whose joins are dissolved in the shader, so MSAA had nothing to smooth except silhouettes. Meanwhile it cost four samples of colour and depth per pixel per eye on a headset already presenting at half rate. makePass still supports MSAA: memoryless multisample targets resolving into the drawable, with depthResolveFilter = .max for reversed depth. The GPU test uses that path. Passes clear colour to opaque black and depth to 0. The tracking attachment is cleared to 0 ("no area").

Per-frame loop

run() (L565-L586) loops until the control is stopped. On .invalidated it resets menu input and returns. On .paused it resets menu input and calls waitUntilRunning(). On .running it runs drawFrame() in an autorelease pool. Every layer-state change is recorded in the trace.

sequenceDiagram
    participant L as LayerRenderer
    participant R as Renderer thread (drawFrame)
    participant P as EnginePanoramaTexture
    participant RT as Runtime (enginevision_*)
    participant AR as WorldTrackingProvider
    participant H as Host (halo_settings, pointer)
    participant GPU as Metal

    R->>L: queryNextFrame()
    R->>L: startUpdate / endUpdate
    R->>L: predictTiming(), wait until optimalInputTime
    R->>GPU: makeCommandBuffer
    R->>L: startSubmission(), queryDrawables()
    alt no drawables
        R->>R: didCancelFrame, return (frame must not be touched again)
    end
    R->>P: refresh(leasedTo: command) unless loading card shown
    P->>RT: enginevision_panorama_gpu_latest (lease) or copy path
    R->>R: rebuild 8 panel meshes if projections/viewports changed
    R->>R: choose flat / loading / black texture, resize panel meshes for aspect
    R->>R: alignment.update or alignment.measure (once per engine frame)
    loop each drawable
        R->>AR: queryDeviceAnchor(at presentationTime)
        alt not tracked
            R->>GPU: encode cleared pass, no content, encodePresent
        else tracked
            R->>R: auto or manual recenter -> neutralOriginFromHead
            R->>H: halo_settings_set_head(yaw, pitch), set_head_roll(roll)
            R->>R: tracking areas, menu input -> control.pointer(u, v, onPanel, action)
            R->>GPU: encode(drawable, view-projection per eye)
            R->>L: drawable.encodePresent
        end
    end
    R->>GPU: addCompletedHandler (GPU ms, errors, release lease)
    R->>GPU: commit
    R->>R: control.didSubmitFrame (counters, world cadence)
    R->>L: endSubmission (deferred)

Loading

Details of drawFrame() (L588-L861):

1. Frame acquisition. If queryNextFrame() returns nil, the loop sleeps 1 ms. queryDrawables() (visionOS 26) returning [] cancels the frame, and per CompositorServices' frame.h the frame must not be touched again, including endSubmission. endSubmission is otherwise deferred to the end of the function. Before the first submitted frame, milestones (frame.queryNextFrame, frame.acquired, frame.predictTiming, ..., frame.commit) are written to the trace. This separates an ARKit wait from a paused compositor.
2. Source selection. Unless the loading card is shown, the presenter calls panorama.refresh(leasedTo: command). If that returns nothing and the source is in the flat state, it calls frame.refresh(expectedSequence: panorama.latestFlatSequence). panoramaActive requires all ten textures.
3. Panel meshes. The eight projections (six ring layers, then zenith and nadir) and viewports (the caps use the full rectangle) form a signature. When it changes, all eight panel meshes are rebuilt together, because every panel's fade is measured against the panels already beneath it. If any allocation fails, the old meshes are kept, and if the count is still wrong, panorama drawing is disabled for the frame.
4. Flat fallback rule. flatTexture is used only when panorama.allowsFlatFallback (source status 0, flat/menu). A failed world pass has an anamorphic centre framebuffer and may already have extracted its HUD, so in that state the presenter keeps the last complete world or shows black (L654-L658). The loading texture takes precedence over both.
5. Aspect. The aspect comes from the first panorama texture or the source texture. When it changes, the panel, HUD and HUD-shadow vertex buffers are replaced with new buffers rather than mutated in place, because earlier GPU submissions may still own them.
6. Surround mesh. It follows whichever surface is on screen: (π/2, panoramaHalfHeight) in game, menuExtent(aspect) in menus. Its textures are the centre view per eye, or the menu image.
7. haptics.pump() (see Haptics).
8. Submission record. The mode is one of loading, panorama (the leased snapshot is the latest source epoch and the status is complete), retainedPanorama, flat, or worldUnavailable. It is recorded along with sequence, epochs, layer epochs, flat sequences, pool slot, status and failure reason.
9. Layer alignment. This runs once per engine frame (the snapshot sequence). With the setting on and its pipeline built, alignment.update returns per-panel constants. Otherwise alignment.measure only records each layer's camera offset as the A/B baseline. See Layer Alignment.
10. Per drawable. See Head pose, recentering and reclined play and Menu input. The view-projection for each view is projection * inverse(neutralFromHead * view.transform), with computeProjection(convention: .rightUpBack, viewIndex:).
11. Completion handler. Records first-frame GPU success or failure. Calls control.didCompleteGPU(error:gpuMilliseconds:) with gpuEndTime - gpuStartTime, and reports GPU errors to the status line. Panorama leases are released by handlers registered in EnginePanoramaTexture.
12. control.didSubmitFrame(trackingMissing:submission:) updates counters and the world cadence. A status line reports transitions between tracked and untracked.

Tracking loss

If queryDeviceAnchor(atTimestamp:) is nil or not tracked, the presenter still has to present the acquired drawable. It sets drawable.deviceAnchor = nil, encodes the passes with drawContent: false (cleared, opaque black), presents, resets menu input, and counts a tracking-loss frame (L735-L746). The GPU test's "blank" case asserts these frames are fully opaque black.

Head pose, recentering and reclined play

The game is played with the controller while reclined. Head motion does not aim. The presenter keeps a neutral pose (neutralOriginFromHead) and renders everything relative to it:

• Auto-recenter happens once, on the first frame with an active panorama (didAutoRecenterPanorama). Loading and menu frames do not fix the world to a transient head pose (L366-L368).
• Manual recenter happens when the control's recenter generation changes ("Recenter view" button). The current anchor transform becomes the new neutral pose.
• neutralFromHead = inverse(neutralOriginFromHead) * anchor.originFromAnchorTransform is the head's motion since recentering. A reclined user who recentres while lying back sees the picture straight ahead of that pose.
• The gaze direction -neutralFromHead.columns.2 becomes halo_settings_set_head(yaw = atan2(x, -z), pitch = asin(y)). The source comment says the mixer pans against this, so a sound holds its place in the world when only the head turns. The head roll, asin(neutralFromHead.columns.0.y), goes to halo_settings_set_head_roll, which the centre passes render with (L770-L779). See Runtime Settings and Audio System.
• Simulator head pose. HALO_SIM_HEAD=yaw,pitch (degrees) replaces the rotation part of neutralFromHead, so a simulator capture can look at any bearing or straight up. The comment says it is never set on the device (L355-L360, L763-L769).

The submission record keeps neutralOriginFromHead, each eye's projection and each eye's neutralFromEye as flat 16-float arrays for the diagnostics report.

Screen geometry

All geometry is in EngineImmersiveScreenGeometry. The coordinate frame is the neutral head frame: forward is -Z, up is +Y, right is +X. The vertex type is EngineImmersiveScreenVertex { position: float4, uv: float2, weight: float2 }, where weight.x is the panel alpha.

Constant	Value	Meaning
panoramaRadius	4.25	Radius of the panorama sphere (all panels)
panoramaHalfAngle	π/6 (30°)	Half a forward band's share of bearing, used only to size the surround
menuRadius	3.1	Interface cylinder, nearer than the picture so it cannot intersect it
menuHalfAngle	19°	Apparent half height of the panel, constant regardless of distance
shadowRadius	3.7	HUD shadow cylinder
HUD shadow offset	0.33° right and down
panelFeatherAcross, panelFeatherUpDown	0.15 (of half width)	Edge fade width. Measured, not guessed: 0.18 left a 2 % dim band and 0.30 left 19 %.
panoramaPanelDivisions	64 (PANEL_DIVISIONS env override)	Grid per panel edge. 48 measured a 0.6 % dip; 64 measures none.
Menu panel segments	96 (194 vertices, 576 indices)
Surround grid	72 columns x 36 rows, shell radius 1.9 * radius	Behind every content surface

The curved interface panel

make(aspect:), hud(aspect:) and hudShadow(aspect:) build a vertical strip of a cylinder (L32-L52, L125-L144):

preferredHalfHeight = menuRadius * tan(19°)
sweep      = min(π, 2 * preferredHalfHeight * aspect / radius)
halfHeight = radius * sweep / (2 * aspect)
vertex(n)  = (r sin θ, ±halfHeight, -r cos θ), θ = (u - 0.5) * sweep + thetaOffset

This gives square pixels measured along the surface. The width is the arc length sweep * radius, and the ratio width / height equals the image aspect exactly; ImmersiveAspectValidation.swift checks this for aspects 4:3, 16:9, 1.6, 1, 32:9 and 8, along with the uv corners. The sweep is capped at 180°. The front-end menu, the pause menu and the in-game HUD all use this same size.

menuHit(origin:direction:halfSweep:halfHeight:) (L101-L116) intersects a ray with the cylinder x² + z² = menuRadius², taking the far root because the viewer is inside. It returns texture coordinates (0,0 at top left) and onPanel.

The panorama sphere

Each of the eight panels is one engine camera's image placed on the sphere. panoramaAngles (L167-L171) mirrors the host's schedule in panorama_hooks.inc: yaw −60°, 0°, +60°, +120°, 180°, −120°, then pitch +90° (sky) and −90° (floor).

cameraBasis turns the whole basis when pitching, as the host does. At 90° up, the camera's own up axis becomes the direction the head was facing. That makes the sky camera's up point behind the viewer, and PanoramaSphereCoverageValidation.swift checks that the caps are not vertically mirrored against the bands.

sphericalPanorama(sources:texelSize:) (L335-L371) builds a (divisions+1)² grid per panel. For image coordinates (s, t) ∈ [-1, 1]²:

ray      = right * (s / projection.x) - up * (t / projection.y) + forward
position = normalize(ray) * panoramaRadius
uv       = viewport centre + (s, t)/2 * viewport size, clamped half a texel inside the viewport
weight.x = panelAlphas(sources, direction)[panel's draw position]

Because every vertex lies on the same sphere, no two panels can meet at a step in depth. The coverage test asserts that the radius varies by less than 0.001.

Panel fades

panoramaPanelOrder = [0, 1, ..., 7]. The ring goes down first, walking round so each band meets one already drawn, and the sky and floor go last. Drawing the caps first was tried and was worse: a cap drawn first has nothing beneath it and must stay solid to its frame edge, which left a 3 % dim seam where two such edges crossed the ring (L241-L253).

panelAlphas(sources:direction:) (L303-L317) walks the panels in draw order, keeping laid = coverage so far:

edge  = Π smoothstep ramps over the enabled edges (feather 0.15)
alpha = 1 - (1 - edge) * laid      // fade only as much as something lies beneath
laid += alpha * (1 - laid)

A panel therefore dissolves only where something already lies beneath it. At the edge of what any camera covers it stays solid. Ordinary over blending is its own normaliser: whatever the alphas, the result stays a proper mix of the pictures, so a coarse mesh cannot leave a bright or dark band. The source comment notes the structural limit. One alpha per panel cannot be right for both a sideways and a vertical fade, which is why the feather cannot simply be widened. Accumulating colour times weight and dividing at the end is described as "the next structural step".

The surround (backdrop)

backdrop(halfSweep:halfHeight:) (L61-L88) is a full sphere at radius 8.075 (1.9 x 4.25). It carries the uv coordinates the content surface would have if it continued past its edges, deliberately outside 0...1. Rows stop just short of the poles (0.995 π) because tan is unbounded there. It is drawn only when there is no panorama. With eight panels covering the whole sphere it can never show, and it cost 13 texture reads per fragment over the whole view for both eyes. The source comment lists this among the reasons the headset presented at half rate (L968-L976).

Encoding a drawable

encode(drawable:matrices:command:texture:panoramaSnapshot:alignment:drawContent:) (L921-L1064):

1. If tracking areas are active and a panorama is present, encodeBands dispatches hud_rows (one thread per HUD row) and hud_bands (one thread) into a pair of alternating shared buffers.
2. Layout: .dedicated issues one pass per view (slice from the view's texture map). Otherwise one pass covers all views, with vertex amplification. .layered passes use renderTargetArrayLength = viewCount.
3. The pass comes from EngineImmersiveRenderTargets.makePass. If allocation fails, a single-sample cleared pass is still presented, and the control stops with "Unable to allocate immersive antialiasing textures."
4. The rasterisation rate map (foveation) is applied when present.
5. Viewports and MTLVertexAmplificationViewMapping are set per selected view. eyeBase is 0 for amplified passes or the pass index for dedicated passes, so the fragment stage always knows which eye it is shading.
6. Draw order:

Step	Condition	Pipeline / depth	Textures
Surround	no panorama	backdropPipeline, depth	centre view or menu image per eye; brightness = halo_settings_backdrop_brightness()
Eight panels in panoramaPanelOrder	panorama	alignedPipeline (if constants for all 8) or panoramaPipeline; panoramaDepth (always)	ring panel: view(eye: 0/1, sector: layer); sky/floor: the zenith/nadir view for both eyes. filter = 1 (cubic) only for panel 1, the centre.
HUD shadow	panorama	overlayPipeline, overlayDepth	HUD, shade = 0.55, hudShadowVertices
HUD	panorama	same	HUD, shade = 1, hudVertices
Flat panel	no panorama	pipeline	texture (flat frame, loading card or black) for both eyes, filter = 1

Shaders

The shader source is a Swift string compiled at runtime with makeLibrary(source:) (L1066-L1277). The GPU tests extract it verbatim from the file.

Function	Purpose
curve_vertex	matrices[eye] * position; passes uv, eye = eyeBase + amplification_id, weight.x
cubic_sample	Mitchell-Netravali (B = C = 1/3) reconstruction in nine bilinear taps (the two middle taps per axis folded into one, valid because their weights are positive). Catmull-Rom outlined every magnified stair-step; Mitchell keeps edges without drawing steps. Panorama texels are magnified several times (vertically 3-4x more than horizontally).
pick	Left/right texture by eye, cubic
curve_fragment	filter == 1 uses cubic, otherwise one bilinear read. Alpha comes from weight (fades inside overlaps). Output is opaque apart from that weight, because the D3D back buffer has no meaningful display alpha. Beyond the source rectangle the colour blends towards the average of its own row (eight taps) and fades out over 0.55 uv.
backdrop_fragment	Clamped border texel softened by five taps, cross-faded towards the row average, fade = (1 - smoothstep(0, 0.85, distance))², multiplied by brightness
hud_shade / hud_fragment	The interface is premultiplied sRGB bytes. The shader unpremultiplies, decodes sRGB to linear and re-premultiplies. Coverage is max(a, r, g, b), because Halo draws the reticle and glows additively with colour but no alpha; dividing by alpha alone made them vanish. shade < 1 produces the shadow: black at a * shade.
hud_fragment_tracked	As above, and writes values[band] to colour attachment 1 for lit pixels (cover > 0.05) of the row's band, while the menu is up
hud_rows (compute)	Per HUD row, the first and last lit column (every second pixel, threshold 0.05) or 0xFFFF
hud_bands (compute)	Runs of lit rows, allowing gaps of up to two rows, become bands 1...32 from the top, each with row and column extents

The aligned variants aligned_vertex / aligned_fragment are covered in Layer Alignment.

Panorama consumption (EnginePanoramaTexture)

refresh(leasedTo:) (EnginePanoramaTexture.swift:137-152) tries the zero-copy lease first when enginevision_panorama_gpu_enabled(), then falls back to the copy pool.

Zero-copy lease

refreshGPU (L158-L187):

1. enginevision_panorama_gpu_latest(&lent) takes one lease and copies the info.
2. The texture pointers are unpacked with EnginePanoramaLayerArray.pointers(of:), which walks the C array's memory rather than naming tuple members. When the sphere grew from seven to ten layers, hand-named members kept only seven, every frame failed the count check, and the headset showed black while the Mac composited perfectly (EnginePanoramaLayerArray.swift).
3. The snapshot is rejected (lease released at once, sourceStatus = 2, failureReason = 5) unless there are exactly ten textures, the width and height match the info, and the projections validate.
4. A completed handler on the command buffer calls enginevision_panorama_gpu_release(slot). The runtime therefore cannot render into the slot until the compositor has finished sampling it.

Copy pool

refresh() (L240-L330) is the byte path, used on the Mac or with HALO_PANORAMA_GPU=0:

• Up to 3 slots of ten bgra8Unorm textures. A slot is reusable only when it is not current, has no in-flight command buffers (inFlight counted by completion handlers) and is not reserved.
• validatedByteCount requires 1...4096 for width and height and byte_count == w * h * 4 * layers, computed with overflow checks.
• If the same sequence is already current, it is reused without copying.
• If every slot is protected, the current snapshot stays published and the newer sequence is retried on the next compositor frame. The panorama texture test asserts that no fourth slot is allocated.
• A failed copy keeps the last complete snapshot, unless the copied info explicitly reports a flat frame (status == 0, flat_sequence > 0). In that case it clears, so the menu shows rather than stale gameplay.

Validity and state

Projections.isValid (L37-L49) requires, for every ring layer and both caps:

• the projection is finite and in (0, 3);
• the viewport lies inside [0, 1] with positive extent;
• projection.x / viewport width ≤ 1.7321.

If any layer fails, the frame is not a complete sphere and is not presented as one. maximumY is the largest ring projection.y / viewport height, used to size the surround.

sourceStatus, failureReason, latestFlatSequence and latestSourceEpoch are updated from every info read. allowsFlatFallback is sourceStatus == 0.

Transition hold. When the info reports no panorama and the status is not "incomplete world", the last complete snapshot is held for up to 22 compositor frames (about a quarter of a second at 90 Hz) before clearing. Crossing between the front end and a mission, the engine can report a plain frame for a moment, and dropping the view for those frames read as a flash (L396-L405). With status 2 (incomplete world), the last complete snapshot is retained indefinitely.

sRGB views. Each texture gets a cached bgra8Unorm_srgb (or rgba8Unorm_srgb) view, keyed by object identity; the cache is cleared above 64 entries. Sampled raw, every byte was taken as linear light and re-encoded: black 8 became 28, a nebula of 34 became 90, and the whole sphere read as washed-out grey (L75-L84). The HUD texture is sampled raw, because its shader decodes it.

Snapshot also carries layerEpochs, layerPoses (nine floats per layer, nil when all zero), cutEpoch, flatSequence, stereo, the zenith and nadir projections, and poolSlot. Layer alignment uses these.

Zero-copy pool (runtime side)

Implemented in EngineVisionRuntime.m:134-457 under runtime_lock:

Element	Detail
Slots	PANORAMA_GPU_SLOTS = 3, each with HALO_PANORAMA_LAYERS private BGRA8Unorm textures that are both shader-read and render targets (world layers arrive through the FXAA pass)
States	FREE, RENDERING, READY, RETIRING (retired but still leased)
gpu_acquire(w, h)	Picks a FREE slot with no leases that is not gpu_latest; reallocates on size change (max 8192); returns texture pointers to the host. No slot means dropped++.
gpu_carry(slot, mask, metadata)	Copies layers the rotating schedule did not redraw this frame from the last published slot. Every frame publishes a different slot, so an untouched layer would otherwise show what that slot held two or three frames ago, or black. The copies ride the same command buffer as the engine's blits and carry over the source's epochs, projections, viewports, poses and cut epoch.
metalwin_present_gpu	Builds the info, calls the frame pacer, then mr_commit_async(gpu_published). An incomplete frame releases the slot without committing.
gpu_published(ok)	On failure: publish_failed++ and retire. If an older world frame completes late, or a menu frame has superseded it: publish_superseded++ and retire. Completion callbacks may arrive out of order. Otherwise mark READY, retire the previous latest, update panorama_info, and record a publish-lag sample.
enginevision_panorama_gpu_latest	Records the display latch for the pacer. Leases only if the latest slot is READY, newer than the last flat frame (gpu_flat_sequence), and of the current scene epoch.
enginevision_panorama_gpu_release	Decrements the lease count; a RETIRING slot with no leases becomes FREE

PanoramaPublicationValidation.m includes the runtime source and checks this logic deterministically: out-of-order completions, failures, carry metadata (including poses and the cut epoch), flat-frame precedence, scene-epoch blocking, lease retirement, the paced commit order, and display latch and period sampling.

Menu input (look and pinch)

visionOS gives apps no continuous eye ray. It gives spatial events: an indirectPinch carries the system's gaze selectionRay at the moment of the pinch, and on visionOS 26 also the tracking-area identifier. pointer events (mouse or simulator) are accepted too.

• handle(spatial:) (L469-L497) runs on the compositor's event callback under pinchLock. It feeds EngineMenuInput.update(id:phase:target:), and keeps feeding terminal events while the menu is hidden so suppressed IDs retire.
• EngineMenuInput (EngineMenuInput.swift):
  • Keeps the first valid ray per interaction ID and commits a click only on .ended.
  • Ignores updates from another hand, and never lets a missing or invalid ray borrow another interaction's target.
  • Disabling it drops everything and suppresses still-held IDs.
  • Backlogs are capped at 16. drain() returns the newest active target as preview plus the completed clicks.
• In drawFrame, once per frame and only while enginevision_menu_active() and no loading card is shown, the presenter transforms each target into the neutral frame and hits the menu cylinder (menuHit).
  • If a tracking-area band is known for the pinch, v snaps to the band's centre row, and u is clamped into the band's lit extent (falling back to its centre).
  • Each click becomes control.pointer(u, v, 1, 2) (HOST_POINTER_TAP). The preview becomes control.pointer(u, v, onPanel, 0).
  • With the "Head pointer" setting (gaze_pointer), the head's forward ray previews when no pinch is active.
  • With no menu, input is reset and the host receives HOST_POINTER_CANCEL.
• registerTrackingAreas adds 32 tracking areas (identifiers 1...32, .automatic hover effect) on each drawable while a menu is up. The rows are measured from the HUD's own pixels each frame, so no menu layout is hard-coded. The band a pinch names can be at most two frames old.

The host side (host_pointer_set, the cursor servo) is described in Input and Controllers.

Loading card

EngineImmersiveLoadingTexture.updateIfNeeded(state) (EngineImmersiveLoadingTexture.swift:24-32) re-rasterises only when the EngineLoadingState (title, detail, progress, show) changes. It draws a 1024x640 premultiplied BGRA bitmap with CoreGraphics and CoreText: "HALO" in cyan, the title, the detail text, and an optional progress bar with percent. It is returned as an sRGB view and drawn on the same curved panel as engine frames. EngineAppSession.refresh sets the state (see visionOS App), and EngineImmersiveSession.setLoading forwards it to the control only when it changed.

Frame counters, GPU time and world cadence

EngineImmersiveControl keeps the following (L139-L184):

• submitted, cancelled, tracking-loss, GPU-completed and GPU-failed frame counts, and the last GPU error;
• GPU milliseconds per compositor frame as an exponential average (avg * 0.98 + sample * 0.02, seeded with the first sample) and a maximum. At 90 Hz the budget is 11.1 ms, and a compositor that takes longer is silently halved to 45 Hz;
• the last EngineImmersiveSubmission;
• an EngineWorldCadence.

EngineWorldCadence.record(scene:epoch:time:eligible:) (EngineWorldCadence.swift:19-43) measures how often a new world epoch is submitted.

• It only counts frames that are eligible: tracking present and mode panorama or retainedPanorama.
• An ineligible frame, a scene change, a regressed epoch or a non-finite time restarts the observation. The scene-change case is what keeps menus and loads out of the gameplay statistics.
• Intervals between distinct epochs, and how many submissions each epoch was held for, are kept in a 4096-entry ring.

statistics (L45-L78) reports:

Key	Meaning
uniqueSubmittedWorldFrames, repeatedSubmittedWorldFrames	New epochs; re-submissions of the same epoch
intervalSamples	Retained intervals (≤ 4096)
uniqueIntervalP50/P95/P99Milliseconds	Percentiles of new-epoch intervals
uniqueIntervalModal90HzPeriods, uniqueIntervalModalShare	Most common interval in nominal 90 Hz periods (round(ms * 0.09)), and its share
uniqueIntervalModalSubmissionFrames, uniqueIntervalModalSubmissionShare	Most common number of actual submissions per epoch, and its share
uniqueIntervalStdDevMilliseconds	Evenness
stallsOver100ms, maximumRetainedMilliseconds	Long gaps; longest time one epoch stayed on screen

These are app submission observations, not proof of display scanout (L3-L4). WorldCadenceValidation.swift checks each of these:

• even 30 fps at 90 Hz gives 3 periods with share 1;
• holds of 5, 6, 7 and 6 frames give a mode of 6 with share 0.5;
• at 45 Hz, 3 submissions span 6 nominal periods;
• scene boundaries discard unfinished holds;
• stalls are counted;
• the ring is bounded at 4096.

Lifecycle trace

EngineImmersiveTrace.shared (EngineImmersiveTrace.swift) is a lock-protected list of the last 128 events. Each entry has a number, seconds since the trace started, the event name, the layer lifecycle ID and the layer state, and every entry is also logged with NSLog. It survives renderer invalidation, and report.json includes it as immersiveLifecycle. Recorded events include:

• app.scenePhase, app.dismissSetup;
• setup.appear/disappear, setup.openImmersive.begin/end, setup.dismissImmersive.*;
• configuration.begin/end;
• session.prepare/start/finish/closeRequest;
• renderer.init.begin/end, renderer.failed, arkit.start.begin/end;
• renderLoop.enter/exit, layer.state, layer.wait.begin/end;
• first-frame milestones frame.*, frame.firstGPUCompleted/Failed.

Simulator capture

Variable	Effect
HALO_SIM_CAPTURE=1	Every 60th composited frame, the left eye's colour target is copied and written as a PNG to Documents/SimCaptures/composite-NNNNN-<yawX-pitchY or head>.png. View-0 tangents and the view-projection are logged. The simulator's own screenshots do not show Metal layers.
HALO_SIM_CAPTURE_LAYERS=1	Also writes every panorama layer at frame 120 and every 600th frame (layerKK-NNNNN-...png)
HALO_SIM_HEAD=yaw,pitch	Scripted head rotation (see above)

Source: EngineImmersive.swift:863-919.

Environment variables and settings

Name	Default	Effect	Read at
HALO_EYE_MENU	unset (off)	1 asks for tracking areas (eye-hover menus)	EngineImmersive.swift:52
HALO_SIM_HEAD	unset	Scripted head yaw/pitch (degrees)	L356
HALO_SIM_CAPTURE	unset	PNG captures of the composite	L869
HALO_SIM_CAPTURE_LAYERS	unset	Also capture each layer	L870
PANEL_DIVISIONS	64	Panel grid divisions	EngineImmersiveScreenGeometry.swift:260-261
HALO_PANORAMA_GPU	on (visionOS), off (Mac)	Zero-copy lease vs byte copy	EngineVisionRuntime.m:704
UserDefaults HaloTrackingAreasAttempted / ...Succeeded	false	Tracking-area safety latch	EngineImmersive.swift:13-14
halo_settings_backdrop_brightness()	HALO_BACKDROP 1.0	Surround brightness	encode
halo_settings_layer_align()	HALO_LAYER_ALIGN off	Aligned pipeline	drawFrame
halo_settings_gaze_pointer()	HALO_GAZE_POINTER off	Head-follow menu cursor	drawFrame

Failure modes

Symptom	Mechanism
Black sphere while the host reports complete frames	Zero-copy lease failed validation: layer count, size or projections. sourceStatus = 2, failureReason = 5; check panoramaPool* counters in the report.
Menu flashes when entering a mission	Mitigated by the 22-frame transition hold
Gameplay world flashes over a newer menu	Prevented: gpu_flat_sequence and scene_epoch checks in publish and lease
Blank frames	Tracking loss (counted as immersiveTrackingLossFrames)
Game stuck behind setup window	Space could not present (for example with tracking areas). The latch disables tracking areas on the next launch, and the setup window auto-opens the space at the first engine frame.
Compositor at 45 Hz	Check immersiveGPUMilliseconds against 11.1 ms

Testing

Test	Asserts	Run by
ImmersiveGPUValidation.swift	The shipped curve_vertex/curve_fragment on the real GPU for layered, shared and dedicated layouts at MSAA 1 and 4. Each eye's quad lands at its own position (mean x 50.5 / 76.5); output is opaque even with alpha-0 input; the background is cleared; blank (tracking-loss) frames are fully opaque black; MSAA produces partial edge pixels.	Manual (macOS Metal)
ImmersiveAspectValidation.swift	Panel arc/height ratio equals the image aspect; sweep ≤ 180°; 194 vertices, 576 indices; full uv range	Manual
ImmersiveOwnershipValidation.swift	Generation interleavings	run_source_checks.py
MenuInputValidation.swift	No menu means no input; two taps between frames both survive; delivered once; another hand cannot release the first; a pinch keeps its original target; missing or invalid rays never click; cancelled pinches never click; reset suppresses a held pinch; hidden-menu terminal events retire reused IDs; backlog bounded at 16	run_source_checks.py
WorldCadenceValidation.swift	See above	run_source_checks.py
PanoramaPublicationValidation.m	Runtime pool, publication order, carry, leases, frame split, pacing, display sampling	run_source_checks.py
PanoramaLeaseLayerCountValidation.swift	The imported textures tuple has HALO_PANORAMA_LAYERS slots, and EnginePanoramaLayerArray.pointers returns all of them in order with none null	Manual (swiftc command in its header)
PanoramaSphereCoverageValidation.swift	All panels on one sphere; caps not mirrored; compositing the mesh-interpolated alphas over a 1° grid of the whole sphere leaves no hole and a shortfall under 1 %	Manual
PanoramaProductionGPUValidation.swift	Production shader and geometry at vertical FOVs of 90/100/110° cover > 99.5 % of the target including the top and bottom rows; never sample outside the source viewport (magenta sentinel); a wide-clip case agrees with an analytic reference	Manual
PanoramaCompositePreview.swift	A tool, not a test: composites captured layers through the production shader from any yaw/pitch to a PPM; optional bilinear comparison	Manual
FrameTextureValidation.swift	Immutable flat-frame snapshots	Manual
PanoramaTextureValidation.swift, PanoramaViewportValidation.swift	The copy pool (three slots, in-flight protection, failure retention, resize) and the older three-band viewport geometry	Not run by run_source_checks.py. Both target earlier APIs: a four-layer fixture, and EngineImmersiveScreenGeometry.panorama(sector:...) / panoramaFrontEdgeEpsilon, which no longer exist in the sources. Treat them as historical.
LayerAlignmentValidation.swift, LayerAlignmentGPUValidation.swift	See Layer Alignment	run_source_checks.py

Related pages

• visionOS App
• Layer Alignment
• Panorama System: host-side layer production, epochs, poses, carry masks
• Panorama Budget and LOD, Frame Pacing
• Metal Renderer: engine-side Metal, mr_commit_async, blits
• Input and Controllers: host pointer and menu cursor
• Runtime Settings, Haptics, Audio System
• Diagnostics and Telemetry