Skip to content

Panorama System

M T edited this page Oct 4, 2026 · 1 revision

Panorama System

The panorama is how Halo Vision turns Halo PC's single flat camera into a picture that surrounds the viewer. On every engine frame the host intercepts Halo's own renderer entry (0050BEA0) and runs it several times. Each run turns the game camera to a different fixed bearing, and the result of each run is captured into its own layer. Eight world bearings (six around the horizon, plus straight up and straight down) cover the whole sphere. A ninth pass draws the forward bearing a second time for the right eye, and a flat HUD layer carries the interface. These ten layers, together with per-layer projection, viewport, epoch and camera metadata, go to the visionOS presenter, which maps each layer onto a panel of one sphere and cross-fades the joins.

The panorama sits between the translated engine (which still does all scene traversal, visibility, animation and shading) and the Immersive Presenter. The host never replaces camera or player globals, game scripts or save data. It changes only the renderer record the game passes to its own renderer, and it puts that record back after every pass.

This page covers the camera hook, the layer layout, the host-side producer state machine (epochs, cuts, failures), HUD/interface separation, and the contract with the Swift consumer. Which bearings are redrawn each frame, and the level-of-detail fix that keeps one object consistent across bearings, are on Panorama Budget and LOD. Publication cadence is on Frame Pacing.

Source files

File Role
native/EngineHost/panorama.h Public contract: layer indices, validity masks, status/failure enums, HaloPanoramaInfo, the zero-copy HaloPanoramaGPUSink, and the producer API.
native/EngineHost/panorama_hooks.inc host_panorama_dispatch: the 0050BEA0 multi-pass hook, the projection-builder hook (0050CC40), overlay/interface scoping, letterbox filtering, first-person cache handling and the LOD dispatch. Included into overrides.c.
native/EngineHost/panorama_render.inc The producer: per-pass begin/end, HUD target, projection recording, held layers, epochs, camera-cut detection, CPU readback or GPU sink blits, carry of undrawn layers. Included into d3d9_render.inc.
native/EngineHost/panorama_overlay_scope.h The guest addresses treated as camera-relative overlays (weapon, HUD, UI screens) and the interface record test.
native/EngineHost/panorama_budget.h Bearing scheduler (see Panorama Budget and LOD).
native/EngineHost/panorama_lod.h Bearing-invariant object pixel size (see Panorama Budget and LOD).
native/EngineHost/d3d9.c Present: carries missing GPU layers, then publishes through the zero-copy, dropped, or CPU path. Reset invalidates the panorama.
native/EngineHost/d3d9_render.inc Draw-time routing: HUD draws to the HUD target, UI draws skipped in side passes, a fallback projection taken from fixed-function draws.
native/EngineHost/metalrenderer.m mr_blit_target_to, mr_fxaa_target_to, mr_blit_texture_copy, mr_commit_async, and the overlay-target pipeline variant used by the HUD layer.
native/EngineHost/halo_settings.c Live settings seeded from the environment: stereo separation, vertical FOV, target FPS, spatial menu, layer alignment, frame pacing, head pose.
native/EngineVision/Sources/EngineVisionRuntime.m Platform bridge: the three-slot zero-copy texture pool, carry, publication completion, compositor leases, and the CPU byte-copy path.
native/EngineVision/Sources/EnginePanoramaTexture.swift Swift consumer: leases a published slot per compositor command buffer, validates projections, exposes per-eye/per-bearing textures.
native/EngineVision/Sources/EnginePanoramaLayerArray.swift Walks C fixed arrays imported as Swift tuples, so the layer count can never fall behind the header.
native/EngineVision/Sources/EngineImmersiveScreenGeometry.swift Panel geometry: mirrors the host's bearing table and builds one spherical panel per camera with edge fades.
native/EngineVision/Sources/EngineWorldCadence.swift Measures distinct world epochs as the compositor submits them (see Frame Pacing).

What the panorama is

Layers, bearings and fields of view

The panorama is not a cubemap or an equirectangular image. It is a set of ten ordinary rectilinear 2D images, each the size of Halo's back buffer, each drawn by Halo's own renderer through a turned camera. The presenter projects each one back onto a sphere using the projection the engine actually used (panorama.h:14-40).

Layer Constant Bearing (yaw from forward) Pitch Eye Drawn
0 -60 deg (left) 0 both once, shown to both eyes
1 HALO_PANORAMA_CENTRE_LEFT 0 (forward) 0 left (or both in mono) every frame; carries the HUD and interface
2 +60 deg (right) 0 both once
3 HALO_PANORAMA_HUD_LAYER flat HUD both redrawn with layer 1
4 HALO_PANORAMA_CENTRE_RIGHT 0 (forward) 0 right only when stereo is active
5 HALO_PANORAMA_UP 0 +90 deg (zenith) both once
6 HALO_PANORAMA_DOWN 0 -90 deg (nadir) both once
7 HALO_PANORAMA_REAR_RIGHT +120 deg 0 both once
8 HALO_PANORAMA_REAR 180 deg 0 both once
9 HALO_PANORAMA_REAR_LEFT -120 deg 0 both once

Positive yaw turns toward the camera's right vector: the pass computes f' = f cos(a) + right sin(a) (panorama_hooks.inc:626). The presenter's head yaw is atan2(gaze.x, -gaze.z) (EngineImmersive.swift:772), so positive is to the right as well.

Eight distinct pictures make up the ten layers. Only the forward bearing is drawn once per eye, because "the engine, not the GPU, is what limits this port and depth is worth paying for only where the viewer is looking". A complete sphere is therefore nine engine passes in stereo and eight in mono.

HALO_PANORAMA_SECTOR_OF(layer) maps layer 4 to 1. It gives the bearing a layer shows, ignoring the eye.

Validity masks. A frame counts as a complete sphere when the persistent ready mask covers:

  • HALO_PANORAMA_MONO_MASK = 0x3E7 (layers 0,1,2,5,6,7,8,9), or
  • HALO_PANORAMA_STEREO_MASK = 0x3F7 (the same plus layer 4).

Layer 3 (HUD) is deliberately absent from both masks. The HUD is redrawn every frame rather than scheduled, and Halo hides it during cutscenes. Requiring it "would reject frames that are perfectly good" (panorama.h:34-40).

Vertical field of view. Every pass writes the configured vertical FOV into the frustum field +0x28 (panorama_hooks.inc:638-648). The default is 105 degrees (HALO_PANORAMA_VFOV, clamped to 60..175). The settings panel offers 90..175 (EngineSettingsView.swift:86-88). The settings comment explains the default: at 150 degrees the centre gets under 3 pixels per degree, and 105 nearly triples that (halo_settings.c:52-57).

Horizontal field of view (DENSE). With HALO_PANORAMA_DENSE set (the headset default), the projection builder sees a narrowed rectangle. The raster stays full size, but its pixels are spent on a narrower horizontal field (panorama_hooks.inc:267-290):

  • Ring bearings: half-angle 32 deg, so 64 deg horizontally. That is the 60 deg sector plus a 2 deg guard each side.
  • Zenith and nadir caps: half-angle 48 deg, so 96 deg. At 32 deg the caps could not reach the sides of the hemisphere. That left "four uncovered wedges past 78 degrees of bearing between 50 and 57 degrees of elevation", and the bands need overlap with the caps to dissolve into them. HALO_PANORAMA_CAP_HALF_FOV (20..80) overrides the cap value only.

Without DENSE, the horizontal field follows the back buffer aspect. At 4:3 and 105 deg vertical that works out to about 120 deg horizontally. This figure is derived from the formula, not stated in the code.

The narrowed width is round(height * tan(half) / tan(fov/2)). It is applied only when it is at least 1, below the full width, and does not overflow int16.

Resolution. Layers are back-buffer-sized. On visionOS the engine starts at -vidmode 2048,1536,60 unless HALO_VIDMODE or the stored HaloRenderResolution default overrides it (EngineVisionRuntime.m:672-696).

Forward stereo and peripheral mono

Stereo is on when halo_settings_stereo_separation() > 0 and the budget is not in a shared-centre tier (tier 2 or 3, see Panorama Budget and LOD) (panorama_hooks.inc:497-499). Separation is half the eye separation in Halo world units (one unit is ten feet). It is seeded from HALO_STEREO_IPD in millimetres (default 63, so 63/1000/2/3.048), HALO_STEREO_SEPARATION overrides it directly, and HALO_STEREO=0 forces mono (halo_settings.c:46-51).

Per pass (panorama_hooks.inc:586-625):

  • Layer 1 in stereo: camera shifted -separation along the head baseline.
  • Layer 4: shifted +separation.
  • Every other layer, and layer 1 in mono: on the head centre line, no shift.
  • The baseline is rolled with the head. position += (right/|right| * cos(roll) + up * sin(roll)) * shift, where roll comes from halo_settings_head_roll() (written by the presenter from the head's right axis). Without roll, a tilted head saw vertical disparity on the weapon. HALO_STEREO_ROLL=0 restores the level baseline.

The saved records are restored before every pass, so the right vector is always the head's, never the previous bearing's.

Cap orientation

For the zenith and nadir, the whole basis is pitched: f' = f cos(p) + u sin(p) and u' = u cos(p) - f sin(p). Rotating only forward would make forward and up parallel at 90 degrees (panorama_hooks.inc:627-636). At 90 deg up, the camera's own up axis becomes the direction the head was facing. The presenter's cameraBasis reproduces exactly this (EngineImmersiveScreenGeometry.swift:183-195).

The camera hook: host_panorama_dispatch

host_panorama_dispatch(cpu, address) is called from the generic override dispatcher for every translated call boundary (overrides.c:251). Returning 1 means the host handled the call. Returning 0 lets the original run. Its branches, in the order they are tested (panorama_hooks.inc:163-690):

Guest address Condition What the host does
0050F740 (object pixel size) inside a panorama frame Native bearing-invariant LOD (panorama_lod_dispatch). Tested first because it runs several times per object per pass.
00492430 (FP node builder, return 0050C09C) HALO_PANORAMA_WORLD_FP=1 Build the first-person cache once per source view from the central pose, then skip it in later bearings.
0052B050, 00518F40 inside the weapon scope Trace-only observation of first-person projection uploads (HALO_PANORAMA_FP_PROJECTION_TRACE).
004924B0, 005154A0 inside a panorama frame Scope trace (HALO_PANORAMA_SCOPE_TRACE).
005537C0 (BSP visibility) HALO_PANORAMA_TRACE Runs the original, then logs leaf/cluster/surface counts. Trace only.
0050CC40 (projection builder) callers 0050BC8B, 0050BCA3, 0050BE01, 0050BE23; not inside an overlay DENSE narrowing, central FP scale derivation, projection recording.
00449780 (solid rectangle) callers 00449B2A, 00449B9E Omit the two cinematic letterbox bars.
004924B0, 00494730, 004984C0; 0050BDC0 with EAX 0 inside a panorama frame Overlay/interface scope: run only in the views their draws can reach, routed to the HUD layer.
0050BEA0 (renderer entry) outermost only The multi-pass panorama frame.

Entry conditions for 0050BEA0

The panorama frame runs only when all of these hold (panorama_hooks.inc:380-386):

  • The call is not already nested in a panorama frame.
  • HALO_PANORAMA is exactly "1". The visionOS worker sets this by default.
  • Either the front end is not showing (shell byte 0x00718FC9 clear), or the spatial menu setting is on (HALO_SPATIAL_MENU, default on).
  • The record count (second argument, low 16 bits) is between 1 and 4.

Otherwise the host calls host_panorama_reset() and lets the original draw a normal flat frame.

The renderer ABI, verified against the lifted instructions, is 0050BEA0(Renderer*, short count, short*, float, float). Renderer records are 0xAC bytes each, with two 0x54-byte frustums at +4 and +0x58. Within a frustum, position is at +0x00, forward at +0x0C, up at +0x18, vertical FOV at +0x28, and the int16 raster rectangle at +0x2C (panorama_hooks.inc:1-5).

One panorama frame, step by step

flowchart TD
    A["Guest calls 0050BEA0"] --> B{"HALO_PANORAMA=1, count 1..4, shell allowed?"}
    B -- no --> R0["host_panorama_reset, original draws a flat frame"]
    B -- yes --> C["Save renderer records, 24 stack bytes, CPU, save central poses and original FOVs"]
    C --> D["host_panorama_reset, host_panorama_set_camera(view 0 frustum pose): cut detection"]
    D --> E{"Shell to game, or a gap of more than 1 s?"}
    E -- yes --> E1["budget scene_entry + host_panorama_invalidate"]
    E -- no --> E2["budget observe_timed(busy, period, pacer target)"]
    E1 --> F
    E2 --> F["Decide stereo, set_stereo, LOD begin for each view"]
    F --> G["budget_plan(gaze yaw/pitch), motion_fill if camera or head moving"]
    G --> H["Clear lens-flare byte unless HALO_LENS_FLARES=1"]
    H --> L{"For each schedule entry: 0, 2, 5, 6, 7, 8, 9, 4, 1"}
    L --> S{"Skip? (mono and layer 4, or all layers ready and not in the mask)"}
    S -- skip --> L
    S -- draw --> P["Restore CPU, stack, records, count=1 except layer 1, same frame counter, time step 0 except layer 1"]
    P --> Q["Yaw/pitch/eye-shift both frustums, write vertical FOV"]
    Q --> T["host_panorama_begin(layer)"]
    T --> U["engine_dispatch 0050BEA0 (timed)"]
    U --> V["host_panorama_end(layer)"]
    V --> W{"Original returned normally?"}
    W -- no --> X["host_panorama_abort, stop the loop"]
    W -- yes --> L
    L -- done --> Y["budget drawn + note_passes, restore records and stack, return 1"]
Loading

Detailed behaviour (panorama_hooks.inc:389-689):

  1. Snapshot. The hook copies count * 0xAC bytes of renderer records, 24 bytes of the stack (return address plus the five arguments), and the whole incoming EngineCPU. For each view it stores 36 bytes of the first frustum as the central first-person pose and both frustum FOVs as the original FOVs.
  2. Frame camera. host_panorama_set_camera receives view 0's first frustum (position, forward, up) before any bearing is applied. Every layer drawn this frame records it. See Pose export and camera cuts.
  3. Budget clock. Busy time is the period since the previous entry, minus the presenting-thread idle accumulated in host_yield_spin_ns (the limiter's Sleep(0) spin, Sleep(n), and the pacer's wait). A transition from shell to game, or an in-game gap of more than one second, counts as a scene entry. That resets the budget heavy and invalidates every held layer, because the carried layers "show the menu or the world before the jump".
  4. Stereo decision (described above).
  5. Plan. halo_panorama_budget_plan returns the layer mask to draw. halo_panorama_budget_motion_fill may add up to two visible neighbours while the camera or head moves. See Panorama Budget and LOD.
  6. Lens flares. Unless HALO_LENS_FLARES=1, the hook clears byte 0x006893FF and the pending queue 0x0071D134 every frame. The comment gives the reason: flares cost 12.5% of each b30 bearing pass and 32% of each a10 pass. They draw "next to nothing" on this host (occlusion queries report one pixel). Their reflections follow each bearing's axis, so they cannot stitch. The video-settings apply (00495580) sets the byte again, so it is cleared every frame (panorama_hooks.inc:529-539).
  7. Pass loop. The schedule order is fixed: 0, 2, UP, DOWN, 7, 8, 9, CENTRE_RIGHT, CENTRE_LEFT (panorama_hooks.inc:427-437). The left-eye centre is last, so the CPU state the guest resumes with, the HUD, the interface record and the render time step all come from the forward view. Until host_panorama_all_layers_ready() is true, the mask is ignored and the whole sphere is drawn.
  8. Per-pass guest state.
    • Record count: for every pass except layer 1, the count argument is rewritten to 1 so the bearing asks for the player's view alone. HALO_PANORAMA_VIEWS=all keeps the game's count everywhere (panorama_hooks.inc:565-578).
    • Frame coherence: 0050BEA0 advances the render-frame counter 0x7C3100 and stores the time step (fifth argument) at 0x7C3110 on every call. Without intervention, particles culled after 15 unseen frames were culled after 15 bearing passes. Lighting was also blended per bearing, and sky (00510C50), weather (00458420) and 004FCE80 integrated time once per bearing. The hook therefore rewrites the counter to the same value for every pass and zeroes the time step except in layer 1. HALO_PANORAMA_FRAME_PER_BEARING=1 restores the old behaviour for comparison (panorama_hooks.inc:516-582).
    • Camera: both frustums of every non-interface view are turned. Views whose saved byte +2 is set (the interface record) are left with the game's camera.
  9. Escape handling. If the original does not return to the expected address (an exception, longjmp-style unwind, or other escape), the host calls host_panorama_abort(). That marks the frame WORLD_INCOMPLETE / RENDER_ABORTED and clears readiness so the partial frame is never published. The escaped CPU state is preserved for the caller.
  10. Cleanup. Records and stack are restored, all scope flags are cleared, and the hook returns 1.

Projection hook (0050CC40)

0050CC40 builds the projection and frustum from ECX (input frustum: rectangle at +0x2C, FOV at +0x28) into ESI (output record whose 4x4 projection starts at +0x144). It is hooked only for the four world call sites: 0050BC8B and 0050BCA3 in 0050BA80 (the world view), and 0050BE01 and 0050BE23 in 0050BDC0 (a player view with no camera). Other callers, such as auxiliary culling at 0050C608 or 005545B3, run unchanged (panorama_hooks.inc:220-297). The hook is skipped inside an overlay scope (panorama_ui_nested), so the interface record's own builds can never narrow a rectangle or replace the world report.

For a hooked call:

  1. Central first-person scales. If ECX is a view's primary frustum, the hook re-runs the original builder with that view's original FOV, reads m[0]/m[5] (at +0x144/+0x158), and then restores the 0x18C-byte output record, the FOV, the 8-byte caller frame and the CPU. These scales (panorama_fp_xy) are what the weapon would have been drawn with had there been no panorama.
  2. DENSE narrowing. The right edge of the rectangle (+0x32) is temporarily moved in (see above).
  3. The original runs under a reentry guard (panorama_projection_nested).
  4. Restore and report. The rectangle is restored, then host_panorama_projection(m11, m22, x, y, w, h, caller) records the projection with the full raster rectangle.

Recording here, rather than from a draw, means frames drawn only with programmable shaders still get a valid projection. A fallback is still taken from the first depth-enabled fixed-function world draw (d3d9_render.inc:679-686).

host_panorama_projection normalises the projection to the back buffer as px = |m11| * viewport_w / bb_width and py = |m22| * viewport_h / bb_height, and stores the viewport as UV fractions. It keeps the first world caller of each pass. A later disagreeing caller is only logged under HALO_PANORAMA_TRACE (panorama_render.inc:162-182). Normalising this way preserves a cinematic letterbox (a shorter raster rectangle) as a reduced viewport rather than a different projection.

Letterbox boundary

The cinematic letterbox renderer 004499C0 draws its two black bars through the shared rectangle renderer 00449780, from return sites 00449B2A and 00449B9E. Inside a panorama frame those two calls are skipped with a plain return. Animation state, subtitles, fades, and every other caller of 00449780 still run (panorama_hooks.inc:298-312). The reduced raster rectangle remains in the reported viewport, so the presenter crops to the valid source rows.

Overlay scope: HUD, weapon, UI screens and the interface record

panorama_overlay_scope.h names the camera-relative overlay entries:

  • 004924B0: first-person weapon and hands (resolved and drawn via 004D6FC0(flags=8) before the generic 005154A0 material queue).
  • 00494730: HUD.
  • 004984C0: UI screens.
  • 0050BDC0 with EAX == 0: the game's interface record.

The interface record needs explanation. 004C9260 appends one extra renderer record after the player views (word -1, byte +2 set). 0050BEA0 hands it to 0050BDC0 with EAX 0 (0050BF2E xor eax,eax). That call draws only 2D over the finished frame:

  • the cinematic letterbox and chapter titles (004499C0)
  • the game timer (004ADD10)
  • the error/loading modal (00497410)
  • messages (00496730)
  • the framerate counter (00512530/00512E80)

It clears nothing. 0050BDC0 with EAX 1 is instead a player view with no camera, which clears the world target and stands in for the world, so it stays a world view.

The routing rules (panorama_hooks.inc:313-373):

  • Where they run. The weapon runs in layers 1, 4, 0 and 2, since a long weapon can reach across the +/-60 deg join. Every other overlay, including the interface record, runs only in layer 1. Elsewhere the entry is skipped with a plain return, because running it only to drop its draws "cost every extra bearing the time of the HUD, the menus and the skinned weapon". HALO_PANORAMA_OVERLAY_ALL_VIEWS=1 runs them in every view.
  • HUD routing. Inside the scope, host_panorama_ui(1) is set (except for the weapon when HALO_PANORAMA_WORLD_FP=1). Draws then go to the HUD target via panorama_draw_surface() in pass 1, and are dropped in every other pass (d3d9_render.inc:824, panorama_render.inc:131-161). The comment records why: drawing the interface in each bearing put the logo and menu text into the side views at different positions, "which is the seam that was visible through the menu".
  • Weapon projection (non-WORLD_FP). During 004924B0, the first-person projection XY at 0x007C13C0/0x007C13D4 is temporarily replaced with the central scales derived by the projection hook. They are restored afterwards and re-uploaded through the original 00518F40, covering both D3D SetTransform and shader constants, with a borrowed 12-byte call frame (panorama_hooks.inc:129-144).
  • Defensive cleanup. After every pass, any viewmodel/UI scope still open is closed, so "a scoped target" is never carried into another pass.

The interface record keeps its camera in every pass: the per-frustum loop skips records whose saved byte +2 is set. 0050BDC0 copies that frustum into the view globals at 0x007C3114, which then end the frame holding what they would hold without the panorama (panorama_hooks.inc:589-594).

First-person original history (HALO_PANORAMA_WORLD_FP)

0050BFB0 calls the no-argument builder 00492430 at 0050C097. That builder reaches 00493740, which updates the first-person aim history (current to previous) before it builds the first-person nodes from the view globals at 0x007C3114. Running it once per bearing fed synthetic yaws into next-tick weapon sway. The fixture test_panorama_fp_original_history.c reproduces this "three-pass yaw contamination" with the lifted original code.

With HALO_PANORAMA_WORLD_FP=1 (the headset default, set in engine_worker), the hook behaves as follows (panorama_hooks.inc:166-186):

  • On the first call for a source view in a frame, it swaps the saved central pose into 0x007C3114 for 36 bytes, runs the builder, restores the yawed pose, and marks the view built (panorama_fp_cache_built).
  • On later bearings, it returns to the caller without running the builder, reusing the central nodes.
  • If the builder escapes, the view is not marked and the pose is still restored.
  • The exact return-address guard (0050C09C) ensures a different caller is never suppressed.

In this mode the weapon is not routed to the HUD. It draws as world geometry anchored to the central frustum in each bearing that can reach it. HALO_PANORAMA_WORLD_FP_TRACE=1 logs bounded FNV hashes of one cache root and the source view matrix.

The producer: panorama_render.inc

The producer runs entirely on the engine thread (panorama.h:11-13).

HaloPanoramaInfo

Field Meaning
width, height Back buffer size shared by every layer this frame.
valid 1 when the ready mask covers the mono or stereo mask.
status HALO_PANORAMA_FLAT (0, no panorama this Present), COMPLETE (1), WORLD_INCOMPLETE (2).
failure_reason OK, MISSING_PASS, NO_PROJECTION, READBACK_FAILED, ALLOCATION_FAILED, INVALID_VIEWPORT, SIZE_MISMATCH, RENDER_ABORTED.
stereo 1 when this frame produced layer 4.
source_epoch Monotonic producer frame id. Assigned on the first actual pass of the frame.
scene_epoch Advances on every host_panorama_invalidate (cut, scene entry, Reset, flat frame).
layer_epoch[10] The source_epoch that actually drew each layer's image, including images held from older frames.
projection_x/y[10] Normalised projection scales per layer.
viewport_{u,v}_{min,max}[10] Valid source rectangle per layer in UV.
layer_pose[10][9] The frame camera (position, forward, up) each layer was drawn from, before bearing/eye offsets. Zero when unknown.
cut_epoch source_epoch of the first frame after the most recent camera cut or scene change.

Pass begin and end

host_panorama_begin(pass) (panorama_render.inc:224-283):

  • First pass of the frame, whichever layer that is (rotation may skip layer 0): resets the info, assigns source_epoch = ++counter, stamps cut_epoch if a cut is pending, and marks the frame WORLD_INCOMPLETE / MISSING_PASS until enough layers exist.
  • Back buffer. Marks the back buffer's GPU copy authoritative (gpu_dirty=1), so the stale guest copy is not uploaded before the pass clears it. Then clears it to opaque black and depth 1.
  • GPU sink. If a sink is registered and no slot is held, acquires a pool slot. A busy pool fails the frame with ALLOCATION_FAILED, and the compositor keeps its last complete frame.
  • HUD target. On a forward pass (layer 1 or 4), (re)allocates the HUD target at back-buffer size when needed, then clears it to transparent 0x00000000 and depth 1.

host_panorama_end(pass) (panorama_render.inc:284-394):

  1. Chooses the builder projection if one was recorded, else the draw-time fallback.

  2. Fails the frame if:

    • the layer index is invalid or is the HUD layer (SIZE_MISMATCH);
    • the projection is not finite and positive (NO_PROJECTION, "the path that turns a presented frame into the flat fallback");
    • the viewport lies outside [0,1] or is empty (INVALID_VIEWPORT);
    • this pass's size differs from an earlier pass in the same frame (SIZE_MISMATCH).

    The size check uses the per-frame mask, not the persistent ready mask. The comment notes that using the persistent mask failed every frame after the first and "blacked the whole view out".

  3. Clears persistent readiness on a resolution change.

  4. Zero-copy: blits the back buffer into the slot's layer texture through FXAA (mr_fxaa_target_to, subpixel 0.5), or a plain blit with HALO_FXAA=0. After layer 1 it also blits the HUD target unfiltered into layer 3. CPU: surface_flush reads the back buffer back, copies it into panorama_pixels[pass], and after layer 1 does the same for the HUD into layer 3.

  5. Records the projection, viewport, epoch and pose for the layer, and for layer 3 when the pass is layer 1. Updates the per-frame and persistent masks.

  6. Carries held metadata. For every layer the current mode shows that is ready but was not drawn this frame, it copies the held epoch, pose, projection and viewport into the info. Without this, a skipped layer published projection zero. "The compositor rejects a frame whose ring or caps carry an impossible projection, and it rejected every one: rotation on the headset was a black screen with the host reporting COMPLETE." In mono, layer 4's held data is not carried. Earlier, a stale right-eye epoch was reported 6342 frames behind.

  7. Sets valid and COMPLETE when the ready mask covers the expected mask. With HALO_FRAME_CAPTURE and HALO_PANORAMA_CAPTURE set, it dumps every layer as panorama-NNNN-K.bgra every 60th frame.

Held layers and the CPU path

On the CPU path, the host keeps one buffer per layer (panorama_pixels), so a layer the schedule skipped keeps its last pixels. A frame is "complete when every layer holds something, not when every layer was drawn this tick" (panorama_render.inc:24-32). If the layer size changes, all buffers are reallocated and readiness is cleared.

GPU carry

A zero-copy frame has no persistent buffers. Each frame publishes a different pool slot, so an undrawn layer would otherwise arrive as whatever that slot held two or three frames earlier. host_panorama_gpu_carry_missing() runs once after the passes, from Present (panorama_render.inc:395-423):

  1. Computes missing = expected_mask & ~drawn_this_frame. An inactive right eye is ignored when switching to mono.
  2. Calls sink.carry(slot, missing, &source). The sink copies those layers from the last published slot and fills source with that slot's metadata.
  3. Rejects everything if source.scene_epoch, width or height differ from the current frame.
  4. Rejects any individual layer whose source epoch is zero, newer than the current source_epoch, or whose projection is not finite and positive. Accepted layers take epoch, pose, projection and viewport from the published metadata, not from the producer's held arrays. A newer producer frame may have failed or still be pending.
  5. If any requested layer was not copied, clears readiness, sets READBACK_FAILED, and fails the frame (releasing the slot). A sink without a carry callback "simply cannot rotate": every partial frame fails.

Pose export and camera cuts

host_panorama_set_camera(pose) is informational: "nothing here changes what is drawn" (panorama_render.inc:45-111). It does three things.

  1. Usability check. A pose is unusable if any value is non-finite or forward/up are near zero. An unusable pose records zeros and makes the next usable one a cut.

  2. Motion flag. camera_moving is set if the position moved more than 0.01 units, or forward/up turned more than 0.003 rad. The budget uses it to admit motion fill.

  3. Cut detection. The constants are PANORAMA_CUT_DISTANCE = 2.0 world units and PANORAMA_CUT_RADIANS = 0.5235988 (30 deg). A frame is a cut when:

    • the position jumped more than 2 units and the jump departs from the previous frame's displacement by more than 2 units, or
    • forward or up turned more than 30 deg.

    The rationale: the stick turns at most about 2.3 deg a frame (d3d9.c adds 25 mouse counts per Present, 0.04 rad), and nothing the player drives covers two units (six metres) between frames. Cinematic sweeps can, so a large step is a cut only if it also departs from the previous motion. The Maw exterior flyby, which moves more than two units each frame, is the regression case.

On a cut, the host calls host_panorama_invalidate(). That advances scene_epoch, sets cut_pending, resets the frame and clears readiness. The whole sphere is redrawn, and a GPU sink cannot offer old-shot metadata during the first new pass. The first frame after the cut becomes cut_epoch. The presenter's layer alignment refuses to turn layers older than cut_epoch (EngineLayerAlignment.swift:284).

Epochs and lifecycle across level loads

Event Where Effect
First pass of a frame host_panorama_begin source_epoch++. A frame may begin on any layer, and a later layer 0 never restarts it.
Present with no panorama this frame (source_epoch == 0) host_panorama_present_complete host_panorama_invalidate: a flat/loading frame breaks continuity, so the next sphere refreshes every bearing.
Present after a panorama frame host_panorama_present_complete host_panorama_reset only. Held layers survive.
D3D Reset d3d9.c:393 host_panorama_invalidate.
Shell closes / >1 s gap in play hook Budget scene_entry (heavy tier, budget scene_epoch++) plus host_panorama_invalidate. The frame pacer resets on the budget's scene epoch.
Camera cut, unusable pose host_panorama_set_camera host_panorama_invalidate (or cut_pending).
Pass failure panorama_fail Frame invalid. Readiness cleared so every bearing refreshes before a later epoch can publish. The slot is released.
Render escape host_panorama_abort RENDER_ABORTED, readiness cleared.
Resolution change host_panorama_end Readiness cleared. On the CPU path, buffers reallocated.

host_panorama_reset clears only the per-frame state. host_panorama_invalidate also bumps scene_epoch and clears persistent readiness (panorama_render.inc:188-207).

Publication to the presenter

Present dispatch

At Present (d3d9.c:714-732):

  1. host_panorama_gpu_carry_missing().
  2. If a GPU slot is held and the info is valid: call metalwin_present_gpu(slot, w, h), then host_panorama_gpu_handoff() (the bridge now owns the slot), then host_panorama_present_complete().
  3. Else if a sink is active and the frame is WORLD_INCOMPLETE: call metalwin_present_dropped. The old fallback (a GPU readback plus a 20 MB flat copy) "only stalled the engine thread".
  4. Else, the CPU path: surface_flush the back buffer, then metalwin_present(bytes). The bridge copies the flat frame and, if the panorama is complete, all ten CPU layers.

The macOS probe has no zero-copy presenter. Weak no-op metalwin_present_gpu/metalwin_present_dropped definitions exist for it, and the visionOS runtime overrides them (panorama_render.inc:427-430).

The zero-copy pool

enginevision_start registers the sink {gpu_acquire, gpu_release, gpu_carry} when HALO_PANORAMA_GPU allows it. That is on by default on visionOS (unless "0") and off by default on the Mac (unless set and not "0"). The path is reachable on both on purpose: "while this path ran only on the headset, a lease that dropped three of the ten layers could not be caught anywhere cheap" (EngineVisionRuntime.m:697-710).

The pool has PANORAMA_GPU_SLOTS = 3 slots. Each holds HALO_PANORAMA_LAYERS BGRA8 private textures usable as shader-read and render-target (the render-target usage is needed for the FXAA pass). The block comment at EngineVisionRuntime.m:134-138 still says "four" textures. The code uses HALO_PANORAMA_LAYERS (10).

State Meaning
GPU_SLOT_FREE Available to gpu_acquire (and not gpu_latest, and no leases).
GPU_SLOT_RENDERING Lent to the engine for this frame.
GPU_SLOT_READY Published; gpu_latest points at it.
GPU_SLOT_RETIRING Superseded while the compositor still holds leases. Becomes FREE when the last lease is released.

Publication rules (EngineVisionRuntime.m:353-478):

  • Commit. metalwin_present_gpu reads host_panorama_frame(), assigns sequence = flat_sequence = ++latest_sequence, raises gpu_newest_scene_epoch, and copies all metadata. If the frame is incomplete, it releases the slot and returns without pacing or committing. Otherwise it calls host_frame_pacer_present() (which may block, with no lock held; see Frame Pacing), records the release time, and mr_commit_async(gpu_published, p).
  • Completion. gpu_published runs on Metal's completion thread. A failed command buffer, or a slot no longer RENDERING, counts as publish_failed. A completion whose flat_sequence is not newer than the last flat frame, whose scene_epoch is older than the newest seen, or whose sequence is not newer than the current latest, is superseded and retired. Completions can arrive out of order, and a pending world frame must never resurrect over a newer menu/loading frame. Otherwise the slot becomes READY, the previous latest is retired, and a publish-lag sample (completion time minus pacer release) is recorded for the pacer.
  • Dropped frame. metalwin_present_dropped updates bridge state as incomplete and counts dropped, without touching the last published slot.
  • Flat frame. The CPU path metalwin_present with status == FLAT sets gpu_flat_sequence, which supersedes any older world slot.
  • Carry. gpu_carry copies from gpu_latest only if that slot is READY, newer than the last flat frame, not from an older scene, and the same size. Copies ride the same command buffer as the engine's own blits, so ordering is program order, and a layer redrawn afterwards simply overwrites its copy.

Swift consumption

EnginePanoramaTexture.refresh(leasedTo:) runs once per compositor frame (EnginePanoramaTexture.swift:137-187):

  • Zero-copy. It calls enginevision_panorama_gpu_latest, which notes a display latch sample for the pacer and increments the slot's lease count. It unpacks the textures with EnginePanoramaLayerArray.pointers, then requires exactly HALO_PANORAMA_LAYERS textures at the published size and a valid Projections. Otherwise it releases the lease and reports status 2 / reason 5. The lease is released from the command buffer's completed handler.
  • Byte copy. It copies the ten layers from the bridge into its own three-slot pool and replaces texture bytes, with the slot protected by an in-flight count. For up to 22 compositor frames (about a quarter second at 90 Hz) it keeps the last complete snapshot across a plain frame, so a front-end-to-mission transition does not flash.

Projections.isValid (EnginePanoramaTexture.swift:37-49) checks every ring layer (0,1,2,7,8,9) and both caps (5,6):

  • the projection is finite and in (0,3);
  • the viewport is finite, within [0,1], and non-empty;
  • projection.x / viewport_width <= 1.7321 (tan 60 deg). That is, each band covers at least its 30 deg half-sector.

Layer 4 and the HUD are not checked here.

The snapshot exposes the textures through bgra8Unorm_srgb views. Sampling raw made every byte read as linear light: "a black of 8 became 28 ... the whole sphere read as washed grey". view(eye:sector:) returns layer 4 for the right eye's forward sector when stereo, and otherwise the same layer for both eyes. The HUD (layer 3) is premultiplied, and its shader decodes it.

The presenter (EngineImmersive.swift:617-645) does the following:

  • Builds source cameras in geometry order: ring layers [0,1,2,7,8,9], then the zenith and nadir.
  • Passes the caps' viewports as the full (0,0,1,1) rectangle.
  • Rebuilds all eight panel meshes whenever any projection or viewport changes, since each panel's fade depends on the panels beneath it.

EngineImmersiveScreenGeometry.panoramaAngles mirrors the host's bearing table (EngineImmersiveScreenGeometry.swift:163-171). The doc comment on PanoramaSource above it still describes "five cameras". The full presenter is documented on Immersive Presenter.

sequenceDiagram
    participant G as Engine thread (hook + D3D9)
    participant P as Producer (panorama_render.inc)
    participant B as Bridge (EngineVisionRuntime.m)
    participant M as Metal completion thread
    participant C as Compositor (EnginePanoramaTexture)
    G->>P: host_panorama_begin(first layer)
    P->>B: sink.acquire gives slot N (RENDERING)
    loop each scheduled bearing
        G->>P: begin(layer), original 0050BEA0, end(layer)
        P->>B: FXAA/blit back buffer into slot N layer (+HUD after layer 1)
    end
    G->>P: Present: host_panorama_gpu_carry_missing()
    P->>B: sink.carry(N, missing) copies from gpu_latest, returns its metadata
    G->>B: metalwin_present_gpu(N)
    B->>P: host_panorama_frame(info)
    B->>B: host_frame_pacer_present() (may wait)
    B->>M: mr_commit_async(gpu_published)
    G->>P: gpu_handoff(), present_complete()
    M->>B: gpu_published: N READY, gpu_latest=N, retire previous
    C->>B: enginevision_panorama_gpu_latest() (latch sample, lease++)
    B-->>C: 10 textures + EngineVisionPanoramaInfo
    C->>C: validate projections, draw 8 panels + HUD
    C->>B: command buffer completed: enginevision_panorama_gpu_release(N)
Loading

EnginePanoramaLayerArray

C void *textures[HALO_PANORAMA_LAYERS] and the per-layer float arrays import into Swift as fixed tuples with no count. The helper binds the tuple's memory to an array instead of naming members. An earlier hand-written unpack kept only seven members when the sphere grew from seven layers to ten. Every frame then failed the count check, "and the compositor was left with nothing to present" (EnginePanoramaLayerArray.swift:3-10).

Seam and stitch handling

Several mechanisms work together at the joins:

  • Overlap. DENSE bands are 64 deg wide for a 60 deg sector (2 deg guard each side). Caps are 96 deg wide, and the default 105 deg vertical field gives band/cap overlap.
  • Single sphere, ordered fades. The presenter draws every panel on one sphere of radius 4.25, ring first and caps last. Each panel's edge fade is applied only in proportion to what is already beneath it, with a feather of 0.15 of the half-width (EngineImmersiveScreenGeometry.swift:208-317).
  • Frame-coherent engine state. One render-frame counter and one time step per frame (see the pass loop above). Two bearings of one frame light and animate an object identically.
  • Bearing-invariant LOD. One pixel size per object per frame, so model LOD, shadow admission and lighting cadence agree on both sides of a join (Panorama Budget and LOD).
  • Radial fog (opt-in, HALO_RADIAL_FOG=1). Fog by distance rather than per-bearing depth (Radial Fog).
  • Seam-aware scheduling. The neighbour on the side of a visible join is refreshed more often (Panorama Budget and LOD).
  • Overlays in one place. The weapon is restricted to the forward three bearings. The HUD, menus and interface draw once into a flat layer.
  • Layer alignment (opt-in, HALO_LAYER_ALIGN=1). The presenter turns layers drawn on earlier frames by the rotation from their layer_pose to layer 1's pose. Layers older than cut_epoch are excluded (Layer Alignment).
  • FXAA on world layers before publication (metalrenderer.m:2713-2852).

Environment variables

HOST_ENV caches getenv per call site for the process lifetime (host.h:121-122). Changing one of these after first use has no effect. Values marked live are seeded from the environment into halo_settings and can then be changed from the in-app settings panel.

Variable Default Effect Read at
HALO_PANORAMA unset (headset worker sets 1) Must be "1" to enable the multi-pass panorama. panorama_hooks.inc:381
HALO_PANORAMA_DENSE unset (headset sets 1) Non-"0": narrow the projection rectangle (32 deg half for bands, 48 for caps). panorama_hooks.inc:267
HALO_PANORAMA_CAP_HALF_FOV 48 (caps only) Cap half-angle in degrees, accepted 20..80. Applies only where the default exceeds 40 (the caps). panorama_hooks.inc:281
HALO_PANORAMA_VFOV 105 (60..175) Vertical FOV of every bearing, in degrees. Live. halo_settings.c:56
HALO_PANORAMA_WORLD_FP unset (headset sets 1) "1": build first-person nodes once per view from the central pose; weapon drawn as world geometry, not HUD-routed. panorama_hooks.inc:396
HALO_PANORAMA_VIEWS unset "all": keep the game's record count in every pass (default: 1 except layer 1). panorama_hooks.inc:30
HALO_PANORAMA_OVERLAY_ALL_VIEWS unset "1": run weapon/HUD/UI/interface entries in every bearing. panorama_hooks.inc:334
HALO_PANORAMA_FRAME_PER_BEARING unset "1": old behaviour, advance the render frame and time step in every bearing. panorama_hooks.inc:527
HALO_LENS_FLARES unset "1": keep Halo's lens flares in panorama frames. panorama_hooks.inc:538
HALO_STEREO on "0": separation 0 (mono). Live via settings. halo_settings.c:46-50
HALO_STEREO_IPD 63 (40..90 mm) Interpupillary distance used to derive half-separation. halo_settings.c:47
HALO_STEREO_SEPARATION derived from IPD (0..1 world units) Half-separation directly. halo_settings.c:49
HALO_STEREO_ROLL on "0": eye offset along the level camera's right vector instead of the rolled head baseline. panorama_hooks.inc:43
HALO_SPATIAL_MENU on "0": render the front end flat instead of through the panorama. Live. halo_settings.c:61-62
HALO_FXAA on "0": publish world layers with a plain blit instead of FXAA (zero-copy path). panorama_render.inc:325
HALO_PANORAMA_GPU on (visionOS), off (Mac) Zero-copy sink. visionOS: "0" disables. Mac: any non-"0" value enables. EngineVisionRuntime.m:704-710
HALO_PANORAMA_TRACE unset Any value: bounded logs ([panorama-entry], [panorama-projection], [panorama-visibility], [panorama-interface]). panorama_hooks.inc:205, panorama_render.inc:170
HALO_PANORAMA_SCOPE_TRACE unset "1": log weapon/material scope boundaries (max 96 reports). panorama_hooks.inc:152
HALO_PANORAMA_FP_PROJECTION_TRACE unset "1": log first-person projection XY at material submit, upload, scope and restore. panorama_hooks.inc:121
HALO_PANORAMA_WORLD_FP_TRACE unset "1": log first-person cache build/reuse with bounded hashes. panorama_hooks.inc:100
HALO_VIEW_DUMP unset Any value: log the first 8 frustum rectangles/poses. panorama_hooks.inc:596
HALO_FRAME_CAPTURE + HALO_PANORAMA_CAPTURE unset Directory plus any value: dump all layers as .bgra every 60th valid frame (CPU path buffers). panorama_render.inc:390-392
HALO_LAYER_ALIGN off "1": presenter turns held layers to meet layer 1. Live. halo_settings.c:75-76
HALO_SIM_HEAD unset Swift: yaw,pitch in degrees replaces the head pose for simulator captures. EngineImmersive.swift:354-356
PANEL_DIVISIONS 64 Swift: grid divisions per panel edge. EngineImmersiveScreenGeometry.swift:260-261

Budget and LOD variables (HALO_PANORAMA_TARGET_FPS, HALO_PANORAMA_TIGHT_BUDGET, HALO_PANORAMA_ALL_VIEWS, HALO_PANORAMA_FORCE_TIER, HALO_PANORAMA_LOD) are listed on Panorama Budget and LOD. HALO_FRAME_PACING is on Frame Pacing.

Threading and ownership

  • The hook, producer state, budget, LOD and pacer controller all belong to the engine thread.
  • halo_settings fields are lock-free atomics. The head pose is written by the compositor thread and read by the hook once per frame, never between the two eyes (halo_settings.h:3-8).
  • The bridge pool is guarded by runtime_lock (os_unfair_lock). The pacer's wait happens with no lock held. gpu_published runs on Metal's completion thread, and enginevision_panorama_gpu_latest/release on the compositor thread.
  • The Swift byte-copy pool uses its own NSLock and per-slot in-flight counts.

Failure modes and recovery

Symptom Cause in code Recovery
Flat fallback instead of the sphere NO_PROJECTION: no builder or fixed-function projection in a pass Next frame redraws everything (readiness cleared).
Black / retained frame on headset Pool busy (ALLOCATION_FAILED), carry failure (READBACK_FAILED), Swift projection validation failure The compositor keeps the last complete published slot. The host forces a full refresh.
Old shot visible after a cut Prevented: cut invalidates, carry rejects scene mismatch Full sphere on the next frame.
World resurrected over a menu Prevented: flat_sequence / scene_epoch supersede checks in gpu_published and gpu_carry
Escape during a pass RENDER_ABORTED; partial frame never published CPU state preserved for normal handling.

Diagnostics counters panoramaPoolPublished, Dropped, PublishFailed, CarryFailed, PublishSuperseded, and the per-layer epochs, are reported by EngineDiagnostics.swift (EngineDiagnostics.swift:184-203). See Diagnostics and Telemetry.

Tests

The C tests are built and run by tools/run_source_checks.py (run_source_checks.py:49-51). See Testing and Source Checks.

Test What it asserts
test_panorama_epoch.c A frame may start on any layer (zenith first). A later layer 0 does not restart it. Held projections are carried. A flat Present resets the epoch.
test_panorama_lifecycle.c Per-layer sentinel pixels survive intact. A no-hook Present is FLAT and invalidates. A missing projection gives NO_PROJECTION. Resize reallocates and frees the old HUD page. Readback failure, viewport-origin normalisation and abort are handled. A new centre alone after a flat frame is not publishable.
test_panorama_gpu_carry.c Full frames need no carry. Partial frames request exactly the missing mask. Metadata comes from the published slot, not held arrays. One failed copy fails the frame and releases the slot. Scene change rejects old carries. A sink without carry cannot publish partial frames. Dropping to mono ignores the right eye.
test_panorama_pose_export.c Drawn layers take the frame camera. Held layers keep theirs. A 0.04 rad turn is not a cut. A 35 deg turn, a 3-unit jump, an unusable pose and a scene change are cuts. Stereo records layer 4. GPU-carried layers take the published pose. A frame with no camera still publishes with zero poses.
test_panorama_camera_cut.c After a cut there is no CPU carry of the old shot, and a GPU sink offering old metadata is rejected. A fast continuous flyby (6+ units per frame) keeps publishing in one scene. A discontinuity during the flyby still invalidates.
test_panorama_projection_hook.c DENSE narrows 640x480 to 173x480 for the builder only and restores the rectangle. Reports use the full raster and keep an off-centre origin. Reentry guard holds. Other callers and UI nesting are untouched. The interface record's builds report nothing.
test_panorama_letterbox_boundary.c Only 449B2A/449B9E are omitted, with an exact stack return. Other callers and calls outside the panorama run. The interface record is routed only in the HUD pass. EAX 1 is left to the original.
test_panorama_reentry.c Stub-boundary ABI test of the full eight-bearing mono frame: one camera record before the first pass, only layer 1 carries the time step, default overlay routing, central FP cache once per view, exact CPU/stack/record restoration, escape invalidation.
test_panorama_interface_record.c Runs the lifted 0050BEA0/0050BDC0/0050CC40/005175C0 under the hook. Only layer 1 gets the game's record count. The interface runs once, after the world, routed to the HUD, clearing nothing and reporting no projection. It keeps the game's camera. Prints SKIP without the generated sources.
test_panorama_fp_original_history.c With the lifted FP builder: three passes contaminate the history without the hook. With WORLD_FP the builder runs once, history is clean, and pose/view are restored. An escape is handled. Requires a generated build directory and is not in the default source-check list.
PanoramaPublicationValidation.m Includes the real EngineVisionRuntime.m. Out-of-order completions are superseded. A failed publish is excluded from lag samples. Carry returns exact per-layer epoch, projection and pose metadata. A flat frame blocks older world completions. Leases and retiring work. A scene change blocks stale slots. Incomplete frames neither pace nor commit. A paced commit publishes only on completion. Display-period sampling is checked.
PanoramaLeaseLayerCountValidation.swift The imported texture tuple has exactly HALO_PANORAMA_LAYERS entries and unpacks in order. Manual build command in the header.
PanoramaSphereCoverageValidation.swift Panels lie on one sphere. Caps are not mirrored. Composited alpha reaches opaque everywhere with no holes.
PanoramaViewportValidation.swift, PanoramaProductionGPUValidation.swift, PanoramaRasterProbe.m, PanoramaCompositePreview.swift Viewport-crop sampling, production shader rasterisation, HUD target reset and premultiplied coverage on Metal, and an offline composite renderer for captured layers. These are listed in SOURCE_MANIFEST.json but not invoked by run_source_checks.py. Several are written against the earlier three-sector layout.
PanoramaTextureValidation.swift Byte-copy pool coherence and failure retention. Its fixture defines a three-element / four-layer EngineVisionPanoramaInfo, which predates the current ten-layer struct. Whether it still builds against the current EnginePanoramaTexture.swift was not verified.

Related pages

Clone this wiki locally