Diagnostics and Telemetry

Diagnostics and Telemetry

The headset build writes a bounded, local evidence trail for every run, so a session can be analysed after the fact without the headset attached. It has five parts:

• report.json: a compact snapshot of the latest state.
• timeline-*.jsonl: one record per second.
• deep-*.jsonl and live.json: a 5 Hz health journal with fault events, plus its latest record.
• host.log: the engine's stdout and stderr.
• engine-frame-NNN.png: up to four PNG captures of engine frames.

The data comes from three layers. The C host keeps cumulative counters, including the core telemetry frame split in core_telemetry.h. The Objective-C bridges sample them (EngineVisionRuntime.m, EngineDiagnosticsBridge.m). Swift (EngineDiagnostics and friends) assembles and writes the records off the main thread.

On the Mac, tools/watch_engine_vision_telemetry.py polls live.json over the paired CoreDevice connection, and tools/engine_vision_report_summary.py summarises a pulled run directory. Everything stays in the app container until the user shares it.

Source files

File	Role
Sources/EngineDiagnostics.swift	Main-actor collector: run directory, samples, timeline/deep/report records, frame captures, flush
Sources/EngineDiagnosticReportWriter.swift	Serial background writer for report.json, timeline, deep history and live.json; report cadence
Sources/EngineDiagnosticHistory.swift	Bounded, append-only JSONL segments
Sources/EngineDeepTelemetry.swift	Deltas, rates, layer ages and fault begin/end events for the deep journal
Sources/EngineCoreTelemetry.swift	Serialises EngineVisionCoreTelemetry into three JSON groups
Sources/EngineGatherDiagnostics.swift	Native BSP-gather switch and counters as JSON
Sources/EngineDiagnosticsBridge.h / .m	EngineDiagnosticSample (memory, controller, audio, voices, cache I/O, draw fast path, gather) and enginevision_capture_host_log
Sources/EngineVisionRuntime.m	engine_thread_sample, enginevision_draw_profile, enginevision_core_telemetry, enginevision_panorama_gpu_stats, enginevision_frame_digest
native/EngineHost/core_telemetry.h	Present-to-Present frame split arithmetic, Halo tick counter read, lock-free publication
native/EngineHost/gather_diagnostics.h	HostGatherDiagnostics
native/EngineHost/host_snapshot.h	Test-only guest memory capture (HALO_CAPTURE_PC, HALO_CAPTURE_DIR)
EngineImmersive.swift, EngineImmersiveTrace.swift, EngineWorldCadence.swift, EngineLayerAlignment.swift	Sources of the immersive, lifecycle, cadence and alignment fields (see Immersive Presenter)
tools/watch_engine_vision_telemetry.py	Mac-side live watcher
tools/engine_vision_report_summary.py	Run summary
tools/check_core_telemetry_xros.py	Links the telemetry kernel APIs for xrOS 26.0
tools/check_probe_runner_exit.py	Exercises the desktop probe runner's receipt and exit-code tail
tools/test_watch_engine_vision_telemetry.py, tools/test_engine_vision_report_summary.py	Python unit tests
native/EngineVision/Tests/*Telemetry*, Diagnostic*, GatherDiagnosticsValidation.swift, *ProbeTests.m, MenuInputProbe.m	See Testing

Where files are written

Every run gets its own directory, named by a fresh UUID (EngineDiagnostics.runID). It lives inside the app's Documents folder, which is visible in Files because UIFileSharingEnabled is set:

Documents/Diagnostics/<runID>/
  report.json             latest state, compact JSON, schema 2
  timeline-000000.jsonl   one-second records (schema 1), 8 MiB segments, at most 16 kept
  deep-000000.jsonl       ~5 Hz deep records (schema 1), 2 MiB segments, at most 16 kept
  live.json               latest deep record plus history bookkeeping (atomic replace)
  host.log                stdout + stderr of the process (append, mode 0600)
  engine-frame-001.png    up to 4 captures of non-black engine frames, at least 10 s apart

Artefact	Bound	Source
Timeline	16 x 8 MiB = 128 MiB; oldest segment deleted, historyTruncated = true	EngineDiagnosticHistory.swift:19
Deep history	16 x 2 MiB = 32 MiB	EngineDiagnosticReportWriter.swift:43-44
Deep write queue	at most 8 pending records; extras counted in deepDroppedRecords / writerDroppedRecords	L69-L76
Frame PNGs	4 per run	EngineDiagnostics.swift:160-162
Immersive lifecycle trace	last 128 events, embedded in the report	EngineImmersiveTrace.swift:26
World cadence intervals	4096	EngineWorldCadence.swift:13
host.log	not bounded by the app	EngineDiagnosticsBridge.m:107-114

The timeline segments hold more than 30 minutes of one-second records (DiagnosticHistoryValidation writes 2100). Old run directories are never deleted by the app.

Data flow and cadence

flowchart TB
    T["EngineRuntimeModel timer, 0.2 s, main actor"] --> S["EngineDiagnostics.sample(state, status, frame)"]
    S --> DS["enginevision_diagnostic_sample()"]
    S --> D{"at least 0.18 s since last deep record?"}
    D -- yes --> RD["recordDeep: EngineDeepTelemetry.sample"]
    RD --> AD["writer.appendDeep (queue cap 8)"]
    AD --> DH["deep-NNNNNN.jsonl"]
    AD --> LV["live.json if 1 s elapsed or events present"]
    S --> W{"at least 1 s since last timeline record?"}
    W -- yes --> TR["timeline record: profile, pool, pacing, core telemetry, voices, host work ..."]
    TR --> AT["writer.appendTimeline"]
    AT --> TL["timeline-NNNNNN.jsonl"]
    TR --> RQ{"reportDue: first, state change, or interval (30 s)"}
    RQ -- yes --> WR["writer.writeReport"]
    WR --> RJ["report.json"]
    F["scenePhase change / willTerminate"] --> FL["flush(reason, wait)"]
    FL --> WR
    subgraph Q["Serial queue halo.diagnostics.writer (QoS utility)"]
        DH
        LV
        TL
        RJ
    end

Loading

• Sampling is driven by EngineRuntimeModel.refresh() every 0.2 s on the main actor (EngineRuntimeModel.swift:34-47). The one-second gate therefore produces records roughly every 1.0-1.2 s, and intervalSeconds records the actual spacing.
• Report cadence (EngineDiagnosticReportWriter.swift:49-61): the first sample, any change of engine state (so a failure or stop does not wait half a minute), or interval seconds. The interval is 30 s by default, or HALO_REPORT_SECONDS clamped to 1...600; non-numeric values fall back to 30.
• Flushes happen on every scenePhase change, waiting for disk when going to .background, and on UIApplication.willTerminateNotification, always waiting (EngineDiagnostics.swift:69-86). flush takes a fresh EngineDiagnosticSample but reuses the last state, status and frame.
• Ordering: all writes go through one serial queue in hand-over order, and only that queue touches the timeline. A report therefore counts exactly the timeline records handed over before it (L98-L138). drain() is queue.sync {}.

Why the writer exists

The comment at EngineDiagnosticReportWriter.swift:3-15 gives the history. report.json used to be rewritten every second on the main actor, pretty-printed and carrying every sample since launch: 1.0-1.25 MB and 25-39 ms per write four minutes in, and 177 ms at the 30-minute cap, on the thread SwiftUI runs on. Since Build76 the one-second samples live only in the timeline, and the report is a few kilobytes of compact (.sortedKeys, no pretty printing) JSON. The report records its own cost (reportWriteMilliseconds, reportWriteMillisecondsMax, reportWrites).

Failure handling

Storage errors never stop the engine:

• EngineDiagnosticHistory makes its first failure sticky (failure) and stops appending. It refuses to open a segment file that already exists, so a reused directory cannot silently overwrite an earlier run. A single record larger than a segment is an error.
• The writer records the last failure in stats.failure. EngineDiagnostics shows it as "Report save failed: ..." in the setup window.
• historyError and deepHistoryError are written into the report and live.json.

report.json (schema 2)

Built by report(state:status:frame:sample:now:) (EngineDiagnostics.swift:355-480), merged with hostWork, voiceFields and coreTelemetry. The writer then adds its own bookkeeping.

Group	Keys
Identity	schema (2), runID, startedAt, updatedAt, bundleID, build (CFBundleVersion), buildID (HaloBuildID or unknown), bundlePath, bundledExecutablePath, os
Engine	state, status, frameSequence, width, height
Panorama publication	panoramaAvailable, panoramaSequence, panoramaSourceEpoch, panoramaFlatSequence, panoramaStatus, panoramaFailureReason, panoramaViewportUMin/VMin/UMax/VMax and panoramaProjectionX/Y (10 entries each), panoramaViewCount, panoramaZeroCopy, radialFogEnabled
Zero-copy pool	panoramaPoolPublished, panoramaPoolDropped, panoramaPoolPublishFailed, panoramaPublishSuperseded, panoramaPoolCarryCopies, panoramaPoolCarryFailed, panoramaPoolLatestSlot, panoramaPoolSlotStates, panoramaPoolSlotLeases, panoramaPoolLastError
Draw profile	enginePasses, enginePassSeconds, engineReadbackSeconds, engineReadbackCalls, engineDraws, engineVertexSeconds, engineSubmitSeconds, engineUploadSeconds, engineSleepSeconds, engineSleepCalls, engineYieldCalls, engineThreadCPUSeconds, engineThreadFramesByCPU (16), performanceCores, efficiencyCores, engineProgram* (compiles, compile seconds, waits, wait seconds, background pipelines/functions/seconds, prewarm hits, archive hits, function compiles), hapticsOnsets, panoramaBudgetExtraHalf, panoramaBudgetTier, panoramaBusyMilliseconds, pointerFrames, pointerCursorX/Y, audioWatchdogRebuilds
Presentation	panoramaLayerAlign (Layer Alignment), framePacing (below), immersiveStatus, immersiveActive, immersiveLayerState, immersiveLifecycle (trace), immersiveSubmittedFrames, immersiveCancelledFrames, immersiveTrackingLossFrames, immersiveGPUCompletedFrames, immersiveGPUFailedFrames, immersiveGPUError, immersiveGPUMilliseconds, immersiveGPUMillisecondsMax, immersiveConfiguration, immersiveSubmission
Audio	audioFrames, audioNonzeroSamples, audioPeak, audioQueueRunning, audioQueueStatus, audioOutputVolume, audioQueueGeneration, audioQueueSuspended, audioQueueRestartRequired, audioQueueDeferredBuffers, audioLast{Prepare,Start,Enqueue,Pause,Dispose,Recovery} plus the voice fields below
Memory	guestHeap (liveBytes, peakLiveBytes, cumulativeRequestedBytes, liveAllocations, metadataBytes, addressHighWaterBytes), peakResidentBytes, peakPhysicalFootprintBytes, peakMetalCurrentAllocatedBytes, physicalFootprintAvailable, metalCurrentAllocatedAvailable
Input	buttonsSeenMask, axisMin, axisMax, controllerConnectedEver, originalEngineGamepadReads, originalEngineKeyboardEvents, controllersConnected, controllerNames (vendor names), controllerHaptics, controllerLink
Haptics	hapticsState, hapticsEventsTaken, hapticsEventsPlayed
Preparation	preparationStatus, preparationCopiedBytes, preparationTotalBytes, hostLogCaptureError
Misc	thermalState, nativeGather, capturedEngineFrames, deepTelemetry (latest deep record)
Pointers and scope strings	liveSnapshotLocation (live.json), deepSamplesLocation (deep-*.jsonl), samplesLocation (timeline-*.jsonl), deepTargetIntervalSeconds (0.2), historyScope, scope
Added by writer	reportReason (first, state, interval, or the flush reason), reportIntervalSeconds, historyFiles, historyRecords, historyTruncated, historyError, deepHistoryFiles, deepHistoryRecords, deepHistoryTruncated, deepHistoryError, reportWrites, deepDroppedRecords, reportWriteMilliseconds, reportWriteMillisecondsMax
Host work (merged)	controllerCaptures, controllerReadingsReused, controllerReuseMicroseconds, controllerLink, dinputAcquireInputLost, engineCacheWaits, engineCacheWaitSeconds, engineCacheWaitBlocks, cacheReaderReads, cacheReaderReadSeconds, guestThreadQoS (name), guestThreads, hostDrawFastpathEnabled, hostDrawResidentVertexBytes, hostDrawArenaVertexBytes, hostDrawFoldedClears
Core telemetry (merged, when on)	engineFrameSplit, engineThreadScheduling, processCores

The report states its own limits in scope: "Native startup and hardware test; not proof of a playable campaign. Pixel samples measure engine output, not headset display quality."

immersiveSubmission holds mode, sequence, sourceEpoch, producerEpoch, sceneEpoch, layerEpochs, worldCadence, sourceFlatSequence, latestFlatSequence, poolSlot, sourceStatus, failureReason, eyeProjections, neutralFromEyes and neutralOriginFromHead (L341-L351). The meaning of mode and worldCadence is on Immersive Presenter.

Timeline records (schema 1)

Built in sample(...) (EngineDiagnostics.swift:176-243). Since Build76 the timeline is the only copy of the one-second samples, so it carries every field the report's old samples array did:

• schema, runID, build, buildID, elapsedSeconds, intervalSeconds, state, engineExitCode, menuActive, frameSequence, enginePresentsPerSecond (frame delta / interval);
• immersiveActive, mode, immersiveSubmittedFrames, immersiveGPUCompletedFrames, immersiveGPUFailedFrames, immersiveGPUMilliseconds;
• panoramaEpoch, panoramaProducerEpoch, panoramaSceneEpoch, panoramaLayerEpochs, panoramaLayerAlign, worldSubmissionCadence, immersiveSubmission;
• guestHeap, framePacing, nativeGather;
• the pool counters, panoramaBudgetTier, panoramaBudgetExtraHalf, panoramaBusyMilliseconds, radialFogEnabled;
• the engine pass, readback, vertex, submit, upload, sleep and yield times and counts, engineThreadCPUSeconds, thermalState, the engineProgram* counters;
• the audio queue fields, hapticsEventsTaken/Played, controllerConnected, gamepadReads, keyboardEvents;
• residentBytes, virtualBytes, physicalFootprintBytes, metalCurrentAllocatedBytes with their availability flags;
• the frame digest: frameSampleCount, nonblackPixelSamples, differingPixelSamples, pixelSampleHash;
• merged voiceFields, hostWork, coreTelemetry.

Counters are cumulative. The difference between two records gives the interval's value, and the summary tool works that way. The historyScope string warns that the timeline is "not per-frame timing or mission-completion proof".

Voice fields

From voiceFields (L497-L522):

• DirectSound activity: audioCallbacks, audioLateCallbacks, audioEnqueueFailures, audioCallbackAgeMilliseconds, audioMaxCallbackGapMilliseconds, audioMaxCallbackWorkMilliseconds, audioSampleRate, audioBufferFrames, audioVoicePlays/Stops, audioStatusProbes/Busy, audioCallsRejected, audioBuffers, audioVoicesPlaying/Peak.
• The original engine's own sound tables, read from guest memory (−1 until readable): soundEngineFlags, soundSources, soundLoopingSounds, soundChannels/Busy/Starved, soundVoices/Assigned/Free/Held, the per-type arrays, soundCache{Sounds,Loaded,Locked}, soundReadsQueued (level map, bitmaps, sounds).
• Cache-file reads by file: cacheFileReads, cacheFileReadBytes (level map, bitmaps.map, sounds.map, other).

The comment explains the intended reading: silence with sources and held voices but none free is voice exhaustion; silence with no sources means nothing asked for sound. See Audio System.

Frame pacing group

framePacingReport() (L111-L130) serialises host_frame_pacer_report:

mode, rung, cadenceHz, latchLocked, displayPeriodMilliseconds, observedDisplayPeriodMilliseconds, targetPeriodMilliseconds, leadMilliseconds, publishLagMilliseconds, workMilliseconds, floorWorkMilliseconds, budgetTargetMilliseconds, lateShare, frames, pacedFrames, lateFrames, freeFrames, rungChanges, framesByRung, waitSeconds, requestedWaitSeconds.

Evenness should be judged against worldSubmissionCadence. The pacer itself is on Frame Pacing.

Deep telemetry and live.json

recordDeep (L250-L301) builds a smaller record at most every 0.18 s. Its inputs are:

• identity: runID, build, buildID, updatedAtUnixSeconds, elapsedSeconds;
• state: state, menuActive, gameplay, mode, frameSequence;
• presentation: submittedFrames, gpuFailures, gpuMilliseconds(Max), sceneEpoch, layerEpochs, layerAlignment, worldCadence;
• budget: budgetTier, extraHalf, busyMilliseconds, enginePasses, enginePassNanoseconds, shaderWaits, shaderWaitNanoseconds;
• audio: callback and voice health;
• controller: link counters, controllerSequence, buttonMask, controllerAxes;
• haptics state and counters, thermalState, residentBytes, physicalFootprintBytes, soundReadsQueued.

gameplay is true when the state is RUNNING, the space is immersive, no menu is up, and the mode is panorama.

EngineDeepTelemetry.sample (EngineDeepTelemetry.swift:15-92) adds derived fields. All deltas are clamped at zero, so a counter reset cannot underflow.

Field	Meaning
intervalSeconds	Measured time since the previous deep record, including observer stalls
engineFPS, submittedFPS, audioFramesPerSecond	Rates over that interval
audioNonzeroSamplesDelta, gamepadReadsDelta, hapticsPlayedDelta, hapticsTakenDelta, <counter>Delta	Deltas
layerObservedAgeMilliseconds	Per layer, time since its epoch was last seen to change (a lower bound, not a render timestamp); −1 = never published. Reset on scene change.
layerEpochLag	Newest epoch minus each layer's epoch (null for 0)
missingLayerCount	Layers with epoch 0
events	Discrete events this record (below)
activeSignals	Faults currently active
eventCounts	Running counts per event
schema (1), recordIndex, scope

Counter events are emitted when the corresponding counter increases:

Counter	Event
controllerDisconnects	controller.disconnect
controllerConnects	controller.connect
audioGeneration	audio.queueRebuilt
audioEnqueueFailures	audio.enqueueFailure
audioLateCallbacks	audio.callbackGap
gpuFailures	gpu.failure
hapticFailures	haptics.failure
hapticResets	haptics.reset
hapticStops	haptics.engineStopped

A scene change emits panorama.sceneChanged.

Faults (each emits <fault>.begin and <fault>.end on transitions):

Fault	Condition
observer.delayed	interval > 0.6 s
engine.noProgress	gameplay and no new engine frame
presenter.noProgress	gameplay and no new submitted frame
audio.noCallbackProgress	gameplay, audio not suspended, voices playing, and no audio frames
audio.zeroMixWithVoices	as above, frames advancing but no nonzero samples
panorama.staleLayer	gameplay and any layer observed older than 250 ms
panorama.missingLayer	gameplay and any layer epoch 0

The record's scope states the limits: "stale-layer and silence signals are not proof of visible seams or audible failure. Controller polls are not Bluetooth packets."

The writer appends every deep record to deep-*.jsonl. It overwrites live.json atomically when at least one second has passed since the last write, or immediately when the record has events, adding writerDroppedRecords, deepHistoryFiles, deepHistoryTruncated and deepHistoryError (EngineDiagnosticReportWriter.swift:72-96).

Diagnostic sample bridge

EngineDiagnosticSample (EngineDiagnosticsBridge.h:5-59) is filled by enginevision_diagnostic_sample (EngineDiagnosticsBridge.m:33-106). New fields are appended for ABI compatibility. Fields that may legitimately be zero carry availability flags.

Fields	Source
resident_bytes, virtual_bytes	task_info(MACH_TASK_BASIC_INFO)
physical_footprint_bytes + ENGINE_DIAGNOSTIC_HAS_PHYSICAL_FOOTPRINT	task_info(TASK_VM_INFO).phys_footprint
metal_current_allocated_bytes + ENGINE_DIAGNOSTIC_HAS_METAL_ALLOCATED_SIZE	MTLDevice.currentAllocatedSize of the system default device
controller_connected, controller_sequence, button_mask, axes[6]	hostgc_poll
gamepad_reads, keyboard_events, dinput_acquire_lost	DirectInput host counters
controller_captures, controller_reuses, controller_reuse_us	hostgc_poll_stats
cache_wait_*, cache_reads, cache_read_ns, cache_wait_blocks	Halo cache-read wait accounting
guest_thread_qos, guest_threads	Guest threads' applied QoS and count
guest_heap[6]	host_heap_stats
audio_*	DirectSound stats, output diagnostics, output state, voice stats; audio_output_volume from AVAudioSession on visionOS (−1 elsewhere)
sound_*	The engine's own sound tables via host_dsound_get_voice_stats
cache_file_reads[4], cache_file_read_bytes[4]	host_async_reads, host_async_read_bytes
draw_fastpath_enabled, draw_resident_vertex_bytes, draw_arena_vertex_bytes, draw_folded_clears	mr_fast_paths_enabled, mr_draw_traffic_stats (Geometry Fast Paths)
native_gather_*	host_gather_get_diagnostics

enginevision_capture_host_log(path) opens host.log (O_WRONLY|O_CREAT|O_APPEND, mode 0600) and dup2s it over both STDERR_FILENO and STDOUT_FILENO, with stderr unbuffered and stdout line-buffered. From then on, everything the host prints to those streams goes to the run directory. That includes [device-config] lines and the [audio-route] and [audio-session] lines. A failure is reported as hostLogCaptureError and in the setup window's save status.

Native gather

gather_diagnostics.h defines HostGatherDiagnostics { calls, native_calls, fallback_calls, enabled }. The counts are cumulative over the process lifetime and cover only enabled dispatch entries; calls includes native and translated-fallback entries, and internal recursion is not counted separately. enabled reflects the effective HALO_NATIVE_GATHER switch after the first intercepted gather. EngineGatherDiagnostics.fields emits {enabled, calls, nativeCalls, fallbackCalls} into both the report and the timeline, so interval differences show whether the optimisation was active during a measured workload.

Draw profile

enginevision_draw_profile (EngineVisionRuntime.m:251-288) collects these fields. All are cumulative for the run.

Fields	Source / meaning
passes, pass_ns, readback_ns, readback_calls, draws, vertex_ns, submit_ns, upload_ns, sleep_calls, sleep_ns, yield_calls	host_draw_profile_snapshot (12 monotonic counters, plain reads, no lock)
haptic_onsets	Controller pulses fired from voice onsets
panorama_extra_half, panorama_tier, panorama_busy_seconds	Bearing budget (Panorama Budget and LOD)
pointer_frames, pointer_x/y	Gaze-pointer servo frames and the engine cursor (640x480 interface pixels)
audio_rebuilds	DirectSound watchdog rebuilds
engine_cpu_ns	thread_info(THREAD_BASIC_INFO) on the engine worker's port
engine_cpu_frames[16]	Per present, which CPU number the engine thread was on (pthread_cpu_number_np, masked to 16)
performance_cores, efficiency_cores	sysctl hw.perflevel0/1.logicalcpu (cached)
program_compiles, program_compile_ns	mr_program_compile_stats
program_waits, program_wait_ns, program_background_pipelines/functions/ns, program_prewarm_hits, program_archive_hits, program_engine_functions	mr_pipeline_stats (Metal Renderer)

The header comments explain why these exist. The headset ran the engine at about four frames a second in play while the Mac ran it at sixteen, and the engine thread ran ticks and passes about 3.4x slower on the headset. These counters measure where that time went and which kind of core the thread got.

Core telemetry

Core telemetry measures, per engine frame (Present to Present), where the engine thread's wall time goes and whether it was on a core. It is on by default (the device defaults set HALO_CORE_TELEMETRY=1). HALO_CORE_TELEMETRY=0 turns it off and "restores the pre-telemetry Present, wait and report paths". The decision is made once in engine_worker, before the engine runs, and stored in host_core_telemetry (EngineVisionRuntime.m:656-659; the variable is defined at threading.c:368, where it also gates timed waits). Several comments cite a CORE_TELEMETRY.md document, but it is not part of this source tree.

Per-Present sample (engine thread)

engine_thread_sample() (EngineVisionRuntime.m:67-88) runs at the start of every metalwin_present*:

1. It always bumps engine_cpu_frames[cpu & 15], including when telemetry is off.
2. If telemetry is on, it fills a HaloFrameSample:

Field	Source
now_ns	clock_gettime_nsec_np(CLOCK_UPTIME_RAW)
pass_ns	host_pass_profile_total_ns() (inside bearing passes)
idle_ns	host_yield_spin_ns (frame limiter idle: Sleep(n) and the Sleep(0) spin)
sleep_ns	host_present_sleep_ns (blocking sleep and cache waits)
wait_ns, waits	host_thread_wait_ns, host_thread_waits (non-polling waits, critical-section contention)
ticks	halo_game_time_read
cpu_ns, counts_valid	thread_info(THREAD_EXTENDED_INFO) user + system

3. It then calls halo_frame_split_add. errno is saved and restored, so the sample cannot disturb the guest; the publication test asserts this.

Frame split arithmetic (core_telemetry.h)

halo_frame_split_add (core_telemetry.h:107-158) takes the period between this Present and the previous one:

• Stall frames (period > HALO_FRAME_STALL_NS = 1 s; loads and stalls) are counted apart: stall_frames, stall_ns, stall_ticks, stall_outside_pass_ns, stall_tick_unknown_frames. They are kept out of the steady-frame means.
• Normal frames add:
  • wall_ns (period), pass_ns, idle_ns, wait_ns, sleep_ns, waits;
  • outside_pass_ns = period - pass (clamped);
  • other_ns = outside - idle - wait (each step clamped). This is a residual estimate, not an exclusive partition, because idle and wait can overlap passes.
• CPU (both samples valid): cpu_ns, off_core_ns = period - cpu, and unaccounted_ns = off_core - sleep - wait. The last is time off a core not explained by timed sleeps or waits: waiting for a core, or blocked somewhere unmeasured.
• Ticks: halo_frame_ticks returns the game-time counter delta unless the interval is unknown. It is unknown if either sample lacks the globals, the globals pointer changed, the counter went backwards, or it jumped by more than HALO_TICK_JUMP (120). Known frames are bucketed by ticks 0, 1, 2, 3, 4+ (tick_frames[5], tick_other_ns[5]). The function also tracks max_ticks and counts tick_mismatch_frames when the delta differs from the driver's possibly stale last-call count. Unknown frames go to tick_unknown_frames, and resyncs to tick_resyncs.
• presents counts every call.

The game-time counter. halo_game_time_read (L83-L96) reads the pointer at guest 0x006F1D6C (game_time_globals), then the dword at +0x0C (tick counter) and the word at +0x10 (the driver's last-call tick count). The header records that this was verified against the generated Build75 sub_00470BF0.c. Pointers below 0x10000 (the host's null guard) or above 0xFFFFFF00 are treated as unavailable. Reading the counter introduces no intercepted address and no guest write.

Publication. The totals are copied into _Atomic uint64_t published[] words between two increments of a sequence counter (odd while writing), with release fences. halo_frame_split_read retries up to 1000 times and returns 0 if it never sees a stable even sequence. The header notes that a plain memcpy under a sequence lock would still be a C data race (L70-L79, L160-L177).

Collection for reports

enginevision_core_telemetry(out) (EngineVisionRuntime.m:289-348) zeroes out. If telemetry is off, it returns false without any kernel query, and the report then omits the groups. Otherwise it fills:

• The frame split snapshot, with snapshot_available = the read result.
• thread_info(engine_worker_port, THREAD_EXTENDED_INFO): run state, flags, policy, current/base/max priority, pth_cpu_usage (decaying, scale 1000), run ns. Also thread_info_result and thread_info_available. With no worker port it reports KERN_INVALID_ARGUMENT.
• proc_pid_rusage(getpid(), RUSAGE_INFO_V6), declared weak because the xrOS SDK omits libproc.h; a missing symbol reports ENOSYS. Mach-unit times (CPU, P-core CPU, runnable, seven QoS buckets) are converted with mach_timebase_info; a zero numerator or denominator counts as KERN_FAILURE. Cycles, instructions, P-cycles, P-instructions, energy, P-energy, disk reads, page-ins and interrupt/idle wakeups are copied directly.
• task_info(TASK_POWER_INFO_V2).task_pset_switches (cluster switches).

EngineCoreTelemetry.fields (EngineCoreTelemetry.swift:9-54) emits the same three groups into both the report and each timeline record:

Group	Keys
engineFrameSplit	snapshotAvailable, frames, wallSeconds, passSeconds, outsidePassSeconds, idleSeconds, waitSeconds, waits, sleepSeconds, otherSeconds, countsAvailable, countsFrames, cpuSeconds, offCoreSeconds, unaccountedOffCoreSeconds, ticks, tickFrames[5], tickOtherSeconds[5], tickResyncs, tickMismatchFrames, tickUnknownFrames, maxTicksPerFrame, stallFrames, stallSeconds, stallOutsidePassSeconds, stallTicks, stallTickUnknownFrames, presents
engineThreadScheduling	available, infoResult, policy, priority, basePriority, maxPriority, runState, flags, cpuUsagePercent (= usage / 10), runSeconds
processCores	available, rusageResult, timebaseResult, cpuSeconds, performanceSeconds, runnableSeconds, cycles, instructions, performanceCycles, performanceInstructions, qosSeconds {default, maintenance, background, utility, legacy, userInitiated, userInteractive}, energyJoules, performanceEnergyJoules, diskReadBytes, pageins, interruptWakeups, idleWakeups, clusterSwitchesAvailable, clusterSwitches

The header comment states the reading rules. Availability is explicit, so zero is not evidence of idle. Delta cycles divided by delta CPU time gives effective GHz while running, not an instantaneous frequency. Runnable time includes running time, so runnable minus CPU is time ready with no core. Process figures cover all threads.

Test-only guest capture (host_snapshot.h)

host_capture_for_test(cpu) (host_snapshot.h:4-36) runs only when both HALO_CAPTURE_PC (hex guest PC) and HALO_CAPTURE_DIR are set, and only once. It uses mincore to find resident pages of the 4 GiB guest space above the 64 KiB null guard. It writes them to memory.bin (capped at 512 MiB) and writes state.json, which holds the PC, flags, GPRs, x87 registers in stack order, the FPU control/status/top/valid words, region offsets, complete and testOnly: true. The header describes it as used for original-instruction differential tests and never loaded by the production game. See EngineReuse Runtime and x87 Floating Point.

Mac-side tools

watch_engine_vision_telemetry.py

The module docstring describes it this way: it reads a selected build's live.json over the paired CoreDevice connection. It opens no listener, needs no new app permission, injects no input and changes no Wi-Fi settings. It never accepts another build's evidence, and a failed or stale pull is displayed as such, never as live data.

python3 tools/watch_engine_vision_telemetry.py \
  --device <device> --bundle-id <bundle id> --build <CFBundleVersion> --build-id <HaloBuildID> \
  [--seconds 900] [--interval 5] [--output native/logs/live]

Option	Default	Constraint
--device	required	passed to xcrun devicectl ... --device
--bundle-id	required	--domain-type appDataContainer --domain-identifier <id>
--build	required	must equal the record's build
--build-id	required	must equal the record's buildID
--seconds	900	0...43200
--interval	5	2...60
--output	native/logs/live	created if needed

Behaviour (L65-L153):

1. Takes an exclusive flock on <output>/watcher.lock, so only one watcher can use an output directory, and writes its PID there.

2. At least every 30 s, or whenever no run is selected, lists Documents/Diagnostics with devicectl device info files. live_candidates keeps only paths of the exact form <UUID>/live.json (rejecting .., absolute paths and non-UUID directories) and sorts them newest first by lastModDate. At most five are tried.

3. Copies each candidate to incoming.json with devicectl device copy from. Each devicectl call has --timeout 12 and a 17 s subprocess timeout. It accepts the first record where buildID and build match and runID is present.

4. Writes status.json (atomically) with expectedBuild, expectedBuildID, polledAtUnixSeconds, pid, and connection:

   • unavailable-or-no-matching-build;
   • live, when the snapshot is younger than 15 s by wall clock and (runID, recordIndex) advanced within the last 15 s by the monotonic clock. The two checks guard against both an old run on the first pull and clock skew between the devices;
   • stale otherwise.

   With a record it also includes snapshotAgeSeconds and a summary: build, budget tier, extra half, run ID, record index, elapsed, mode, engine and submitted fps, maxObservedLayerAgeMs, audio frames/s, late callbacks, controller connected and disconnects, haptics played and failures, active signals, event counts, writer drops. The full record goes to latest-snapshot.json.

5. Appends each status to monitor.jsonl, rotating it to monitor-previous.jsonl above 8 MiB, and prints it as one JSON line.

6. Each devicectl call leaves list.json/list.log or copy.json/copy.log in the output directory.

7. Exits 0 if at least one good record was read, otherwise 1.

accepted() defaults build to '80' for direct callers; the CLI always passes --build. The full deep-*.jsonl evidence stays on the headset.

engine_vision_report_summary.py

python3 tools/engine_vision_report_summary.py <run dir> [more run dirs...]

The run directory is a pulled Documents/Diagnostics/<runID>; the docstring's example path is native/logs/engine-vision-reports/<stamp>/Diagnostics/<runID>. For each directory the tool reads every timeline-*.jsonl in name order and report.json if present (main, L231-L330). A play interval is a pair of consecutive records where the later one has mode == "panorama", no menu, and intervalSeconds > 0.5.

It prints:

Line	Computation
Header	Run directory name, build, sample count, play intervals
audio silent in play	Seconds and stretches where audioFrames advanced but audioNonzeroSamples did not, between two play records; the longest stretch. With Build76+ voice counters: median voice starts/s in play vs in silence; ranges of sources, busy channels, free and held voices, cache sounds, queued sound reads during silence; seconds with voices held and none free (voice exhaustion); sounds.map reads/s.
fps	Median, p10, p90 of frame delta / dt
passes/frame, ms/pass, pass share of wall, busy ms (EMA)	Same statistics
tier seconds	Per budget tier, 1.2 s * samples (assumes 1.2 s per record)
engine thread CPU/wall	From engineThreadCPUSeconds
pad captures/frame, pad reuses/frame, cache waits/frame, cache wait ms/frame, cache reads/frame, cache read ms/frame	Build76+ host work per engine frame
host switches	Guest thread QoS, pad reuse window, cache waits block or spin, report write cost
thermal state seconds
layer alignment	Final state, frames by level, frames with a cut. Then an on/off split by each record's enabled, with per-layer mean degrees misaligned → turned, and run maxima. See Layer Alignment.
cores / engine frames by CPU	P/E counts, share of frames per CPU number
pipelines	Engine-built pipelines and time, waits, background builds, prewarm and archive hits, and intervals with more than 50 ms of pipeline stall
Core telemetry block	Per play interval (core_summary, L135-L227): median, p10, p90 of frame ms, passes ms/frame, outside passes, limiter idle, guest waits, residual, ticks per known frame, engine CPU/wall, unaccounted off-core ms, process queued cores (max(runnable - cpu, 0) / wall), process P share, effective GHz (all and P), IPC, P/E switches/s. Then a frame-weighted linear fit of residual ms against ticks per frame (tick_fit), explicitly an association, not isolated tick cost. Then whole-run stalls, tick resyncs/unknown/mismatch, max ticks, process CPU by applied QoS, and the engine thread's current policy and priority.

delta() treats a counter as unknown when it is missing, negative, non-finite, a boolean, or went backwards, and when its group is marked unavailable. A failed sample can therefore never turn stale or zero values into measurements. Reports older than the availability flags are still accepted.

check_core_telemetry_xros.py

This checks that the kernel APIs the collector uses resolve in the visionOS SDK, without running anything on a device (L83-L122):

• It prints SKIP (success) when xcrun --sdk xros has no SDK.
• It compiles a probe with -target arm64-apple-xros26.0 -Wall -Wextra -Werror -Wl,-fatal_warnings. The probe calls proc_pid_rusage(RUSAGE_INFO_V6) with a strong declaration, because a weak import would link even when the symbol is absent. It also calls thread_info (basic and extended), pthread_cpu_number_np, pthread_mach_thread_np, clock_gettime_nsec_np(CLOCK_UPTIME_RAW), task_info(TASK_POWER_INFO_V2) and mach_timebase_info.
• nm -m must show each of eleven names as (undefined) external _<name> (from libSystem).
• otool -l must show LC_BUILD_VERSION with platform 11 (XROS) and minos 26.0, which rejects an accidental macOS or simulator build.

check_probe_runner_exit.py

This parses tools/probe_engine_menu_input.py with ast, takes the statements from the receipt = assignment onward (the reporting tail), and executes them with stubbed child exit codes 0, 7, 124 and −11. It asserts that the shell exit codes are 0, 7, 124 and 139; that run.json records the child's exitCode and lists frame-0001.bgra as captured; that host.log is echoed; and that the evidence files are preserved. It launches no game and no GPU work (check_probe_runner_exit.py).

Desktop probe and its diagnostic tests

tools/probe_engine_menu_input.py builds a macOS executable from the host objects, a patched copy of EngineVisionRuntime.m, and Tests/MenuInputProbe.m. The patch inserts a call to menu_probe_present() at the end of metalwin_present and of metalwin_present_gpu. Hooking only the byte path once meant every probe measurement silently skipped the path the headset uses (probe_engine_menu_input.py:40-53). The probe runs the original engine with a cloned game directory and feeds input through the host's controller injection API. Its header says it "never writes engine/UI state to fake a result". Details of the runner are on Testing and Source Checks.

Probe environment variables (all opt-in, read in MenuInputProbe.m):

Variable	Effect
HALO_FRAME_CAPTURE	Directory for frame and snapshot captures
HALO_PROBE_SNAPSHOT_AT=<n>	One read-only snapshot at exactly published frame n: flat frame, HUD and panorama planes with epochs. Strict decimal in 1...INT_MAX; a missed frame is reported, never substituted.
HALO_PROBE_CAPTURE_PANORAMA	Also capture panorama planes with live snapshots
HALO_PROBE_FRAME_TIMING=1	Every 120 published frames: wall seconds, shell, tick, panorama state, [probe-cpu], [probe-split] (passes, readback, upload, sleep, yields, spin, budget, busy, onsets)
HALO_PROBE_GPU_CENSUS=<n>	Every n frames, lease the zero-copy slot and report missing or empty required layers
HALO_PROBE_GPU_CAPTURE_AT=<n[,n...]>	Save the actual leased layers with their epochs
HALO_PROBE_RENDER_AB=1	ABBA toggle of draw fast paths and texture CRC fast path in 240-frame blocks after frame 1441
HALO_OBJECT_CENSUS	Object table dump every 120 frames from 450
HALO_PROBE_FIND_CURSOR, HALO_PROBE_POINTER=u,v, HALO_PROBE_MOUSE_TRACE=dx,dy,frames, HALO_PROBE_MOUSE_STAIR=1	Menu cursor and pointer experiments
HALO_PROBE_INPUT_FILE, HALO_PROBE_INPUT_FROM_FRAME, HALO_PROBE_LIVE_SECONDS	Live JSON input mailbox. Each command has a short lease (20-2000 ms, default 500), must have an increasing sequence, and returns to neutral on expiry.
HALO_PROBE_FRAMES, HALO_PROBE_SUSPEND_AUDIO=1	Frame limit; suspend audio output for rendering-only runs
HALO_KEYSEQ, HALO_PADSEQ, HALO_STICKSEQ, HALO_NO_PROBE_INPUT	Scripted input sequences / disable scripted probe input

Three CPU-only tests #include "MenuInputProbe.m" (with its main renamed) and exercise its diagnostic helpers against synthetic guest memory. None of them starts the engine, a renderer or the GPU.

Test	Asserts
ExactFrameSnapshotProbeTests.m	Default off; rejects "", 0, -1200, +1200, " 1200", 1200junk and out-of-range values; nothing before the target; at the target, exactly 10 files (flat, three panorama planes and the HUD, each with JSON metadata carrying frame sequence, flat sequence, source epoch 777, status 1); single-shot; a missed target is never replaced by a later frame; a mismatched flat or panorama sequence saves nothing or only the flat frame; the guest sentinel memory is untouched
GuestPoseProbeTests.m	capture_guest_pose is read-only; the snapshot keeps frame and tick provenance, the unit datum, observer position and forward, and distinguishes the unit's origin from its animated bounding centre (used by original trigger tests); stale object datums expose no position; the UI-shell flag hides the unit; range checks and non-finite vector rejection
GuestWeaponTelemetryProbeTests.m	Read-only weapon telemetry: unit and weapon validity, magazine loaded and reserved rounds, trigger state, observed primary trigger from the host's merged mouse and pad state. Rejects wrong salts, slot indices, object types, table capacity, sizes and out-of-range pointers; unavailable input reported as null; output is valid JSON

These three are not invoked by tools/run_source_checks.py at this commit. They need the host headers and are compiled by hand.

Testing

Test	Asserts	Run by
CoreTelemetryValidation.swift	EngineCoreTelemetry.fields emits exactly the three groups; units (ns to s), tick bucket arrays, QoS order, availability flags and failure result codes survive a JSON round trip	run_source_checks.py (macOS)
CoreTelemetryCollectionValidation.m	Includes the production runtime with mocked proc_pid_rusage, thread_info, task_info and mach_timebase_info (125/3 timebase). Checks V6 field mappings and conversions, exactly four kernel queries, timebase failure and zero-denominator handling, reuse of an output after failure not retaining old counters, no worker port giving KERN_INVALID_ARGUMENT, NULL output, and off meaning false, zeroed output and no kernel query	run_source_checks.py
PanoramaPublicationValidation.m	Besides the pool: every Present feeds the frame split; the first sight of the game-time globals is an unknown interval; later deltas count ticks into the right bucket; enginevision_core_telemetry matches the split; errno preserved; with telemetry off a Present only bumps the CPU histogram and the split is unchanged	run_source_checks.py
DeepTelemetryValidation.swift	30 fps from measured intervals; disconnect events and counts; audio.zeroMixWithVoices, audio.noCallbackProgress.begin, panorama.staleLayer.begin; never-published layer age −1; a scene switch resets ages and clears signals when not in gameplay; a counter rollback gives 0 fps; observer.delayed.begin; JSON valid	run_source_checks.py
DiagnosticHistoryValidation.swift	2100 records retained in order; bounded rotation (128-byte segments, 2 kept, truncated flag, files ≤ 128 bytes); no overwrite of an existing timeline-000000.jsonl; a missing directory gives a nonfatal failure	run_source_checks.py
DiagnosticReportWriterValidation.swift	HALO_REPORT_SECONDS parsing and clamping; first/state/interval triggers; compact report with no samples and correct historyRecords; the write counter accounts for earlier writes; a 20,000-sample report serialises on the writer queue rather than the caller; deep history and live.json (latest record, written on events); unwritable report recorded, not fatal	run_source_checks.py
GatherDiagnosticsValidation.swift	Default fields; counts above 32 bits survive the C import and JSON; turning the switch off keeps accumulated counts	run_source_checks.py
test_watch_engine_vision_telemetry.py	Only <uuid>/live.json candidates; both build and buildID must match and runID must exist; maxObservedLayerAgeMs defaults to −1	run_source_checks.py
test_engine_vision_report_summary.py	16 cases: measured split and clock arithmetic and bucket fit; Build75 compatibility; Build78 audio, alignment and host fields; process counters without a frame snapshot; old counters without availability flags; missing optional counters; failed process and scheduling samples; a failed previous sample is not a zero baseline; failed frame snapshot; CPU coverage and switch availability; unknown ticks excluded from the denominator; counter reset, missing and bad numbers; zero denominators; rusage without hardware counters; the 4+ bucket and degenerate fit; legacy CPU baseline	run_source_checks.py
check_core_telemetry_xros.py, check_probe_runner_exit.py	See above	run_source_checks.py

Environment variables

Variable	Default	Effect	Read at
HALO_CORE_TELEMETRY	1 on device (worker default); on unless it starts with 0	Frame split, scheduling and process groups	EngineVisionRuntime.m:640, 658
HALO_REPORT_SECONDS	30	Report period, clamped to 1...600	EngineDiagnosticReportWriter.swift:49-52
HALO_AUDIO_VOICE_TRACE	1 on device	Bounded five-second DirectSound summaries in host.log	EngineVisionRuntime.m:634
HALO_CTLLOG, HALO_HSC_TRACE, HALO_HSC_TRACE_MAX	1, 120, 128 on device	Controller and script trace logging (host side)	L635-L636
HALO_CAPTURE_PC, HALO_CAPTURE_DIR	unset	Test-only guest memory capture	host_snapshot.h:6
Probe variables	unset	See the probe table	MenuInputProbe.m

Privacy

• Local only. Nothing in native/EngineVision/Sources opens a network connection. Reports are files in the app container; the user can share report.json from the setup window (ShareLink), browse the Files app, or pull files with the watcher over the paired device link.
• Privacy manifest. NSPrivacyTracking = false, no collected data types, one accessed-API category (file timestamp). See visionOS App.
• Minimised device details. Audio route logging records port types only, never device names or UIDs, and is capped at 64 lines (audio_session.inc:19-28). The hands-tracking usage string states that hands are never recorded. No hand or eye data is stored: menu targets are reduced to panel coordinates and sent to the engine.
• What reports do contain: the app bundle path, the OS version string, controller vendor names, head and eye matrices relative to the recentred pose (neutralOriginFromHead, neutralFromEyes, eyeProjections), thermal state, memory sizes, and up to four engine-frame PNGs. docs/BUILDING.md advises reviewing logs and reports before attaching them to public issues ("Logs and reports can contain paths and device information"), and the README asks reporters to keep raw diagnostics private.
• The watcher accepts only the expected build's records and keeps the full deep history on the headset.

Related pages

• visionOS App: the runtime that collects these counters
• Immersive Presenter: submission modes, world cadence, lifecycle trace, GPU time
• Layer Alignment: panoramaLayerAlign
• Frame Pacing, Panorama Budget and LOD, Panorama System
• Audio System, Input and Controllers, Haptics
• Threading and Synchronization: timed waits gated by host_core_telemetry
• Metal Renderer: pipeline statistics
• Testing and Source Checks, Release Process and Hygiene