[Leia SR SDK] Add EXTERNAL_ROUTING mode for multi-display integration

[dfattal]

Background

DisplayXR is implementing multi-display vendor DP routing (ADR-015). In a multi-display setup, DisplayXR needs to control which display processor is instantiated for which monitor. The Leia SR SDK currently handles multi-monitor detection internally, which conflicts with DisplayXR's routing.

Problem: When a machine has Monitor A (Leia 3D) and Monitor B (another vendor or sim_display), DisplayXR must route the correct DP to each monitor. The Leia SDK's internal monitor detection (getDrawRegions(), canWeave(), polling threads, installCustomWindowProc()) interferes because:

• The SDK's WndProc hook conflicts with DisplayXR's window management
• Background polling threads create race conditions with DisplayXR's routing decisions
• canWeave() may return false when DisplayXR has already guaranteed the window is on a Leia display
• Developers cannot force sim_display on a Leia-connected monitor for testing

What We Need

A new initialization mode (EXTERNAL_ROUTING or SR_WEAVER_FLAG_EXTERNAL_ROUTING) that, when set at weaver creation time:

1. Disable monitor polling threads

The two background threads (pollingLoopMonitors, pollingLoopFPCs) must not be spawned. DisplayXR will notify the SDK of relevant display changes via existing APIs or a new callback.

2. Skip installCustomWindowProc()

The SDK must NOT subclass the application's window procedure to intercept WM_WINDOWPOSCHANGING for phase snapping. DisplayXR handles window positioning.

3. Always weave (skip canWeave checks)

canWeave() / canWeaveInternal() should always return true (or the equivalent). DisplayXR guarantees it will only create a Leia DP instance for windows that are actually on a Leia display.

4. Skip getDrawRegions() splitting

Treat the full window client area as a single 3D region. DisplayXR provides the correct canvas region via the process_atlas() parameters (canvas_offset_x/y, canvas_width/height).

5. Backward compatible

No flag = current behavior. Existing integrations are unaffected.

Why This Is Needed

• DisplayXR may need Leia weaving on monitor A and sim_display passthrough on monitor B. The SDK cannot detect or know about non-Leia monitors; only DisplayXR has the full picture.
• When a window moves from a Leia display to a non-Leia display, DisplayXR destroys the Leia DP and creates a sim_display DP. If the SDK is also tracking window position, there will be race conditions.
• For development, developers need to force sim_display on a Leia-connected monitor without the SDK auto-detecting and weaving.

Suggested API

// Option 1: Flag at weaver creation
WeaverErrorCode CreateDX11Weaver(SRContext* context,
                                 ID3D11DeviceContext* d3d11Context,
                                 HWND window,
                                 DWORD flags,          // NEW: SR_WEAVER_FLAG_EXTERNAL_ROUTING
                                 IDX11Weaver1** weaver);

// Option 2: Flag on context
sr::SRContext context(sr::SRContext::Params()
    .setFlag(SR_FLAG_EXTERNAL_DISPLAY_ROUTING));

Acceptance Criteria

• When EXTERNAL_ROUTING is set, no background threads are spawned for monitor polling
• When EXTERNAL_ROUTING is set, installCustomWindowProc() is not called
• When EXTERNAL_ROUTING is set, the weaver always produces interlaced output (no canWeave check)
• When EXTERNAL_ROUTING is set, getDrawRegions() returns a single region covering the full window
• Existing behavior (no flag) is unchanged for backward compatibility
• Unit test demonstrating the flag works

Key SDK Files

File	What needs to change
modules/DimencoWeaving/sr/weaver/WeaverBaseImpl.h	Add flag field, conditional thread/WndProc setup
modules/DimencoWeaving/sr/weaver/WeaverBaseImpl.ipp	Guard pollingLoopMonitors, pollingLoopFPCs, installCustomWindowProc, canWeaveInternal, getDrawRegions
modules/srDirectX/dxweaver/sr/weaver/dx11weaver.h	Expose flag in constructor/factory
modules/srDirectX/dxweaver/sr/weaver/dx12weaver.h	Expose flag in constructor/factory

Related

• Relates to Multi-display compositing on single machine #69 (multi-display compositing on a single machine)
• See ADR-015 for the full architectural decision

[dfattal]

Reference: Acer SpatialLabs ships this as "Panoramic View" — three SpatialLabs Pro displays combined into one wider 3D viewing area. Each display is weaved independently but presents a continuous 3D scene with a widened viewing frustum.

This confirms split-weave across display boundaries is a first-class use case. DisplayXR needs to own the atlas splitting and per-display DP routing so this works with any vendor combination, not just same-vendor setups like Acer's.

[dfattal]

Additional requirement: probe_displays() interface

After further analysis, DisplayXR also needs Leia to expose a display probe function that returns which OS monitors are Leia hardware. This wraps the existing internal logic:

1. getMonitorList() + getKnownMonitors() — EDID-based identification (manufacturer+product ID matching)
2. FPC shared memory read (Global\sharedDeviceSerialMemory) — serial number / product code

The probe should return a list of (HMONITOR, confidence, serial_number) tuples:

• EDID match only (no FPC connected): lower confidence — display identified but lens control unavailable
• EDID + FPC match: full confidence — display identified and FPC authenticated

This is separate from the EXTERNAL_ROUTING mode. The probe is called at startup and on hot-plug events to build the DP factory registry. EXTERNAL_ROUTING governs how the weaver behaves once created.

Suggested API:

struct SRDisplayClaim {
    HMONITOR monitor;
    int confidence;          // 50 = EDID only, 100 = EDID + FPC
    char serial[64];         // SR serial number (empty if no FPC)
};

// Returns which monitors are recognized as SR displays
sr::Result srProbeDisplays(SRContext* context,
                           SRDisplayClaim* out_claims,
                           uint32_t* out_count);

[dfattal]

Additional requirement: multi-FPC eye tracking

In multi-display setups (e.g., Panoramic View with 3 displays), each display has its own FPC + camera. Currently the SDK only supports one primary display/eye tracker per SR Service instance (there are TODOs in the code for this).

For DisplayXR multi-display to work, the SDK needs:

1. Multiple simultaneous eye trackers — one per FPC/display, each returning eye positions calibrated to its own display's camera
2. Per-display eye tracker creation — the eye tracker factory must accept the FPC serial/display identity so it binds to the correct camera and calibration files
3. probe_displays() must return the FPC serial in each claim so the DP factory can create a DP tied to the correct FPC/camera/calibration

On the DisplayXR side, each DP instance already provides get_predicted_eye_positions() per display — we just need the SDK to support creating multiple independent instances.

[dfattal]

Revised model: HWND presence/absence replaces EXTERNAL_ROUTING flag

After further design work (ADR-015 updated), we've simplified the vendor contract. Instead of an explicit EXTERNAL_ROUTING flag, the behavior is controlled by whether window_handle is provided or NULL:

When HWND is provided (primary display DP):

• DP owns phase snapping via WndProc (WM_WINDOWPOSCHANGING) — this is critical because Windows DWM blits the stale weaved frame during drag without re-running the shader, so phase-snapping the window position is the only way to preserve 3D quality during drag
• DP disables internal monitor polling, canWeave(), getDrawRegions() — DisplayXR handles routing
• DP weaves everything it receives via process_atlas()

When window_handle = NULL (secondary display DP):

• DP must not hook WndProc or call MonitorFromWindow()
• DP computes phase entirely from canvas_offset_x/y in process_atlas()
• DP weaves everything it receives — no canWeave() check

How DisplayXR uses this:

• DisplayXR determines which display has the majority of the window area → "primary display"
• Primary display's DP gets the real HWND → handles phase snapping
• Secondary display DPs get NULL → just interlace based on canvas offset
• When the window moves mostly onto a different display → hot-swap: destroy old primary DP, create new with HWND

What Leia needs to implement:

1. Accept window_handle = NULL in CreateDX11Weaver / CreateDX12Weaver — skip installCustomWindowProc(), skip monitor polling threads, always weave
2. When HWND is provided, still install WndProc for phase snapping but skip canWeave() / getDrawRegions() (DisplayXR guarantees the window is on a Leia display)
3. Backward compatible: existing single-display behavior unchanged when HWND is provided without the new SDK version

This replaces the earlier SR_WEAVER_FLAG_EXTERNAL_ROUTING concept with a simpler contract.

[dfattal]

User Stories — Reframed

Thanks for the feedback. Here's a reframe as user stories. We're describing what DisplayXR needs from the SDK — how you implement it is entirely up to you.

---

User Story 1: Display Probing

As the DisplayXR runtime,
I need to ask the Leia SR SDK "which OS monitors are your hardware?",
so that I can build a per-display routing table that maps each monitor to the correct vendor display processor.

Why: DisplayXR manages multiple displays that may come from different vendors (or sim_display for development). Only the vendor SDK knows how to identify its own hardware (EDID matching, FPC handshake, etc.). DisplayXR doesn't need to understand the detection method — it just needs the result.

Acceptance criteria:

• Given a machine with monitors A (Leia), B (non-Leia), and C (Leia): the SDK returns a list claiming A and C, with enough identity info per claim to later create a display processor tied to the correct physical display + camera/calibration
• The probe works at startup and can be re-called on hot-plug events
• If an FPC is connected (display fully authenticated), the claim includes an identifier that can be used to create a per-display eye tracker
• If no FPC is connected (EDID-only detection), the claim still identifies the display but indicates reduced confidence

---

User Story 2: Cooperative Weaving (DisplayXR Controls Routing)

As the DisplayXR runtime,
I need to create a weaver that trusts my routing decisions and always weaves what I send it,
so that DisplayXR — not the vendor SDK — controls which display gets 3D content and which doesn't.

Why: Today the SDK auto-detects whether the window is on a Leia display and may refuse to weave (canWeave() returns false), split the window across display boundaries (getDrawRegions()), or track window position via background threads. This conflicts with DisplayXR's routing because:

• DisplayXR may intentionally route sim_display to a Leia monitor (for development/testing)
• DisplayXR handles display-boundary splitting itself (per ADR-015)
• DisplayXR needs to destroy/create DPs as windows move between displays — racing with SDK polling threads causes issues

Acceptance criteria:

• DisplayXR can create a weaver in a mode where: the weaver always produces interlaced output for whatever content DisplayXR sends (no internal "am I on a 3D display?" check)
• The SDK does not independently track which monitor the window is on — DisplayXR guarantees it only sends content destined for a Leia display
• No background monitor/FPC polling threads that could conflict with DisplayXR's routing decisions
• Existing behavior (SDK manages everything) remains the default for apps using the SDK directly — this is opt-in

---

User Story 3: Headless Weaving (No HWND)

As the DisplayXR runtime,
I need to create a weaver without providing a window handle,
so that I can weave content for a secondary display that doesn't own the application window.

Why: When a window spans two Leia displays, DisplayXR creates two DP instances — one per display. Only one display is the "primary" (where the window mostly is). The primary DP gets the HWND so it can handle phase snapping during drag. The secondary DP just needs to interlace its portion of the atlas using canvas offset parameters — it has no window to hook.

Acceptance criteria:

• A weaver can be created with window_handle = NULL
• The NULL-HWND weaver computes interlacing phase from the canvas offset parameters provided in process_atlas() (not from window position)
• The NULL-HWND weaver does not attempt to hook a window procedure or call MonitorFromWindow
• A weaver created with a real HWND continues to handle phase snapping (vendor-owned, as today) but respects the cooperative weaving contract from Story 2

---

What DisplayXR owns vs. what the vendor owns

Responsibility	Owner
Which monitors exist and their geometry	OS / DisplayXR
Which monitors are vendor hardware	Vendor SDK (via probe)
Which DP is used for which monitor	DisplayXR (routing)
Display-boundary detection and atlas splitting	DisplayXR
Interlacing / weaving algorithm	Vendor SDK
Phase snapping during window drag	Vendor SDK (for primary HWND only)
Eye tracking per display	Vendor SDK (per-DP instance)
Creating/destroying DPs on display changes	DisplayXR

---

Priority

Story 2 (cooperative weaving) is the highest priority — it unblocks single-display integration improvements and is a prerequisite for Stories 1 and 3. Story 1 (probing) is needed for multi-display. Story 3 (headless) is needed for spanning windows.

Happy to discuss any of these further. The implementation approach is yours — flags, context modes, separate factory methods, whatever works best in the SDK architecture.

[dfattal]

Addendum: DisplayXR already runs three-layer display identification

To motivate why we believe this user story is sound — DisplayXR already implements its own display identification pipeline on our side, independent of the SR SDK:

Layer 1 — EDID hardware match (leia_edid_probe.c): We enumerate all OS monitors via GDI + SetupDI, read each monitor's EDID blob from the registry, and match manufacturer+product IDs against a table of 160+ known Leia display panels (Dell, Acer, LG, Samsung, Barco, etc.). This gives us the HMONITOR handle, pixel dimensions, refresh rate, and screen position for each matched display.

Layer 2 — SR SDK installation check: We check the Windows registry for HKLM\SOFTWARE\Dimenco\Simulated Reality. This tells us the SDK is installed on the machine.

Layer 3 — SR Service runtime check: We probe the shared memory mapping Global\sharedDeviceSerialMemory to confirm the SR Service is actively running.

All three layers must pass before we claim a display as Leia hardware. If EDID matches but the SDK isn't installed or the service isn't running, we log a warning and fall back to sim_display.

What this means for the user stories: DisplayXR already knows which monitors are Leia displays before we ever talk to the SR SDK. What we need from the SDK is:

1. Probing (Story 1): Confirmation of our detection + the FPC serial/identity needed to create per-display eye trackers. Our EDID detection can't provide this — it's the FPC handshake that ties a specific camera+calibration to a specific display.
2. Cooperative weaving (Story 2): Since we already handle display routing, we need the SDK to trust our decisions rather than running its own parallel detection.
3. Headless weaving (Story 3): For secondary displays in a multi-monitor setup, we need a weaver that works from canvas offsets alone.

The three-layer detection is already shipping in the runtime (src/xrt/drivers/leia/leia_edid_probe.c, src/xrt/auxiliary/os/os_display_edid_win32.c). We're not proposing a theoretical architecture — this is the foundation we're building on.

[dfattal]

Great questions — let me address each one. Some of this touches on how DisplayXR is architected, so I'll provide context where helpful.

Architecture context

DisplayXR is a full OpenXR runtime that sits between apps and the display hardware. The stack looks like this:

App (D3D11/D3D12/VK/GL/Metal)
       │
  OpenXR State Tracker (API implementation)
       │
  Native Compositor (one per graphics API — owns the swapchain, window, rendering)
       │
  Display Processor (vendor-specific — weaving, eye tracking)
       │
  3D Display

The Display Processor (DP) is the integration point for Leia. It's a vtable with methods like process_atlas(), get_predicted_eye_positions(), etc. The DP implementation wraps the Leia SR SDK's weaver API. Today this code lives in src/xrt/drivers/leia/ and is written on the DisplayXR side, but it could be written by anyone with access to the SR SDK — the boundary is the DP vtable interface, not who authors the code.

The key point: the compositor owns the window, the backbuffer, and the rendering pipeline. The DP receives a fully composited atlas (left + right eye tiles) and weaves it. The DP never creates windows, never manages swapchains, and never decides whether to weave or not — it always weaves what it receives.

---

Polling FPC thread (correction map reload)

Will still be needed. What is the experienced issue?

Today, no issue — we only support single-display. The concern is forward-looking for multi-display (#69):

In the multi-display model, DisplayXR creates one DP instance per display. Each DP is tied to a specific physical display + FPC. If correction maps change, the polling should be scoped to that specific DP's FPC — which it naturally would be if the DP instance knows which FPC it belongs to.

The potential issue is if a single global polling thread is making decisions that assume there's only one active display. As long as multiple independent DP instances can coexist (each polling its own FPC), this should be fine. So this may be a non-issue — just something to verify works with multiple simultaneous instances.

---

Polling monitor thread (display topology)

Display topology doesn't change much. What is the experienced issue?

The issue isn't frequency — it's ownership. In DisplayXR's architecture:

• DisplayXR enumerates monitors at startup (via our EDID probe — see previous comment about three-layer detection)
• DisplayXR decides which DP to create for which monitor
• DisplayXR monitors for topology changes (hot-plug, display connect/disconnect)

If the SDK is also independently tracking monitor topology, the two systems may disagree during transitions. For example: user unplugs a Leia display → DisplayXR detects this and destroys the DP → but the SDK's polling thread hasn't noticed yet and is still referencing the old state. Or vice versa.

This is a coordination problem. It's not a bug today (single display), but becomes one when DisplayXR is actively creating/destroying DP instances based on its own routing decisions. The simplest solution is for DisplayXR to be the single source of truth for "which displays exist and which DP serves each one," and for the SDK to not independently track this.

That said, if the SDK's monitor polling is lightweight and doesn't make routing decisions (just tracks topology passively), it may coexist fine. The real issue is if the SDK uses this information to change behavior — e.g., refusing to weave because it thinks the window moved off a Leia display.

---

Phase alignment and frame delay

Is DisplayXR adding a frame delay due to copying app backbuffer into a composited DisplayXR backbuffer?

Single-app mode (standalone): No extra copy. The native compositor renders directly into its own backbuffer, calls the DP's process_atlas() on that backbuffer, and presents. Same frame, same pipeline as a direct SDK integration. No phase alignment issue.

Multi-app mode (shell): The multi-compositor imports shared textures from per-app compositors, renders all windows as spatial quads into a combined backbuffer, then calls the DP's process_atlas(). This is one additional blit (the spatial compositing pass), but it happens in the same frame — there's no extra frame of latency. The weaver receives the final composited backbuffer at the correct window position, same as single-app.

Phase is computed from the window's position on screen. Since the compositor owns the window and the window position doesn't change between compositing and weaving (it's all in the same frame), phase should be correct.

One nuance: In the current single-display integration, the SDK's WndProc hook does phase snapping during window drag (WM_WINDOWPOSCHANGING → constrain to lens pitch grid). This works today and we want to keep it working — that's why User Story 3 specifies that the primary DP (the one with the HWND) continues to own phase snapping. We're not trying to take that away.

---

sim_display

What is sim_display and what is the actual issue?

sim_display is a simulation driver built into DisplayXR. It provides a fake 3D display with WASD + mouse eye position control, so developers can build and test apps on any standard monitor — no Leia hardware needed. It outputs a side-by-side stereo view without any interlacing/weaving.

It's essential for development — most of our contributors don't have Leia hardware on their desk.

The routing issue: Today, if a developer does have a Leia display connected but wants to test with sim_display (e.g., to debug rendering without weaving, or to compare behavior), they can't — because the SDK auto-detects the Leia display and the system always routes to the Leia DP. With runtime-controlled routing, DisplayXR could say "use sim_display on this monitor even though it's Leia hardware." This is a developer workflow concern, not an end-user scenario.

The comment "DisplayXR can just bypass calling weave()" is close — but sim_display isn't just "skip weaving." It's a complete alternative driver with its own eye simulation, view computation, and display output. Bypassing weave() would give you an unweaved Leia output, which isn't the same as a simulated display.

---

"LeiaSR runtime code could take care of these things already"

You're right that the DP integration code (whoever writes it) could handle some of this. For single-display, it mostly does today. The gap is:

1. Multi-display (Multi-display compositing on single machine #69): Multiple DP instances need to coexist, each tied to a specific display + FPC. The current SDK assumes one primary display per process.
2. Display routing: The DP integration code can wrap the SDK and add logic, but it can't override SDK behavior that happens inside the weaver (auto-detection, internal polling, canWeave checks). That's why we need a cooperative mode — so the wrapper can tell the SDK "trust me, just weave."

---

Auto-occlusion, auto-lens, auto-2D

I would opt for a method to disable the auto occlusion handling and auto lens management. Possibly also disable auto 2D mode when user lost.

Yes — this aligns well with User Story 2 (cooperative weaving). DisplayXR already manages:

• Rendering mode switching (2D/3D/multiview) via XR_EXT_display_info and user input (V key, mode keys)
• "User lost" behavior — the compositor continues rendering; the app decides what to do

If the SDK is independently switching to 2D mode when it loses eye tracking, or managing lens states, that conflicts with DisplayXR's mode management. Disabling these auto-behaviors when in cooperative mode is exactly what we need.

---

Summary

For the immediate term (single display), the main ask is cooperative weaving (Story 2) — a mode where the SDK trusts the runtime's routing decisions, doesn't auto-switch to 2D, and doesn't manage occlusion/lens state. This is close to what you suggested with disabling auto-occlusion/lens/2D.

For multi-display (#69, future), we'll additionally need display probing (Story 1) for FPC identity and headless weaving (Story 3) for secondary displays. But those can come later.

Happy to jump on a call if it would help to walk through the architecture together.

[dfattal]

sim_display — you're right

Fair point. The sim_display scenario is a routing decision on our side — we just don't create a Leia DP and create a sim_display DP instead. That's our problem to solve, not an SDK change. I'll drop that as a motivation for the SDK work.

Atlas positioning without HWND

Good question. Here's how it works:

Single-display (today): The compositor owns the HWND. The backbuffer is the full window client area. The DP's process_atlas() receives the backbuffer plus canvas offset parameters (canvas_offset_x/y, canvas_width/height) that describe where the app's content sits within the backbuffer. The weaver uses the HWND position on the monitor to compute phase.

Multi-display (future, spanning window): When a window spans two monitors, DisplayXR splits the atlas at the display boundary and calls each DP with only its portion. Each DP gets a backbuffer-sized texture containing just the content for its monitor. So yes — your intuition is correct: without an HWND, the atlas fills the whole monitor. The DP can assume position (0,0) relative to the target monitor's top-left corner.

setMonitor(HMONITOR) — I like this

This is a clean approach. If the weaver knows which HMONITOR it targets, it has everything it needs:

• Phase computation from the monitor's position in the virtual desktop
• Lens parameters / correction maps tied to that specific display
• No HWND dependency for secondary displays

For the primary display (where the window physically lives), we'd still pass the HWND so the weaver can do phase snapping during drag — that's the one case where the weaver needs to hook WM_WINDOWPOSCHANGING. But setMonitor() would tell the weaver which monitor it's working on regardless.

So the model could be:

Case	HWND	HMONITOR	Phase source
Single display (today)	provided	could be inferred or explicit	HWND position (+ phase snapping during drag)
Multi-display, primary	provided	explicit via setMonitor()	HWND position (+ phase snapping)
Multi-display, secondary	NULL	explicit via setMonitor()	Monitor origin (0,0) — atlas fills monitor

This feels like it converges nicely. The HMONITOR tells the weaver which display (and therefore which correction maps, lens parameters, FPC) to use. The HWND, when present, adds drag-time phase snapping on top.

For the call

Would be great to walk through:

1. The setMonitor(HMONITOR) approach — how it fits into weaver creation or as a runtime method
2. The cooperative mode (disabling auto-occlusion, auto-lens, auto-2D) — what's the easiest way to expose this
3. Whether multiple weaver instances can coexist in one process (each targeting a different HMONITOR/FPC)

Let me know when works for you.