Proposal: DisplayXR for Blender (World Builder add-on: own-viewport 3D, measurements, host-library needs; Leia SR tested)

[joehillthunder]

Summary

Blender's built-in OpenXR session (OpenGL), driven by a Blender add-on, shows a world map or region in 3D on the tested display (Leia SR). It runs inside Blender's own 3D Viewport, beside other 3D apps. The pattern matches the architecture direction:

• a true-geometry rig;
• the content is brought to the glass and held inside a comfort budget;
• Blender's own viewport camera (mouse and gamepad) drives the 3D;
• the scene goes to Omniverse as USD.

The overlay embed (#2332) and its window properties are test scaffolding here. The target class for Blender is own viewport through an add-on, and texture mode is reachable: Blender's Python can render each eye offscreen (gpu.types.GPUOffScreen) and draw a texture into the viewport with a draw handler.

What it does (Blender 5.2.2, Windows 11, dev runtime local/test-build v2.53.0-24)

• Rig. It sets DXR_LEGACY_CAMERA_RIG before every session (Blender creates a new OpenXR instance per session): convergence at the nominal viewer distance, and a vertical FOV from the 3D view's own physical height (1:1). Absolute tracking. The base pose puts the glass centre at the stage origin, with the eye at the nominal viewer.
• Depth. The comfort budget is 0.14 % of panel width in front and 0.42 % behind. It stays a fraction of the whole panel's width even in a small window, because disparity limits are physical. A −/+ control (25 % steps, live) scales it, and Reset restores it. The panel size and nominal viewer come from display info (displayxr-cli status --json). The budget itself is a constant in the add-on's coupling module until display info publishes one (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388).
• Windowed. The 3D sits on the 3D Viewport's visible region, with the add-on sidebar left uncovered. Full panel is F11 only. Tested alongside USD Composer, Substance Modeler and VRED (logs: each kept weaving).
• Input. Blender's viewport camera is the single source of truth:
  • Orbit → the subject turns and tilts under the glass, capped by the depth budget. Zoom → fill.
  • The mouse reaches Blender through the 3D view: a posted middle-drag orbited the viewport −40° and the map turned 40°.
  • An XInput pad orbits and zooms the same camera, only while Blender is in front. The runtime's pad fly camera and controller emulation are off.
• Omniverse. "Send to Omniverse (USD)" writes Blender's own USD export in Kit's layout (Y-up, cm, defaultPrim). It opened in USD Composer with 0 composition errors, upright at 592 × 40.5 × 392 m, with 5 UsdPreviewSurface materials, and rendered with 3D active.

Measurements

• Disparity (sim-display, so true stereo without a seated viewer; GL atlas capture; 24 px block matching over the map): −0.08 % to +0.32 % of panel width (min to p98), against the −0.14 / +0.42 % budget.
• Placement. Each view is an off-axis frustum through the glass, and the untracked eye sits at the nominal viewer, 0.1 m above the glass centre. Treating the eye as level with the glass centre put the subject 10 cm high and clipped at the top edge. With the eye placed at the nominal viewer, the atlas bounding box is 179–2203 of 2400 rows (centred, 85 % fill). This corrects the base pose given in Proposal: panel-relative legacy camera profile (converge on the glass) + shipped blender.exe profile + desktop comfort budget #2330.

What a shared host library would need, from this adapter

1. Return the woven texture and its screen rect, so a Python add-on can draw it into the app's own viewport. That's texture mode, with no overlay window over menus and gizmos.
2. Display info readable from Python: panel size, nominal viewer, and the comfort budget (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388).
3. Let the app keep its own input. Its camera drives the rig; there's no runtime fly camera and no synthetic controllers by default.
4. A rig seeded per view from the view's physical size, re-seeded on resize. Today the profile is instance-scoped, so it's fixed until the next session.

Open

• No seated eyeball check yet of the re-centred framing.
• No physical-pad test yet (the mapping is tested headless).
• Linux is blocked: Blender's XR path is OpenGL and the runtime is Vulkan-only there (Linux: run OpenGL (XR_KHR_opengl_enable) legacy apps — VTK-based ParaView / 3D Slicer, Blender VR #2354).
• macOS is blocked: Blender has no VR session there, and there's no plug-in.

Related: #2330 (legacy profile), #2332 (overlay embed, test scaffolding), #2388 (comfort budget in display info).

🤖 Generated with Claude Code

[dfattal]

Following up with a result. Woven 3D now runs inside Blender's own 3D View and inside USD Composer's own viewport, with no overlay window. The app keeps its input and presents its own pixels. I checked both seated at a Leia SR panel.

How it works: the app renders two off-axis eyes into textures it owns and hands one D3D11 shared texture to the runtime through the XR_DXR_win32_window_binding v9 texture target. The runtime's display processor weaves into that texture, and the app draws the result back at its viewport rect.

• Blender: gpu.offscreen + draw_view3d for the eyes, and WGL_NV_DX_interop2 for the hand-off.
• Composer: a C++ extension and an omni.ui image layer, D3D12 → D3D11.

The runtime side is on main: #2373 (texture target), #2379 (windowless weave), #2380 (the hub weaves texture sessions). Correct phase inside a sub-rect of the window needs the Leia plug-in's windowless path, DisplayXR/displayxr-leia-plugin#450. That PR is open; until it ships, the released plug-in phases the weave from the window origin.

Measured:

• Blender 5.2.3: placement pixel-exact, including after a move or resize. Native per-view resolution. 92 Hz Workbench / 80 Hz EEVEE with both eyes rendered every frame. Clean start/stop/restart. It follows the 3D View into a maximized area or a second Blender window.
• Composer: placement exact, and the weave runs at 60–72 Hz. The eye images come at 13–14 new frames/s on a laptop RTX 3080, so Kit is limited by the render, not the weave.

On your Kit black-frame finding: it's the same root cause. Kit drops a frustum that doesn't contain the view axis. The prototype works around it with a frustum that contains the axis, cropped back to the true off-axis frustum. A view-centered frustum option on Kit's side would remove the crop.