Proposal: generated 3D content in DisplayXR apps (image→3D asset bridge with comfort-budget placement)

[joehillthunder]

AI image→3D models now turn one picture into a textured mesh in seconds. We tested whether that content can flow straight into DisplayXR-compatible apps and appear in their own 3D window, inside the comfort budget, with no per-app viewer.

Code: joehillthunder/displayxr-stability @ 6a311d7 (private, Apache-2.0 + NOTICE). It's standard-library-only Python, so it runs inside DCC interpreters; 30 tests pass.

What we built

• Generation: a hosted API or local open weights, run in a separate process so app interpreters never load ML frameworks. Tested models: SPAR3D and SF3D, under their Community License.
• Conversion: GLB → USD, Y-up cm. A single defaultPrim holds the materials, bindings and relative textures, so the file still works when referenced; it opens in Omniverse Kit with materials bound. OBJ is the fallback.
• Normalisation: each asset is scaled to a stated real-world size, with its base on the ground. Generated meshes have arbitrary scale and a root rotation, so without this, depth fitting is meaningless.
• Provenance: model, licence, source-image hash and an ai_generated flag, stored in USD customData plus a sidecar.
• Placement: depthfit.fit() returns the glass distance and an optional push-back from the asset bounds and the host camera. It uses the same true-geometry rig maths as the 3D Slicer (Proposal: DisplayXR for 3D Slicer (medical visualization, Leia SR tested) #2352) and ParaView (Proposal: DisplayXR for ParaView (scientific visualization, Leia SR tested) #2360) integrations. Panel size and viewing distance come from display info.

Hosts

• Native DCC plug-in (Maya): live, with 3D active.
• Offline-render stereo path: a renderer-neutral USD camera pair with film-back offset, tested with Cycles as a stand-in for a production renderer.
• Nuke and Houdini adapters: built, waiting on licences and installs.

Measured (Leia SR is the tested display)

• Rendered pair: −0.126 % / +0.414 % of panel width, against a budget of −0.14 / +0.42.
• Atlas capture in a 2268 px window: −0.112 / +0.375 %, against −0.130 / +0.390 % predicted.
• Maya asset: 0.140 % front analytically, ±0.05 % in pixels.
• Concurrent weaving beside another app: confirmed in runtime logs, not yet by eye.

What we'd ask of the contract

1. A comfort budget in display info (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388). Today it's the constants 0.14 % / 0.42 %, read through one function (comfort_budget()) that already prefers display info when it's present.
2. A "place content" helper in the shared host library, so adapters stop re-deriving the true-geometry glass solve. It takes bounds + camera + view rect and returns glass/push-back.
3. Nothing generation-specific in the runtime. Generated content goes through the same texture-mode / dedicated-3D-window path as any other app content.

App-side lessons, not runtime issues

• Depth fits must sample every shape type, not meshes only. A USD proxy shape was missed until that was fixed.
• For fixed film shots, move the glass, not the camera: g* = (f+b)/(b/near + f/far).

Pending: a hosted API run with a key, and Nuke and Houdini in-app tests.

🤖 Generated with Claude Code

[misakaai-dev]

A generated object feels much more useful when its scale survives the trip into a scene.