Deformable shadow model sync to support ovphysx + ovrtx/newton_warp combinations

[huidongc]

Description

Context

When PhysX / OVPhysX simulates and Newton Warp / OVRTX renders:

Role	Who owns it	What it looks like
Simulation	PhysX / OVPhysX SceneData get_points	Soft-body nodal positions (tet nodes for volumes; surface verts for cloth)
Visualization	Newton shadow particle_q and/or USD visual Mesh.points	Authored visual triangle mesh

Issues before this PR

1. Warp “tet spaghetti” — shadow model built from sim tet topology + live sim nodes → renderer draws tet surfaces, not the visual cuboid/cloth. See Before vs. After section.
2. OVRTX static rest pose — live sync broken → clean visual mesh but no motion/deformation. See Before vs. After section.

Design goal

Keep SceneData sim-true (nodal counts from physics), while consumers that render the authored mesh see vis-sized positions every frame — shared between Warp and OVRTX.

Architecture

Schema hierarchy on the USD stage

Spawners (define_deformable_body_properties) and pre-authored USD share the same OmniPhysics layout.
OmniPhysicsDeformableBodyAPI marks the body root; sim/vis meshes carry the sim and pose APIs.

Root Xform / Mesh
  OmniPhysicsDeformableBodyAPI          ← body identity (discovery key)
  │
  ├── SimulationMesh  (TetMesh volume | Mesh surface; purpose=guide)
  │     OmniPhysicsVolumeDeformableSimAPI  or  OmniPhysicsSurfaceDeformableSimAPI
  │     UsdPhysics.CollisionAPI             ← collider usually = sim mesh
  │     OmniPhysicsDeformablePoseAPI:default
  │       purposes = ["bindPose"]           ← register pose for embedding
  │
  └── VisualMesh  (UsdGeom.Mesh; often different vertex count)
        OmniPhysicsDeformablePoseAPI:default
          purposes = ["bindPose"]
          points   = authored vis rest verts  ← skin/embed to sim via bindPose

Schema	Where	Role
OmniPhysicsDeformableBodyAPI	body root	Marks the deformable; subtree deforms with it
OmniPhysics*DeformableSimAPI	sim mesh	FEA topology / rest shape (volume tet or surface tri)
UsdPhysics.CollisionAPI	sim mesh (typical)	Collision participation
OmniPhysicsDeformablePoseAPI	sim and vis meshes	Multiple-apply bind pose so vis/collision can differ from sim

Optional third child (collision ≠ sim) also gets CollisionAPI + DeformablePoseAPI:bindPose. Isaac Lab spawners usually collide on the sim mesh.

Pipeline (spawn → render)

┌──────────────────────────────────────────────────────────────────────────┐
│ Spawners (meshes / from_files) — asset authoring                         │
│  create visual Mesh, then define_deformable_body_properties()            │
│  → applies schema hierarchy above (Body / Sim / Pose / Collision)        │
│  → authors sim TetMesh/Mesh (+ keeps sibling vis Mesh)                   │
│  (pre-authored USD may already have this setup and skip spawners)        │
└───────────────────────────────┬──────────────────────────────────────────┘
                                |
                                |
                                |
                                ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ USD stage (see schema hierarchy above)                                   │
│  BodyAPI root → sim TetMesh/Mesh + vis sibling Mesh (V counts may differ)│
└───────────────────────────────┬──────────────────────────────────────────┘
                                |
                                |
─────────────────────────────── │ ─ UNCHANGED above (authoring prerequisite) ─
                                |
                                │
                                │ discover_deformables_on_stage()
                                │ called by PhysxManager / OvPhysxManager /
                                │ NewtonManager
                                │
              ┌─────────────────┴─────────────────┐
              ▼                                   ▼
┌─────────────────────────────┐   ┌──────────────────────────────────────────┐
│ PhysxManager /              │   │ NewtonManager                            │
│ OvPhysxManager              │   │  visualization_builder                   │
│  DeformableStageEntry →     │   │    → add_shadow_deformables_to_builder   │
│  SceneDataBackend points    │   │    • volume + count mismatch →           │
│  (sim-sized nodal export)   │──▶│      VolumeVisRemap + add_cloth_mesh     │
│                             │   │    • volume 1:1 → add_soft_mesh          │
│                             │   │    • surface → add_cloth_mesh            │
│                             │   │  ShadowDeformableEntity (sim+vis slots)  │
│                             │   │  _sim_particle_q ← SceneData (sim-sized) │
│                             │   │  particle_q ← remapped/copied (vis-sized)│
└─────────────────────────────┘   └───────────┬──────────────┬───────────────┘
                                              │              │
                                              ▼              ▼
                                   Newton Warp renderer   OVRTX (USD vis
                                   reads shadow           Mesh.points ←
                                   particle_q             particle_q slices)

One-sentence summary: managers discover sim/vis meshes; shadow build allocates dual particle layouts and (for volumes) a barycentric table; each frame PhysX/OVPhysX SceneData fills sim slots and a Warp kernel fills vis slots for renderers.

Type of change

• New feature (non-breaking change which adds functionality)
• Documentation update

Before vs. After

Rendering

	Before	After
“tet spaghetti”
Rendering Sync

Total startup (s)

(128 envs, 100 steps)

Task	Physics	Renderer	Before	After	Δ %
Isaac-Lift-Cloth-Franka-Camera	ovphysx	ovrtx	81.4	93.0	+14%
Isaac-Lift-Cloth-Franka-Camera	ovphysx	newton_renderer	75.0	90.6	+21%
Isaac-Lift-Soft-Franka-Camera	ovphysx	ovrtx	77.0	81.4	+6%
Isaac-Lift-Soft-Franka-Camera	ovphysx	newton_renderer	70.0	77.5	+11%

Mean Total FPS

(128 envs, 100 steps)

Task	Physics	Renderer	Before	After	Δ %
Isaac-Lift-Cloth-Franka-Camera	ovphysx	ovrtx	1841	1574	−15%
Isaac-Lift-Cloth-Franka-Camera	ovphysx	newton_renderer	5072	4575	−10%
Isaac-Lift-Soft-Franka-Camera	ovphysx	ovrtx	1907	1601	−16%
Isaac-Lift-Soft-Franka-Camera	ovphysx	newton_renderer	5766	5729	−1%

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

The PR adds deformable scene-data synchronization into shadow Newton models and addresses the previously reported discovery, registry, and rendering-validation gaps.

• Adds PhysX and OVPhysX deformable point backends, geometry mapping, and per-frame shadow-particle synchronization.
• Adds deformable USD discovery, visual-mesh selection, and barycentric simulation-to-visual remapping.
• Populates OVRTX deformable registry metadata for cloned and standalone scenes.
• Restores Franka cloth and soft-body golden-image validation and adds corresponding fixtures.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains in the reviewed fixes, and the previously reported standalone metadata, registry path, sibling discovery, visual selection, and golden-validation issues are addressed by the current code and focused tests.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/scene_data/deformable_discovery.py	Discovers deformable simulation and visual meshes, including direct siblings, and deterministically ranks multiple visual candidates.
source/isaaclab/isaaclab/scene_data/deformable_vis_remap.py	Builds and executes barycentric mappings from volume simulation nodes to visual-mesh vertices.
source/isaaclab/isaaclab/scene_data/scene_data_provider.py	Adds geometry path mapping and bounded point-copy support to the central scene-data provider.
source/isaaclab_newton/isaaclab_newton/physics/visualization_builder.py	Builds shadow visualization models with deformable metadata in cloned and standalone scenes.
source/isaaclab_newton/isaaclab_newton/physics/visualization_deformables.py	Creates shadow deformable render slots and registry paths while preserving exact standalone paths.
source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py	Synchronizes backend nodal positions into shadow Newton particle state and applies volume visual remaps.
source/isaaclab_tasks/test/rendering_test_utils.py	Restores direct golden-image validation for Franka cloth and soft-body rendering tests.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    USD[USD deformable geometry] --> Discovery[Deformable discovery]
    PhysX[PhysX / OVPhysX nodal tensors] --> Backend[SceneDataBackend points]
    Backend --> Provider[SceneDataProvider geometry mapping]
    Discovery --> Shadow[Shadow Newton model and registry]
    Provider --> SimBuffer[Simulation-sized particle buffer]
    SimBuffer -->|direct copy| RenderBuffer[Newton particle_q render slots]
    SimBuffer -->|barycentric remap| RenderBuffer
    Shadow --> RenderBuffer
    RenderBuffer --> Consumers[Newton Warp and OVRTX renderers]

Loading

_{Reviews (4): Last reviewed commit: "Document shadow deformable and SceneData..." | Re-trigger Greptile}

[huidongc]

@greptile-apps

[huidongc]

@greptile-apps

[huidongc]

@greptile-apps

[isaaclab-review-bot]

Isaac Lab Review Bot

The deformable SceneData and Newton shadow-model integration is well structured, but concrete correctness and lifecycle issues remain: shadow particle offsets can target the wrong slots when the builder already contains particles, deformable-only OVPhysX scenes skip binding setup, the changelog gives impossible migration guidance, and teardown retains the simulation particle buffer.

• Design and architecture: The backend/provider/shadow-builder split is coherent. However, shadow deformables must account for particles already present in the Newton builder; using a separate zero-based render cursor breaks synchronization for builders containing pre-existing particle data.
• API: The new SceneData properties and methods are additive with safe backend defaults. The Newton changelog incorrectly tells callers of update_visualization_state to pass allow_passthrough=True even though that method has no such parameter, so the migration guidance must be corrected.
• Implementation: OVPhysX deformable setup is currently bypassed when no rigid-body patterns are found, preventing deformable-only scenes from exporting points. Shadow render offsets ignore existing builder particles, and NewtonManager.clear() does not release the newly added _sim_particle_q device buffer.

Significant concerns. Posted 4 actionable findings inline.

Conservative automated review; human maintainers own approval decisions.

[huidongc]

@isaaclab-review-bot review

[isaaclab-review-bot]

Isaac Lab Review Bot

The dual sim/render particle layout is coherent, but two concrete issues remain: failed volume remaps can trigger an invalid direct copy, and new public Warp interfaces lack the required concrete dtype annotations.

• Design and architecture: The shared discovery and dual-buffer design is sound. However, mismatched volume meshes need an explicit safe fallback when barycentric remap construction fails.
• API: The additive SceneData geometry API is compatible, but VolumeVisRemap and launch_volume_vis_remap use generic wp.array annotations instead of the concrete Warp dtypes required for public interfaces.
• Implementation: When a mismatched volume remap is unavailable, the render sync copies vis_particle_count values directly from the sim slice. This can cross entity or buffer bounds and does not provide a valid sim-to-visual mapping; the fallback must skip the copy or use compatible sim topology.

Minor fixes needed. Posted 2 actionable findings inline.

Conservative automated review; human maintainers own approval decisions.

[huidongc]

@isaaclab-review-bot review

[isaaclab-review-bot]

Isaac Lab Review Bot

The sim-sized SceneData to vis-sized shadow-render design is reasonable, but three concrete correctness issues remain: unresolved geometry mappings collapse deformables to zero, non-environment deformables are imported twice in clone-plan scenes, and shadow mesh placement applies the deformable root transform twice.

• Design and architecture: The clone-plan builder path imports deformables outside /World/envs through the top-level USD import and then adds them again through the shadow-deformable path, producing duplicate particle allocations and invalid offsets.
• API: The additive SceneData point and geometry APIs are documented and retain safe backend defaults. However, geometry mapping uses exact path equality while backends may expose related child paths; unresolved mappings are not propagated to the subsequent render sync, causing skipped source slices to be treated as valid zero data.
• Implementation: Unmapped deformables have zero-initialized sim slices copied into render slots, collapsing their meshes to the origin. In addition, discovered vertices are baked into the deformable root's parent frame but are instantiated using the root pose, double-applying non-identity root transforms.

Significant concerns. Posted 3 actionable findings inline.

Conservative automated review; human maintainers own approval decisions.

[huidongc]

@isaaclab-review-bot review

[huidongc]

@isaaclab-review-bot review

[huidongc]

@isaaclab-review-bot review

[isaaclab-review-bot]

Isaac Lab Review Bot

The SceneData geometry contract and dual sim/visual shadow-particle design provide a coherent path for PhysX/OVPhysX deformables to drive Newton Warp and OVRTX. Before merge, the per-frame sync metadata churn should be removed, and wildcard-grouped deformables must not share barycentric remaps unless their topology is verified compatible.

• Design and architecture: Keeping SceneData simulation-true while the Newton shadow model owns visualization-sized render slots is a sound separation. However, grouping deformables solely by environment-wildcard paths assumes matching simulation and visual topology across environments; heterogeneous entries can consequently reuse incompatible remap tables and particle counts. Grouping must include topology characteristics or retain per-entry remaps.
• API: The additive SceneData points, geometry-path, and geometry-count interfaces have safe backend defaults and corresponding documentation and changelog coverage. The accepted concerns are internal implementation and grouping behavior rather than the public SceneData contract.
• Implementation: The discovery-to-render path correctly separates simulation and visualization buffers, but its hot path has avoidable costs: geometry scatter performs host synchronization, repeated metadata uploads, and a quadratic destination scan each frame, while identity mappings invalidate and rebuild batched remap/copy metadata every frame. Additionally, sharing a template remap across wildcard-grouped entries can produce invalid barycentric and tet indexing when their topologies differ.

Significant concerns. Posted 3 actionable findings inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🟡 Warning · Implementation — Per-frame host scan in points scatter

get_points calls this on every render frame, and each call performs mapping.numpy() (a device sync), a linear scan of positive_dests per entity (quadratic in deformable body count, i.e. envs × assets), and three fresh Warp uploads. Precompute the source/destination offset and count device arrays once and rebuild only when entity_counts or the mapping change; use bisect over the sorted offsets.

[isaaclab-review-bot]

🟡 Warning · Implementation — Batch sync cache invalidated on identity mappings

Identity layouts make create_geometry_mapping return None, so _scene_data_geometry_mapping stays None and this block re-runs every render frame, calling _invalidate_shadow_deformable_batch_sync(). _ensure_shadow_deformable_batch_sync then rebuilds per-particle Python id lists and several Warp arrays every frame, defeating the sync_key cache it was added for. The sync_key comparison already detects real layout changes, so drop the unconditional invalidation here.

[isaaclab-review-bot]

🟡 Warning · Implementation — Shared remap assumes identical group topology

Wildcard groups are keyed only on env-rewritten root/sim/vis paths, then every member reuses the template's remap and entities[0].vis_particle_count. If two environments share prim names but differ in tet or visual vertex counts (heterogeneous multi-asset spawning), the batch kernel indexes bary_weights[local_vis] beyond the template rows and applies template tet indices to a different sim slice. Include per-entry vertex counts in the grouping key, or build a remap per entry.

[daniela-hase]

The Newton and Scene Data Provider changes seem good to me :)