Reuse existing visual material in randomization

[yuecideng]

Description

This PR updates visual-material handling so simulation assets retain the material instances parsed by dexsim and visual randomization can work from those existing materials instead of replacing them with a newly created VisualMaterial template.

For rigid objects and articulations, the reuse path keeps each original per-segment MaterialInst immutable and creates a working instance from the existing material template. Randomization then switches render-body bindings between the original instance and the working instance. This preserves the asset's authored appearance while avoiding repeated material creation, texture upload, and global clean_materials() calls.

Key changes

• Capture and wrap asset-provided materials when rigid, articulated, soft, and cloth objects are constructed.
• Add material restoration APIs and restore original per-segment assignments during object reset.
• Add get_existing_visual_material() and apply_render_material_inst() for rigid objects and articulations.
• Extend VisualMaterialInst.set_base_color_texture() to bind a pre-created dexsim Texture directly.
• Rework randomize_visual_material around three selectable tiers:
  • original: reattach the asset's original material instances.
  • library: bind a preloaded library texture to the working instance.
  • solid: bind a color from a bounded, pre-created texture palette.
• Add p_original, p_library, p_solid, solid_texture_count, and shared configuration.
• Cache uploaded library textures and solid-color textures at simulation scope.
• Support partial env_ids, multi-segment render bodies, and independent per-link tier sampling for articulations.
• Keep fallback_to_new=True as the explicit legacy create-and-replace path; reuse failures also fall back automatically.
• Keep default_plane randomization in place without cleaning all materials.
• Add API and overview documentation for material instances, restoration, reuse semantics, and randomization parameters.

Compatibility

This is a non-breaking enhancement. Existing configurations that only use random_texture_prob continue to work: when all p_* values are omitted, the probabilities resolve to p_original=0, p_library=random_texture_prob, and p_solid=1-random_texture_prob. If no texture library is loaded, the library probability is folded into the solid tier.

Dependencies: none; this uses existing dexsim material and texture APIs.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Enhancement (non-breaking change which improves an existing functionality)
• New feature (non-breaking change which adds functionality)
• Breaking change (existing functionality will not work without user modification)
• Documentation update

Screenshots

N/A — this changes simulation material lifecycle and randomization behavior rather than UI.

Validation

• black --check . — 460 files unchanged.
• Material reuse, randomization, texture, asset initialization, and simulation-manager tests — 65 passed.

Checklist

• I have run the black . command to format the code base.
• I have made corresponding changes to the documentation.
• I have added tests that prove my fix is effective or that my feature works.
• Dependencies have been updated, if applicable (not applicable; no dependency changes).

[Copilot]

Pull request overview

This PR refactors randomize_visual_material to reuse dexsim-parsed per-object materials (via instance swapping) instead of creating and replacing a new material each reset, adds a three-tier texture selection policy (original/library/solid) with configurable probabilities, and keeps the prior behavior behind a fallback_to_new flag. It also introduces supporting sim APIs (ReuseSegmentState, “get existing material” helpers) and adds mocked unit tests to validate the new code paths.

Changes:

• Extend VisualMaterialInst.set_base_color_texture(...) to accept a pre-created dexsim Texture object (skip re-upload).
• Add ReuseSegmentState plus get_existing_visual_material(...) / apply_render_material_inst(...) helpers on RigidObject and Articulation.
• Implement reuse-mode init/call paths in randomize_visual_material, including tier-probability resolution and sim-level texture object caching; add extensive mocked tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
embodichain/lab/gym/envs/managers/randomization/visual.py	Implements reuse-mode initialization and per-reset swapping, tier sampling, and cached texture objects; retains legacy path behind flag.
embodichain/lab/sim/material.py	Adds texture_obj support for base-color maps and introduces ReuseSegmentState.
embodichain/lab/sim/objects/rigid_object.py	Adds helpers to capture existing segment materials and swap MaterialInst back onto render bodies.
embodichain/lab/sim/objects/articulation.py	Adds link-aware equivalents of existing-material capture and render-body instance swapping.
embodichain/lab/sim/__init__.py	Exports ReuseSegmentState and defines an explicit package __all__.
tests/gym/envs/managers/test_randomize_visual_material.py	Adds characterization and reuse-mode tests (mocked dexsim) for legacy/reuse/compat scenarios.
tests/sim/test_material_texture_obj.py	Tests texture_obj path binds Texture without upload.
tests/sim/objects/test_rigid_object_reuse_material.py	Tests rigid-object reuse-state building and render-body swapping.
tests/sim/objects/test_articulation_reuse_material.py	Tests articulation reuse-state building per link and swapping per link segment.
docs/superpowers/specs/2026-07-14-reuse-existing-visual-material-design.md	Design write-up for the reuse-material approach and constraints.
docs/superpowers/plans/2026-07-14-reuse-existing-visual-material.md	Detailed implementation plan and checklist for the feature.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

embodichain/lab/sim/material.py:306

• set_base_color_texture() bypasses the VisualMaterialInst.mat property and always calls self._mat.get_inst(self.uid). That ignores the _mat_inst path introduced by from_existing(), so updates may be applied to a looked-up/new instance instead of the wrapped existing MaterialInst.

        if texture_path is not None:
            self.base_color_texture = texture_path
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_path)
        elif texture_obj is not None:
            self.base_color_texture = texture_obj
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_obj)

embodichain/lab/sim/objects/rigid_object.py:956

• get_existing_visual_material() assumes get_render_body() is always non-None and will raise an AttributeError on render_body.get_mesh_count() otherwise. The docstring says this method raises ValueError for invalid reuse state; render-body absence should be handled explicitly and raised as ValueError (so callers like randomize_visual_material can degrade cleanly).

        for env_idx in local_env_ids:
            render_body = self._entities[env_idx].get_render_body()
            mesh_count = render_body.get_mesh_count()
            segments: List[ReuseSegmentState] = []

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

embodichain/lab/sim/material.py:306

• VisualMaterialInst.set_base_color_texture() bypasses the mat property (and therefore _mat_inst from from_existing) by calling self._mat.get_inst(self.uid) directly. This can cause updates to target a different handle than the wrapped existing instance. Use self.mat consistently here.

        if texture_path is not None:
            self.base_color_texture = texture_path
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_path)
        elif texture_obj is not None:
            self.base_color_texture = texture_obj
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_obj)

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

embodichain/lab/sim/material.py:303

• In set_base_color_texture(), the texture-path branch uses self._mat.get_inst(self.uid) which bypasses _mat_inst for wrappers created via from_existing(). This can cause texture changes to target a different instance than the one attached to the asset. Use self.mat here.

        if texture_path is not None:
            self.base_color_texture = texture_path
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_path)
        elif texture_obj is not None:

embodichain/lab/sim/material.py:306

• In set_base_color_texture(), the texture_obj branch uses self._mat.get_inst(self.uid) which bypasses _mat_inst for wrappers created via from_existing(). Use self.mat so the pre-created dexsim Texture is bound onto the actual wrapped instance.

        elif texture_obj is not None:
            self.base_color_texture = texture_obj
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_obj)

embodichain/lab/sim/material.py:310

• In set_base_color_texture(), the texture_data branch uses self._mat.get_inst(self.uid), which bypasses _mat_inst for from_existing() wrappers. Use self.mat so the uploaded texture is applied to the wrapped instance.

        elif texture_data is not None:
            self.base_color_texture = texture_data
            inst = self._mat.get_inst(self.uid)

embodichain/lab/gym/envs/managers/randomization/visual.py:872

• clean_materials() is now only called when fallback_to_new=True. When reuse init fails and the functor degrades to the legacy path, clean_materials() will not run, which can leak per-call uploaded textures/material instances (the legacy path uploads textures via set_base_color_texture(texture_data=...)). Consider cleaning whenever the legacy path is used (except default_plane).

        clean = bool(self._fallback_to_new)
        return self._call_legacy(
            env,
            env_ids,
            random_texture_prob,

embodichain/lab/sim/material.py:246

• VisualMaterialInst.from_existing() introduces _mat_inst and mat property correctly returns it, but several setters still call self._mat.get_inst(self.uid). That bypasses the wrapped existing instance, so calls like set_roughness()/set_metallic() may not affect the material instance actually attached to the asset. These setters should use self.mat consistently (as set_base_color() already does).

    @property
    def mat(self) -> MaterialInst:
        existing_inst = getattr(self, "_mat_inst", None)
        if existing_inst is not None:
            return existing_inst

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

embodichain/lab/gym/envs/managers/randomization/visual.py:1125

• In legacy mode, set_visual_material() is called on every functor invocation for the selected envs. For RigidObject/Articulation, set_visual_material() creates new dexsim material instances each time, which is expensive and can leak/accumulate instances; the material assignment was already done during _init_legacy() and the call should only fetch existing instances for the selected envs.

        selected_env_ids = [int(env_idx) for env_idx in env_ids]
        if isinstance(self.entity, RigidObject):
            self.entity.set_visual_material(self._legacy_mat, env_ids=selected_env_ids)
            self._mat_insts = self.entity.get_visual_material_inst(
                env_ids=selected_env_ids
            )
        elif isinstance(self.entity, Articulation):
            self.entity.set_visual_material(
                self._legacy_mat,
                env_ids=selected_env_ids,
                link_names=self.entity_cfg.link_names,
            )
            self._mat_insts = self.entity.get_visual_material_inst(
                env_ids=selected_env_ids,
                link_names=self.entity_cfg.link_names,
            )

embodichain/lab/gym/envs/managers/randomization/visual.py:1008

• The solid tier currently hard-codes metallic=0.0 and roughness=0.7 and never applies metallic_range/roughness_range/ior_range from the sampled plan. This makes user-provided ranges ineffective whenever the solid tier is selected (including when library textures are missing and probability is folded into solid).

    def _apply_solid_props(self, working_inst, plan, idx) -> None:
        texture_idx = torch.randint(0, len(self._solid_textures), (1,)).item()
        working_inst.set_base_color([1.0, 1.0, 1.0, 1.0])
        working_inst.set_metallic(0.0)
        working_inst.set_roughness(0.7)
        working_inst.set_base_color_texture(
            texture_obj=self._solid_textures[texture_idx]
        )

embodichain/lab/sim/material.py:317

• VisualMaterialInst.set_base_color_texture() uses self._mat.get_inst(self.uid) even when this wrapper was created via from_existing() with a concrete dexsim MaterialInst. That bypasses the mat property (which correctly returns the existing instance) and can apply updates to the wrong instance if name lookup doesn’t resolve to the wrapped handle.

        if texture_path is not None:
            self.base_color_texture = texture_path
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_path)
        elif texture_obj is not None:
            self.base_color_texture = texture_obj
            inst = self._mat.get_inst(self.uid)
            inst.set_base_color_map(texture_obj)
        elif texture_data is not None:
            self.base_color_texture = texture_data
            inst = self._mat.get_inst(self.uid)

            # TODO: Optimize texture creation method.
            world = dexsim.default_world()
            env = world.get_env()
            color_texture = env.create_color_texture(
                texture_data.cpu().numpy(), has_alpha=True
            )
            inst.set_base_color_map(color_texture)

embodichain/lab/sim/material.py:214

• VisualMaterialCfg defaults roughness to 0.7, but VisualMaterialInst initializes self.roughness to 0.5. This makes the wrapper’s stored defaults inconsistent with the config defaults and the docs (and can confuse callers relying on default roughness).

        # Init properties with default values
        self.base_color = [0.5, 0.5, 0.5, 1.0]
        self.metallic = 0.0
        self.roughness = 0.5
        self.emissive = [0.0, 0.0, 0.0]