Introduce runtime scene packages for MuJoCo and Rerun#2594
Merged
Conversation
❌ 1 Tests Failed:
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Nabla7
changed the title
Nabla7
added
the
backport:skip
ChatGPT Codex Connector
Nabla7
requested review from
leshy,
mustafab0,
paul-nechifor and
spomichter
as code owners
github-actions
Bot
added
the
ready-to-merge
Contributor
Greptile SummaryIntroduces a
Confidence Score: 5/5Safe to merge; the threading and scene-lookup regressions flagged in earlier reviews are addressed, and the new runtime paths are well-guarded against missing files and bad physics state. The scene-package contract, composed-model pipeline, and raycast lidar are all implemented defensively: missing artifacts raise early with clear messages, penetrating entities are re-welded rather than ejected, and the multi-waiter reset correctly releases all blocked callers. No data-loss or correctness regressions were found in the changed paths.
Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant CLI as dimos CLI
participant Catalog as scenes/catalog.py
participant Spec as scene_assets/spec.py
participant SimModule as MujocoSimModule
participant Engine as MujocoEngine
participant EntityScene as entity_scene.py
participant Rerun as RerunBridgeModule
participant ScenePkg as rerun/scene_package.py
CLI->>Catalog: resolve_scene_package(name)
Catalog->>Spec: load_scene_package(meta.json)
Spec-->>Catalog: ScenePackage
Catalog->>Catalog: _validate_package_artifacts()
Catalog-->>CLI: ScenePackage
CLI->>SimModule: MujocoSimModule.blueprint(scene_xml, robot_mjcf, entities)
SimModule->>SimModule: _compose_model()
SimModule->>EntityScene: add_entities_to_spec(spec, entities)
EntityScene-->>SimModule: spec with entity bodies
SimModule->>SimModule: spec.compile() to MjModel
SimModule->>EntityScene: spawn_penetrators(model)
EntityScene-->>SimModule: frozenset[penetrator_ids]
SimModule->>EntityScene: "add_entities_to_spec(fresh_spec, force_static=penetrators)"
SimModule->>Engine: "MujocoEngine(model=compiled_model)"
Engine->>Engine: _apply_spawn_pose_unlocked()
loop sim thread
Engine->>Engine: mj_step()
Engine->>Engine: _raycast_lidars() via mj_multiRay
Engine->>Engine: _render_cameras()
end
CLI->>ScenePkg: scene_package_static_entities(scene)
ScenePkg->>Catalog: resolve_scene_package(scene)
ScenePkg-->>Rerun: entity_path to SceneVisualFactory
Rerun->>Rerun: _log_static() logs Transform3D and Asset3D as static
Engine-->>SimModule: RaycastLidarFrame(points)
SimModule->>SimModule: _generate_mujoco_lidar_pointcloud()
SimModule->>Rerun: pointcloud topic to world-frame PointCloud2
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant CLI as dimos CLI
participant Catalog as scenes/catalog.py
participant Spec as scene_assets/spec.py
participant SimModule as MujocoSimModule
participant Engine as MujocoEngine
participant EntityScene as entity_scene.py
participant Rerun as RerunBridgeModule
participant ScenePkg as rerun/scene_package.py
CLI->>Catalog: resolve_scene_package(name)
Catalog->>Spec: load_scene_package(meta.json)
Spec-->>Catalog: ScenePackage
Catalog->>Catalog: _validate_package_artifacts()
Catalog-->>CLI: ScenePackage
CLI->>SimModule: MujocoSimModule.blueprint(scene_xml, robot_mjcf, entities)
SimModule->>SimModule: _compose_model()
SimModule->>EntityScene: add_entities_to_spec(spec, entities)
EntityScene-->>SimModule: spec with entity bodies
SimModule->>SimModule: spec.compile() to MjModel
SimModule->>EntityScene: spawn_penetrators(model)
EntityScene-->>SimModule: frozenset[penetrator_ids]
SimModule->>EntityScene: "add_entities_to_spec(fresh_spec, force_static=penetrators)"
SimModule->>Engine: "MujocoEngine(model=compiled_model)"
Engine->>Engine: _apply_spawn_pose_unlocked()
loop sim thread
Engine->>Engine: mj_step()
Engine->>Engine: _raycast_lidars() via mj_multiRay
Engine->>Engine: _render_cameras()
end
CLI->>ScenePkg: scene_package_static_entities(scene)
ScenePkg->>Catalog: resolve_scene_package(scene)
ScenePkg-->>Rerun: entity_path to SceneVisualFactory
Rerun->>Rerun: _log_static() logs Transform3D and Asset3D as static
Engine-->>SimModule: RaycastLidarFrame(points)
SimModule->>SimModule: _generate_mujoco_lidar_pointcloud()
SimModule->>Rerun: pointcloud topic to world-frame PointCloud2
Reviews (6): Last reviewed commit: "Merge branch 'main' into pim/feat/runtim..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Zero callers in the repo; the runtime path is MujocoSimModule._compose_model, which calls add_entities_to_spec directly on the MjSpec. Drop the now-unused ScenePackage import.
Add reset_runtime_state to the ControlTask Protocol with a no-op default on BaseControlTask, so the coordinator can call it directly instead of probing each task with inspect.signature. All concrete tasks inherit BaseControlTask, so the direct call is uniform. Drops the now-unused import inspect.
The MujocoEngine in test_engine_request_reset_to_applies_pose_in_sim_loop was torn down via a hand-rolled try/finally. Move creation and disconnect into a fixture so teardown runs even on failure.
github-actions
Bot
removed
the
ready-to-merge
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
github-actions
Bot
added
the
ready-to-merge
paul-nechifor
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DimOS has no clean runtime concept for a simulated environment.
Until now, a "scene" was usually a loose set of backend-specific files: MuJoCo XML, meshes, browser assets, object metadata, and local path assumptions. That makes scenes hard to reuse across simulation, visualization, navigation, and future renderers.
This PR introduces scene packages: cooked environments with one metadata file and a stable runtime contract.
Scene Packages
A scene package is a directory with
scene.meta.jsonplus backend-specific artifacts.The metadata describes what the package contains and where each consumer should look. A package can include:
.mjbbinaries for faster loading of large scenesThe core rule is: runtime code resolves a scene package once, then asks the package for the artifact it needs. Consumers should not hardcode scene-specific folders.
Runtime API
Scene inputs are resolved through the catalog/loader:
Supported inputs:
None/--scene noneofficeorsupermarketscene.meta.json.mjbin consumers that support direct binary loadingOnce resolved, consumers use the package object instead of knowing folder layout:
For example:
package.mujoco_scene_path, or a direct.mjbpath when a large scene has been precompiled.package.browser_visual_path("rerun").What This PR Adds
Adds the runtime scene package contract and resolver.
Adds MuJoCo loading for scene packages, including XML scenes, optional
.mjbloading, entity metadata, and precomposed G1 demo scenes for large environments.Adds Rerun support for scene package visuals, G1 URDF robot visualization, raycast lidar, map/costmap/path visualization, and command velocity wiring from the Rerun WASD controls.
Uses
unitree-g1-groot-wbcas the first full integration path. The same blueprint can now run with no scene, the office scene, or the supermarket scene.Ships the required runtime data through LFS:
scene_packages.tar.gz:dimos_office,supermarketg1_urdf.tar.gz: G1 URDF and meshes for Rerun visualizationWhy This Touches Multiple Systems
The reusable piece is the scene package contract.
The extra plumbing is here because the first consumer is a real robot stack, not a metadata-only test. To prove the package works, G1 needs to spawn in the scene, see it with lidar, build a map, plan through it, show the scene and robot in Rerun, reset cleanly, and respond to velocity commands.
Non-Goals
This PR does not add the scene cooking pipeline.
It also does not solve every large-scene problem. Large scenes may need
.mjbartifacts for startup time, and many dynamic objects remain a MuJoCo scaling issue. The package format is designed to support richer dynamic assets, reused prototypes, browser assets, and future render targets such as splats, but this PR focuses on runtime loading and the first end-to-end G1 demo.How To Test
No scene:
Office:
Supermarket:
Prefer headless MuJoCo with Rerun native for normal testing.
mujocosimmodule.headless=falseopens the MuJoCo viewer but can run much slower.Review Guide
Start with:
dimos/simulation/scene_assets/spec.pydimos/simulation/scenes/catalog.pyThen review the consumers:
Checks
Focused checks run locally across the branch included
ruff,py_compile, mypy on touched Rerun scene code, commit hooks, LFS checks, and runtime smoke testing through the G1 MuJoCo/Rerun commands above.Contributor License Agreement