-
Notifications
You must be signed in to change notification settings - Fork 15
sim-ready pipeline #271
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
sim-ready pipeline #271
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
14699c1
first version of sim_ready
XuanchaoPENG 06de01b
clean code
XuanchaoPENG 6148c21
clean code
XuanchaoPENG d44696b
Merge branch 'main' into xuanchao/sim_ready
yuecideng 8624b49
add bake and remesh
XuanchaoPENG d57e0ae
fix review bugs
XuanchaoPENG 9b8409d
change decimate_ratio to keep robust
XuanchaoPENG 0514d2f
run black command
XuanchaoPENG 11630e2
updata sim_ready config/create intro/update install config
XuanchaoPENG 856d071
add sim_ready pipeline
XuanchaoPENG ce0b37a
translate chinese to english
XuanchaoPENG 0cac1ca
add cli commands
XuanchaoPENG 8a40566
Merge branch 'main' into xuanchao/sim_ready
XuanchaoPENG f1afa96
run black command
XuanchaoPENG c66bd8b
Merge branch 'xuanchao/sim_ready' of github.com:DexForce/EmbodiChain …
XuanchaoPENG babb897
update bpy and pyrender install
XuanchaoPENG 3c0d7c8
move simready from toolkit to gensim and update relative docs
XuanchaoPENG 2c97592
delete bpy test
XuanchaoPENG 07ee920
add openai compatible api support and update the related docs and upd…
XuanchaoPENG 41f0019
refine the params in doc and api format
XuanchaoPENG File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| Generative Simulation | ||
| ===================== | ||
|
|
||
| Generative Simulation collects EmbodiChain features for generating simulation-ready assets and executing agent-driven task workflows. | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 2 | ||
|
|
||
| SimReady Asset Pipeline <simready_pipeline.md> |
224 changes: 224 additions & 0 deletions
224
docs/source/features/generative_sim/simready_pipeline.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,224 @@ | ||
| # SimReady Asset Pipeline | ||
|
|
||
| The SimReady asset pipeline converts raw mesh archives into normalized simulation assets. It ingests a source mesh, preserves or bakes visual materials, cleans mesh topology, estimates real-world scale and semantics with multimodal LLMs, and exports assets that can be loaded directly in EmbodiChain simulations. | ||
|
|
||
| ## Quick Start | ||
|
|
||
| Run the pipeline on a single asset directory: | ||
|
|
||
| ```bash | ||
| python -m embodichain.gen_sim.simready_pipeline.cli.start \ | ||
| --input_dir /path/to/raw_mesh_folder \ | ||
| --output_root /path/to/output_folder \ | ||
| --category YourCategory | ||
| ``` | ||
|
|
||
| Preview the generated SimReady mesh: | ||
|
|
||
| ```bash | ||
| python -m embodichain preview-asset \ | ||
| --asset_path /path/to/sim_ready_asset_or_usd_asset \ | ||
| --asset_type rigid | ||
| ``` | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| The full pipeline uses Blender, trimesh, pyrender, and an OpenAI-compatible multimodal chat completions endpoint. Install EmbodiChain with the `gensim` extra and enable both the EmbodiChain package index and Blender package index. | ||
|
|
||
| Install from PyPI with `uv`: | ||
|
|
||
| ```bash | ||
| uv pip install "embodichain[gensim]" \ | ||
| --extra-index-url http://pyp.open3dv.site:2345/simple/ \ | ||
| --trusted-host pyp.open3dv.site \ | ||
| --extra-index-url https://download.blender.org/pypi/ | ||
| ``` | ||
|
|
||
| Install from source with `uv`: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/DexForce/EmbodiChain.git | ||
| cd EmbodiChain | ||
| uv pip install -e ".[gensim]" \ | ||
| --extra-index-url http://pyp.open3dv.site:2345/simple/ \ | ||
| --trusted-host pyp.open3dv.site \ | ||
| --extra-index-url https://download.blender.org/pypi/ | ||
| ``` | ||
|
|
||
| Install from PyPI with `pip`: | ||
|
|
||
| ```bash | ||
| pip install "embodichain[gensim]" \ | ||
| --extra-index-url http://pyp.open3dv.site:2345/simple/ \ | ||
| --trusted-host pyp.open3dv.site \ | ||
| --extra-index-url https://download.blender.org/pypi/ | ||
| ``` | ||
|
|
||
| Install from source with `pip`: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/DexForce/EmbodiChain.git | ||
| cd EmbodiChain | ||
| pip install -e ".[gensim]" \ | ||
| --extra-index-url http://pyp.open3dv.site:2345/simple/ \ | ||
| --trusted-host pyp.open3dv.site \ | ||
| --extra-index-url https://download.blender.org/pypi/ | ||
| ``` | ||
|
|
||
| Set the OpenAI-compatible LLM api(OpenAI, Gemini, Doubao, etc.) before running the pipeline, or configure them in `embodichain/gen_sim/simready_pipeline/configs/gen_config.json`. Environment variables override the JSON config. | ||
|
|
||
| OpenAI-compatible API example: | ||
|
|
||
| ```bash | ||
| export OPENAI_API_KEY="your-openai-api-key" | ||
| export OPENAI_MODEL="gpt-4o" | ||
| export OPENAI_BASE_URL="https://api.openai.com/v1" | ||
| ``` | ||
|
|
||
| ## Processing Flow | ||
|
|
||
| The command above runs the full parser sequence: | ||
|
|
||
| - **Ingest**: finds the first parseable mesh (`.glb`, `.gltf`, `.obj`, `.ply`, `.stl`), archives the raw input, and writes a canonical `asset_source/asset.obj`. | ||
| - **Visual processing**: by default, Blender remeshes the source mesh, unwraps UVs, and bakes diffuse and normal textures. With `--simple`, ingest uses trimesh only and skips Blender remesh/bake. | ||
| - **Inspection**: detects whether the normalized source is a mesh, articulation, or scene. | ||
| - **Geometry processing**: cleans topology and applies Blender decimation to the canonical mesh. | ||
| - **SimReady finalization**: renders multi-view images, uses the LLM to infer object orientation, physical dimensions, and semantics, then exports `asset_simready/asset_simready.obj`. | ||
| - **Physics and USD export**: infers physics properties and writes a USD package when possible. | ||
| - **Internal preview assets**: generates thumbnails and internal metadata for asset browsing. | ||
|
|
||
| ## Output Layout | ||
|
|
||
| Each processed asset is written under a generated asset ID: | ||
|
|
||
| ```text | ||
| simready_car/ | ||
| `-- <asset_id>/ | ||
| |-- asset_archive/ # Raw source directory copy | ||
| |-- asset_source/ # Canonical normalized source mesh and textures | ||
| | |-- asset.obj | ||
| | |-- asset.mtl | ||
| | |-- diffuse.png | ||
| | `-- normal.png | ||
| |-- asset_simready/ # Final oriented and scaled mesh | ||
| | `-- asset_simready.obj | ||
| |-- asset_usd/ # USD export | ||
| `-- asset.json # Metadata, geometry, semantics, physics, and paths | ||
| ``` | ||
|
|
||
| Use `asset_simready/asset_simready.obj` or `asset_usd/` for simulation preview and downstream scene construction. | ||
|
|
||
| ## Command-Line Arguments | ||
|
|
||
| | Argument | Description | Default | | ||
| | :--- | :--- | :--- | | ||
| | `--input_dir` | Directory containing the raw asset files. | **required** | | ||
| | `--output_root` | Directory where processed assets are written. | **required** | | ||
| | `--category` | Category hint passed into the pipeline, such as `car`, `bowl`, or `chair`. | **required** | | ||
| | `--simple` | Use trimesh-only ingest and skip Blender remesh/bake during ingest. Geometry cleanup later in the pipeline still uses Blender. | `False` | | ||
|
|
||
| ## Configuration | ||
|
|
||
| Pipeline hyperparameters live in `embodichain/gen_sim/simready_pipeline/configs/gen_config.json`. The main hyperparameters are as follow: | ||
|
|
||
| ### Mesh Processing | ||
|
|
||
| ```json | ||
| "mesh_processing": { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe remove the parameters that would never be changed |
||
| "blender_remesh_bake": { | ||
| "remesh": { | ||
| "voxel_size": 0.01, | ||
| "min_voxel_size_ratio": 0.005, | ||
| "use_smooth_shade": true | ||
| }, | ||
| "decimate": { | ||
| "ratio": 0.9 | ||
| }, | ||
| "uv": { | ||
| "angle_limit": 66.0, | ||
| "island_margin": 0.02 | ||
| }, | ||
| "bake": { | ||
| "texture_size": 2048, | ||
| "cage_extrusion_ratio": 0.05 | ||
| } | ||
| }, | ||
| "blender_cleanup_decimate": { | ||
| "enabled": true, | ||
| "cleanup": { | ||
| "merge_dist": 0.00001, | ||
| "remove_non_manifold": true, | ||
| "triangulate": false | ||
| }, | ||
| "simplify": { | ||
| "ratio": 0.5, | ||
| "weld_distance": 0.0001, | ||
| "collapse_triangulate": true | ||
| } | ||
| }, | ||
| } | ||
| ``` | ||
|
|
||
| `blender_remesh_bake` controls the default ingest path when `--simple` is not provided. It remeshes the raw mesh, decimates it, unwraps UVs, and bakes textures. | ||
|
|
||
| `blender_cleanup_decimate` controls the later geometry parser stage. It uses Blender mesh operators and the Blender Decimate modifier to clean and simplify the canonical mesh. | ||
|
|
||
|
|
||
| ### LLM | ||
|
|
||
| ```json | ||
| "llm": { | ||
| "openai_compatible": { | ||
| "api_key": "", | ||
| "model": "gpt-4o", | ||
| "base_url": "https://api.openai.com/v1", | ||
| "default_query": {} | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| This section configures the multimodal LLM used for object classification, orientation selection, dimension inference, semantic annotation, and physics inference. Any provider that supports the OpenAI-compatible chat completions API can be used by changing `api_key`, `model`, `base_url`, and optional `default_query` parameters. | ||
|
|
||
| For Azure-style OpenAI-compatible endpoints that require an API version query parameter, use `default_query`: | ||
|
|
||
| ```json | ||
| "llm": { | ||
| "openai_compatible": { | ||
| "api_key": "your-api-key", | ||
| "model": "gpt-4o", | ||
| "base_url": "your_api", | ||
| "default_query": { | ||
| "api-version": "2025-01-01-preview" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Default vs Simple Ingest | ||
|
|
||
| The default command uses Blender during ingest: | ||
|
|
||
| ```bash | ||
| python -m embodichain.gen_sim.simready_pipeline.cli.start \ | ||
| --input_dir /path/to/raw_mesh_folder \ | ||
| --output_root /path/to/output_folder \ | ||
| --category YourCategory | ||
| ``` | ||
|
|
||
| Use `--simple` when you want faster trimesh-only ingest: | ||
|
|
||
| ```bash | ||
| python -m embodichain.gen_sim.simready_pipeline.cli.start \ | ||
| --input_dir /path/to/raw_mesh_folder \ | ||
| --output_root /path/to/output_folder \ | ||
| --category YourCategory \ | ||
| --simple | ||
| ``` | ||
|
|
||
| The simple mode only affects the ingest step. The downstream geometry parser still uses Blender cleanup and decimation unless `mesh_processing.blender_cleanup_decimate.enabled` is set to `false`. | ||
|
|
||
| ## See Also | ||
|
|
||
| - [Asset Preview](../interaction/preview_asset.md): Load generated meshes and USD assets in the simulator. | ||
| - [Installation](../../quick_start/install.md): Install EmbodiChain with Blender and rendering dependencies. | ||
| - [Toolkits](../toolkits/index.rst): Other asset preparation utilities. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add support for openai and gemini interfaces.