Transform 2D images into physics-ready 3D scene assets for robotics training.
MARS (Multi Asset Reconstruction for Simulation) is a complete pipeline that:
- Detects objects using hybrid vision-language models (Qwen 2.5 VL + GroundingDINO)
- Segments objects from images using SAM (Segment Anything Model)
- Reconstructs full 3D geometry and textures using SAM 3D Objects
- Estimates physics properties (mass, friction, inertia)
- Validates scenes with PyBullet physics simulation
- Exports to multiple formats (USD, MJCF, URDF)
- Rich TUI Summary: Pipeline displays a comprehensive summary table at completion showing all detected objects, segmentation results, reconstruction status, physics properties, and validation results
- Prefect Integration: Full workflow orchestration with Prefect, including plain-text logging compatible with Prefect's logging system
- Configurable Models: Support for multiple model variants (Qwen 3B/7B, GroundingDINO tiny/base)
- Intelligent Filtering: NMS-based duplicate removal and configurable area filters with
include_backgroundoption for large objects
Input: A simple 2D image of a real-world setup
Output: Physics-ready 3D scene imported into Isaac Sim
The pipeline automatically detects objects, reconstructs their 3D geometry, estimates physics properties, and exports a complete USD scene ready for robotics simulation.
Image → Detection → Segmentation → 3D Reconstruction → Physics → Validation → Export
(Qwen+DINO) (SAM 3) (SAM 3D Objects)
- Ingestion - Image validation and deduplication
- Detection - Object detection with Qwen 2.5 VL + GroundingDINO (hybrid approach)
- Segmentation - Precise mask generation with SAM 3
- 3D Reconstruction - Mesh generation with SAM 3D Objects + V-HACD collision meshes
- Physics Estimation - Material properties and mass calculation
- Scene Composition - Spatial layout reconstruction
- Validation - PyBullet physics simulation testing
- Storage - Indexing and export to multiple formats
At the end of each pipeline run, MARS displays a comprehensive TUI summary showing:
- Detection Results: Objects detected with confidence scores and bounding boxes
- Segmentation Results: Mask areas, stability scores, and IoU predictions
- Reconstruction Results: Mesh details, collision mesh status, vertex/face counts
- Physics Properties: Mass, friction, restitution for each object
- Validation Results: Stability scores and displacement measurements
See docs/DESIGN.md for detailed setup and usage instructions.
# Build and start container
cd develop
./scripts/build.sh
./scripts/start.sh
# Run test pipeline
./tests/scripts/test_pipeline.shSee docs/QUICKSTART.md for detailed setup and usage instructions.
MARS/
├── develop/ # Docker and development tools
│ ├── Dockerfile
│ ├── docker-compose.yml
│ └── scripts/
│ ├── build.sh
│ ├── start.sh
│ ├── enter.sh
│ ├── stop.sh
│ └── logs.sh
├── src/ # Source code (mounted to container)
│ └── mars/
│ ├── pipeline.py # Main pipeline orchestrator
│ ├── ingestion/ # Stage 1: Image validation
│ ├── detection/ # Stage 2: Object detection (Qwen + DINO)
│ ├── segmentation/ # Stage 3: Mask generation (SAM)
│ ├── reconstruction/ # Stage 4: 3D mesh generation
│ ├── physics/ # Stage 5: Physics estimation
│ ├── composition/ # Stage 6: Scene layout
│ ├── validation/ # Stage 7: PyBullet validation
│ ├── storage/ # Stage 8: Indexing/export
│ ├── exporters/ # Export utilities (USD, MJCF, URDF)
│ └── utils/ # Shared utilities
│ ├── display.py # TUI summary tables (Rich)
│ └── logging_config.py # Prefect-compatible logging
├── config/ # Configuration files
├── tests/ # Test suite
├── assets/ # Example assets and outputs
│ ├── images/ # Test input images
│ └── usd/ # Example USD scene files
└── README.md # This file
The src/ directory is mounted to /workspace/src in the container, so:
- Edit code on your host machine
- Changes are immediately available in the container
- No need to rebuild for code changes
# 1. Edit code on host
vim src/mars/segmentation/segmentation.py
# 2. Test in container
cd develop
./scripts/enter.sh
# Inside container:
python -m pytest /workspace/src/tests/
# 3. Run pipeline
python -m mars.pipeline --input ...cd develop
# Build container
./scripts/build.sh
# Start container
./scripts/start.sh
# Enter container
./scripts/enter.sh
# View logs
./scripts/logs.sh
# Stop container
./scripts/stop.sh
# Rebuild from scratch
./scripts/rebuild.shEdit configuration files in config/ to customize pipeline behavior:
- Model selection:
hybrid(Qwen + GroundingDINO) orqwen_direct - Model variants: Qwen 3B/7B, GroundingDINO tiny/base
- NMS threshold for duplicate removal
- Confidence thresholds
max_area_ratio: Maximum mask area relative to image (default: 0.80)include_background: Keep large objects like tables even if they exceed max_area_ratio- Stability and IoU score thresholds
- V-HACD collision mesh parameters
- Mesh repair settings
- Material properties (density, friction, restitution)
- Category-to-material mapping
- Stage selection
- Output formats
- Logging verbosity
Use the test script to run the pipeline on test images:
./tests/scripts/test_pipeline.sh [image_path] [--run-until STAGE] [--output-dir DIR]See docs/QUICKSTART.md for detailed testing instructions.
Output Contract:
- Scene configs (JSON with meshes, physics, zones)
- Queryable index (task type, objects, quality)
- Multiple export formats (USD, MJCF, URDF)
- Randomization bounds for domain randomization
develop/docs/- Full architecture documentationconfig/- Configuration examplessrc/mars/- Code documentation (docstrings)
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
