SimFoundry

SimFoundry turns a short real-world video into a physics-ready simulation scene in under an hour, with no manual annotation required. Point it at a tabletop, and it automatically segments every object, reconstructs geometry, generates textured 3D meshes, and compiles the result into an OmniGibson scene complete with physical parameters, digital cousin variations, and task proposals.

Unlike prior scene reconstruction approaches, SimFoundry is fully modular: each stage is an independently swappable component. As foundation models improve, the SimFoundry pipeline improves with them.

News

Date	Update
2026-08-14	🚀 Initial open-source release: V0 rigid-body and articulation generation
Coming Soon	Example scenes and assets
Coming Soon	Automated background generation
Coming Soon	Robotics data generation, training, and evaluation

Requirements

Linux with an NVIDIA GPU and CUDA
Mamba or Conda with mamba
ffmpeg
~250 GB of free disk space for a full install
Hugging Face account
Google Cloud project or Gemini API key

Quick Start

1. Build the conda environments (takes a while) or ask your agent to do with AGENT_INSTALL.md:

bash scripts/installation/install_everything.sh

2. Set up service access. Request access to these gated Hugging Face models (approval can take time):

VLM stages run on Google Cloud Vertex AI (Gemini). Set up a gcloud project with Vertex AI enabled, then authenticate:

export GCLOUD_PROJECT=<your-gcp-project>
gcloud auth application-default login
hf auth login

No GCP project? Generate a Gemini API key at AI Studio and run export GEMINI_API_KEY=<your-key> instead.

Alternatively, run the interactive login helper which covers all services at once:

bash scripts/installation/login_services.sh

3. Download model checkpoints:

bash scripts/installation/download_checkpoints.sh --default

Already logged in to Hugging Face? Fold this into step 1 with bash scripts/installation/install_everything.sh --checkpoints.

4. (Optional) Install the articulation pipeline:

bash scripts/installation/install_articulate.sh

Full installation details: INSTALL.md

Common Examples

Reconstruct a scene from video (example inputs in docs/assets/example_videos/, capture tips in the pipeline README):

bash scripts/pipeline/A_reconstruction/run.sh \
  --scene-name my_scene \
  --video-fpath /path/to/video.mov

The streamed stages budget VRAM as a fraction of the card (90% by default), so this works unchanged on a 24 GiB or a 96 GiB GPU. Add --max-vram-gb N only to pin an absolute cap. On a 24 GiB card, also pass -- s7_mesh.low_vram=true — the default needs ~29 GiB for mesh shape generation at stage 7.

Enable automatic articulation decomposition (requires the optional articulate environments — see INSTALL.md):

bash scripts/pipeline/A_reconstruction/run.sh \
  --scene-name my_scene \
  --video-fpath /path/to/video.mov \
  --detect-articulation

Generate digital cousins, scene variants, and task proposals:

bash scripts/pipeline/B_augmentation/run.sh \
  --scene-name my_scene \
  -- prompt_cousin_structured.max_objects=2 \
       prompt_cousin_structured.max_generated_images_per_object=1

Smoke-test the reconstructed scene in OmniGibson:

bash scripts/pipeline/C_application/run.sh \
  --scene-name my_scene \
  --mode smoke-random

You can also use the unified dispatcher:

scripts/pipeline/run.sh A_reconstruction --help
scripts/pipeline/run.sh B_augmentation --help
scripts/pipeline/run.sh C_application --help

Pipeline Overview

SimFoundry extracts per-object relevant information (segmentation masks, depth, etc.), generates 3D visual meshes via 2D-to-3D generation models, and compiles the final output scene by annotating relevant physical parameters and sanity checking the overall scene configuration in a physics simulator. SimFoundry additionally supports diverse simulated augmentations of objects, scenes, and tasks. SimFoundry's modular design ensures that as individual foundation models improve, the SimFoundry pipeline improves with them.

SimFoundry is organized into three modular pipelines:

Pipeline	Description
A: Reconstruction	Reconstructs a simulation-ready scene from a real video across 13 stages: video processing, depth estimation, ground segmentation, object decomposition, mesh generation, pose estimation, physics compilation, and USD/OmniGibson export.
B: Augmentation	Generates digital cousin variations of the reconstructed objects spanning geometry, topology, and visual appearance, and proposes manipulation tasks for each scene.
C: Application	Loads the scene into OmniGibson for robot policy evaluation, teleoperation data collection, and pipeline smoke testing.

Scene Gallery

Each object in the simulation column was generated fully automatically from a single 2D crop using Hunyuan3D, compiled into a physics-ready simulation scene.

Try it for yourself with Pipeline A.

Real World	SimFoundry Reconstruction
_Bathroom	_{Bathroom Digital Twin}
_{Dining Room}	_{Dining Room Digital Twin}
_{Home Coffee Table}	_{Home Coffee Table Digital Twin}
_Kitchen	_{Kitchen Digital Twin}
_Outdoor	_{Outdoor Digital Twin}
_Toys	_{Toys Digital Twin}

Digital Cousins

Given a reconstructed scene, SimFoundry uses a VLM to propose geometry, topology, and appearance variations of each object, then generates the resulting 3D assets automatically.

This functionality is powered by Pipeline B.

Dining Room	Home Coffee	Toys
Digital Twin	Digital Twin	Digital Twin
Digital Cousins	Digital Cousins	Digital Cousins

Sim-to-Real Policy Training

Policies trained entirely on SimFoundry data transfer zero-shot to real-world tasks. The table below shows simulation evaluation, real-world evaluation, and generalization to unseen digital cousin objects for two robot platforms:

Important

Not yet released. Data generation and policy training code is not included in this repository; the training / data-generation pipeline will ship in a future release.

	Simulated Evaluation	Real World Evaluation	Real World Evaluation (Unseen Objects)
DROID
YAM (Bimanual)

Outputs

Pipeline data is written under Data/<scene_name>/. Key outputs:

Path	Description
`s13_og/reconstructed_og_scene.json`	Final OmniGibson scene descriptor
`s13_og/reconstructed_scene.png`	Scene preview image
`prompt_cousin_structured/`	Digital cousin image proposals
`sim_cousins/` and `usd_cousins/`	Simulation-ready cousin assets
`proposed_tasks/`	Generated task YAMLs
`application_smoke/`	C pipeline smoke-test videos

What's Included

Component	Description
`scripts/pipeline/A_reconstruction/`	13-stage real-to-sim reconstruction pipeline
`scripts/pipeline/B_augmentation/`	Digital cousin generation and task proposal
`scripts/pipeline/C_application/`	OmniGibson scene loading, teleoperation, and evaluation
`scripts/installation/`	Environment and checkpoint installers
`scripts/cfg/`	Hydra config files for all pipeline stages
`simfoundry/`	Core Python library (models, utils, pipeline orchestration)

Documentation

INSTALL.md — full installation and service setup guide
scripts/pipeline/README.md — stage-by-stage pipeline reference
Auto-background README — optional 3D Gaussian Splat background reconstruction

Citation

If you find SimFoundry useful in your research, please cite:

@article{ranawaka2026simfoundry,
  title   = {SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation},
  author  = {Ranawaka, Nadun and Wong, Josiah and Pai, Wei-Lin and Chu, Wei-Teng and
             Dai, Tianyuan and Moghani, Masoud and Yin, Hang and Jiang, Yunfan and
             Durbano, Wesley and Huynh, Brandon and Fang, Yu and Xu, Danfei and
             Zhang, Ruohan and {Fei-Fei}, Li and Fan, Linxi and Wen, Bowen and
             Mandlekar, Ajay and Zhu, Yuke},
  journal = {arXiv preprint arXiv:2606.28276},
  year    = {2026},
}

Acknowledgments

SimFoundry builds on a number of excellent open-source projects. We thank the teams behind OmniGibson, BEHAVIOR-1K, Hunyuan3D-2, Depth Anything 3, SAM3, FoundationPose, and the digital-cousins project, on which portions of this codebase are based.

License

NVIDIA-owned SimFoundry source code is licensed under the Apache License 2.0.

Portions of SimFoundry are derived from the ACDC / digital-cousins project, Copyright (c) 2024 the ACDC authors, also licensed under Apache 2.0. Files containing derived code carry an attribution note in their header.

SimFoundry can optionally download or integrate third-party source code, models, datasets, and SDKs governed by separate terms. The Apache 2.0 license does not apply to those materials. Several optional components are non-commercial, research-only, or otherwise restricted.

See Third-Party Licenses, Third-Party Notices, Patch Provenance, and INSTALL.md for component boundaries.

Contact

For questions or support, reach out to Nadun Ranawaka at nadun.ranawaka@gatech.edu or Ajay Mandlekar at amandlekar@nvidia.com.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
docs		docs
patches		patches
scripts		scripts
simfoundry		simfoundry
tests		tests
third_party_notices		third_party_notices
.gitattributes		.gitattributes
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
PATCH_PROVENANCE.md		PATCH_PROVENANCE.md
README.md		README.md
SECURITY.md		SECURITY.md
THIRD_PARTY_LICENSES.md		THIRD_PARTY_LICENSES.md
THIRD_PARTY_NOTICES.md		THIRD_PARTY_NOTICES.md
install.sh		install.sh
pytest.ini		pytest.ini
requirements.txt		requirements.txt
requirements_dev.txt		requirements_dev.txt
requirements_hunyuan.txt		requirements_hunyuan.txt
requirements_teleop.txt		requirements_teleop.txt
setup.py		setup.py

Navigation Menu

Uh oh!

Repository files navigation

SimFoundry

News

Table of Contents

Requirements

Quick Start

Common Examples

Pipeline Overview

Scene Gallery

Digital Cousins

Sim-to-Real Policy Training

Outputs

What's Included

Documentation

Citation

Acknowledgments

License

Contact

About

Releases

Packages

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

SimFoundry

News

Table of Contents

Requirements

Quick Start

Common Examples

Pipeline Overview

Scene Gallery

Digital Cousins

Sim-to-Real Policy Training

Outputs

What's Included

Documentation

Citation

Acknowledgments

License

Contact

About

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages