SceneFactory is a GPU-vectorized platform for procedural scene construction, physics-based multi-agent simulation, and reinforcement learning in autonomous driving environments.
Built on NVIDIA Isaac Sim and Isaac Lab, SceneFactory represents worlds and agents as batched tensors — vehicle control, observations, rewards, resets, and policy inference are all GPU tensor operations.
Paper: SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics (under review)
- 127× throughput over a non-vectorized PhysX baseline — up to 19,250 CASPS at 256 worlds × 16 agents on a single GPU
- Waymo Open Motion Dataset scenes converted to USD road environments offline; diverse topologies loaded at runtime with no code changes
- Articulated PhysX vehicle (10-DOF rigid body) with system-identified dynamics, fully accessible as GPU tensors
- Weather-to-friction module implementing the modified ALL (Average Lumped LuGre) model — maps precipitation + road surface to per-world PhysX friction coefficients
- Bicycle-model backend (
backend: bicycle) for fast prototyping and transfer studies, sharing the same observation/action/reward interface as PhysX - Single YAML hierarchy controls everything: scene pool, parallelism, weather distribution, dynamics backend, and all reward weights
| Component | Version |
|---|---|
| NVIDIA Isaac Sim Full | 5.1.0 |
| Isaac Lab | 0.54.3 |
| RSL-RL | 3.1.2 |
| PyTorch | 2.7.0+cu128 |
| CUDA | 12.8 |
| GPU (tested) | NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM) |
Isaac Sim can be installed as a pip package inside a conda environment (Python 3.11 recommended).
conda create -n scenefactory python=3.11 -y
conda activate scenefactorypip install isaacsim[all,extscache]==5.1.0 --extra-index-url https://pypi.nvidia.com
pip install isaacsim-rl==5.1.0 --extra-index-url https://pypi.nvidia.comNote: The first install pulls ~8 GB of Isaac Sim extensions. Subsequent runs use the cached extensions.
Isaac Lab 0.54.3 is not on PyPI — install it directly from the GitHub repo:
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab
git checkout v0.54.3
pip install -e source/isaaclab
pip install -e source/isaaclab_assets
pip install -e source/isaaclab_tasks
cd ..pip install plotly h5py
pip install rsl-rl-lib==3.1.2git clone https://github.com/[anonymous]/SceneFactory.git
cd SceneFactoryDownload the Waymo Open Motion Dataset (requires accepting the Waymo license).
Place the downloaded *.tfrecord files into:
data/waymo_tfrecords/
Then run the offline extraction script to produce per-scenario JSON files:
Note:
waymo-open-dataset-tf-2-12-0does not support Python 3.11 (see upstream issue). Run this step in a separate Python 3.10 environment:conda create -n waymo-extract python=3.10 -y conda activate waymo-extract pip install tensorflow==2.12.0 waymo-open-dataset-tf-2-12-0 numpy
python scripts/convert_waymo_tfrecord_to_json.py \
--tfrecord-dir data/waymo_tfrecords \
--output-dir data/processed/waymo_scenes_jsonProcessed JSONs are written to data/processed/waymo_scenes_json/, which is already the default scene_json_dir in all provided configs — no further changes needed.
Exercises the full feature stack — sysid-calibrated PhysX vehicle, weather-to-friction module, real Waymo road geometry — at a scale that fits on a single GPU (~8–10 GB VRAM):
bash run_demo_train.shThis runs 32 worlds × 4 agents across four weather conditions (dry / light / moderate / heavy rain) for 500 iterations (~30–60 min on an RTX 3090 / A100).
Logs and checkpoints are saved to logs/rsl_rl/scene_factory_demo/demo_weather_physx/.
To ablate the weather module, add --weather_context_blind true or edit observation.weather_context_blind: true in configs/scene_factory/demo_weather_physx_train.yaml.
Two trained checkpoints are included in checkpoints/:
| File | Description |
|---|---|
checkpoints/v7_weather_aware_iter600.pt |
Weather-aware policy — conditions on friction token; paper Table 3 |
checkpoints/v8_no_weather_iter300.pt |
No-weather baseline — friction token masked; same architecture |
Run all four conditions (v7 vs v8 × dry vs heavy-wet) with one command:
bash run_eval_pretrained.shExpected runtime: ~5 min per condition. Results are written to timestamped directories.
bash run_visualize_scene.sh --world_count 4PYTHONPATH=. python src/train_student_vehicle_goal_multiagent_rsl_rl.py \
--config configs/scene_factory/generated/scene_factory_256scene_random_0414_train_fastgoal_v8_sysid4_noweather.yaml \
--headlessPYTHONPATH=. python src/train_student_vehicle_goal_multiagent_rsl_rl.py \
--config configs/scene_factory/generated/scene_factory_256scene_random_0414_train_fastgoal_v7_sysid4_weather.yaml \
--headlessbash run_bicycle_train.sh --headlessAll paper experiments use the pre-trained checkpoints in checkpoints/. No download required — they are included in this repository.
| Experiment | Script |
|---|---|
| Physics-gap cross-evaluation (Table 2) | bash run_v8_vs_v7_physics_blind_eval.sh |
| Friction conditioning ablation — moderate wet (Table 3) | bash run_v8_vs_v7_moderate_wet_eval.sh |
| Friction conditioning ablation — heavy wet | bash run_v8_vs_v7_heavy_wet_eval.sh |
| Bicycle → PhysX transfer | bash run_bicycle_physx_transfer_eval.sh |
| PhysX → Bicycle transfer | bash run_v8_physx_to_bicycle_transfer_eval.sh |
| Scene diversity ablation (train) | bash run_scene_diversity_ablation.sh |
| Scene diversity ablation (eval) | bash run_scene_diversity_eval.sh |
Summarize 2×2 friction results:
python scripts/summarize_2x2_eval.pyThe pre-fitted sysid parameters are in artifacts/student_vehicle_sysid/comprehensive_fwd_v1_cem_v4/best_config.json and are loaded automatically.
To re-run sysid from scratch:
Step 1 — Generate teacher maneuver programs:
PYTHONPATH=. python -m src.physx_teacher_command_program_generator \
--output-dir artifacts/physx_teacher_programsStep 2 — Record teacher rollouts:
PYTHONPATH=. python -m src.physx_teacher_dataset_builder \
--dataset-dir artifacts/physx_teacher_datasets/comprehensive_fwd_v1 \
--suite sysid-comprehensive-fwd \
--headlessStep 3 — Run CEM fitting:
PYTHONPATH=. python -m src.student_vehicle_sysid \
--headless \
--teacher-dataset-manifest artifacts/physx_teacher_datasets/comprehensive_fwd_v1/manifest.json \
--student-usd artifacts/student_vehicle_assets/vehicle_student/student_fwd_vehicle.usd \
--output-dir artifacts/student_vehicle_sysid/my_run \
--search-mode staged \
--optimizer cemSceneFactory/
├── src/
│ ├── trfc/ # Friction API, lane sampler, obs helpers
│ ├── student_vehicle_multiagent_goal_env.py # Main RL environment
│ ├── scene_factory_multiworld_scene.py # Multi-world USD scene builder
│ ├── scene_factory_obs_contract.py # Observation space
│ ├── scene_factory_late_fusion_actor_critic.py # Policy network
│ ├── scene_factory_config_wizard.py # Config loader
│ ├── procedural_student_vehicle*.py # Vehicle URDF/USD generation
│ ├── student_vehicle_sysid.py # CEM system identification
│ ├── train_student_vehicle_goal_multiagent_rsl_rl.py # Training entry point
│ └── eval_student_vehicle_goal_ppo.py # Eval entry point
├── configs/scene_factory/
│ ├── generated/ # Per-experiment YAML configs
│ └── *.yaml # Canonical base configs
├── artifacts/
│ ├── student_vehicle_assets/vehicle_student/ # URDF + USD + spec
│ └── student_vehicle_sysid/ # Final sysid result
├── scripts/
│ ├── convert_waymo_tfrecord_to_json.py # TFRecord → scene JSON extractor
│ ├── summarize_2x2_eval.py
│ └── generate_weather_eval_configs.py
└── run_*.sh # Paper experiment launchers
Citation information will be provided upon paper acceptance (anonymous submission).
The Waymo Open Motion Dataset is subject to its own license terms. NVIDIA Isaac Sim is used under NVIDIA's non-commercial research license.