D2GS: Dense Depth Regularization for LiDAR-free Urban Scene Reconstruction

Official research code for the NeurIPS 2025 paper D2GS: Dense Depth Regularization for LiDAR-free Urban Scene Reconstruction.

D2GS reconstructs dynamic urban scenes from camera input without using ground-truth LiDAR supervision. It starts from metric depth estimates, prunes the resulting dense Gaussian initialization, alternates Gaussian optimization with a diffusion-based Depth Enhancer, and applies explicit road-plane priors.

Method-to-code map

Paper component	Primary implementation	Configuration
Metric-depth initialization	depthsplat/; loading in datasets/base/pixel_source.py and datasets/waymo/waymo_sourceloader.py	configs/datasets/waymo/3cams_waymo_nvs_smpl.yaml
Progressive Pruning	warm-up orchestration in tools/train_online_pooling_proj_dcgs_emphDloss_dense.py; Gaussian culling in models/gaussians/vanilla.py	d2gs.progressive_pruning and trainer.gaussian_ctrl_general_cfg
Depth Enhancer	Marigold_DC/pooling_proj_marigold_dc.py	d2gs.depth_enhancer
Dense depth supervision	models/losses.py and models/trainers/base.py	trainer.losses.mono_depth
Road Node	models/nodes/road.py and models/trainers/scene_graph.py	model.RoadNodes and road losses under trainer.losses
Training orchestration	tools/train_online_pooling_proj_dcgs_emphDloss_dense.py	configs/dcgs_onlineDC_nvs_emphDloss.yaml
Rendering evaluation	tools/our_eval.py	saved run config.yaml

The release contains only the canonical three-camera Waymo NVS setup in configs/dcgs_onlineDC_nvs_emphDloss.yaml. Internal ablations, alternative dataset adapters, and machine-specific launch scripts are intentionally omitted.

Repository layout

configs/          canonical model, schedule, and Waymo dataset configuration
datasets/         Waymo loader and shared dataset primitives
models/           Gaussian representations, scene graph, trainers, and losses
Marigold_DC/      D2GS diffusion-based Depth Enhancer
depthsplat/       vendored metric multi-view depth initializer
tools/            canonical training, rendering, and evaluation entrypoints
third_party/smplx vendored SMPL-X Python package
data/car_para/    camera calibration used by novel-view evaluation
tests/            lightweight depth-metric regression tests

Installation

The Gaussian training code and DepthSplat use different dependency stacks. Use separate environments so that changing the DepthSplat PyTorch version cannot change the Gaussian training environment.

1. Clone the repository

git clone <repository-url> d2gs
cd d2gs

2. Gaussian training and Depth Enhancer environment

Python 3.10 is recommended. Install a CUDA-compatible PyTorch build first, then:

pip install -r requirements.txt
pip install git+https://github.com/nerfstudio-project/gsplat.git@v1.3.0
pip install git+https://github.com/facebookresearch/pytorch3d.git
pip install git+https://github.com/NVlabs/nvdiffrast
pip install -e third_party/smplx

The exact PyTorch/CUDA build is intentionally not installed by requirements.txt; it must match the local driver and the compiled gsplat, PyTorch3D, and nvdiffrast extensions. Record the resolved environment before a paper run, for example with pip freeze > environment.txt.

Download the Marigold depth checkpoint into:

Marigold_DC/marigold-depth-v1-0/

or pass another local directory/model identifier with --depth-enhancer-checkpoint.

3. DepthSplat environment

Follow depthsplat/README.md and download its pretrained weight. The paper uses:

depthsplat-gs-base-dl3dv-256x448-randview2-6-d94d996f.pth

Generate one metric depth map per image and back-project those maps into the pseudo-LiDAR files consumed by the Gaussian loader.

From the separate DepthSplat environment, the canonical export is:

cd depthsplat
python export_d2gs.py \
  --data-root <data-root> \
  --scenes <scene-id> \
  --checkpoint pretrained/depthsplat-gs-base-dl3dv-256x448-randview2-6-d94d996f.pth \
  --camera-ids 0 1 2 \
  --start-timestep 0 \
  --end-timestep 50 \
  --window-size 10
cd ..

This writes ab_depths/ and the camera-derived pseudo-point cloud under lidar/ in each selected scene. See depthsplat/README.md for filtering, offline DINOv2, and alternate-output options.

Data preparation

Start from an OmniRe/DriveStudio-style processed Waymo scene. For timesteps 000 through 049, the D2GS paths used by the main entrypoint are:

<data-root>/<scene-id>/
  images/                 000_0.jpg, 000_1.jpg, ...
  intrinsics/             0.txt, 1.txt, 2.txt
  extrinsics/             0.txt, 1.txt, 2.txt
  ego_pose/               000.txt, 001.txt, ...
  ab_depths/              000_0.pt, 000_1.pt, ...
  lidar/                  000.bin, 001.bin, ...  # pseudo-LiDAR from metric depth
  sky_masks/
  road_masks/
  dynamic_masks/
  instances/

Although the loader directory is named lidar/, the canonical LiDAR-free run expects the pseudo point cloud generated from metric image depth, not ground-truth LiDAR. Do not mix those files with the GT LiDAR used only for final depth evaluation.

Set the dataset root either in configs/datasets/waymo/3cams_waymo_nvs_smpl.yaml or as the final OmegaConf override shown below.

Training

The portable single-scene wrapper is:

bash tools/run_d2gs.sh <scene-id> <data-root> [output-root] [marigold-checkpoint]

Equivalent direct command:

export PYTHONPATH="$(pwd)"
python tools/train_online_pooling_proj_dcgs_emphDloss_dense.py \
  --config-file configs/dcgs_onlineDC_nvs_emphDloss.yaml \
  --scene-id <scene-id> \
  --output-root outputs \
  --project d2gs \
  --run_name <scene-id> \
  --depth-enhancer-checkpoint Marigold_DC/marigold-depth-v1-0 \
  --enable_tensorboard \
  data.data_root=<data-root>

Outputs are written to outputs/d2gs/<scene-id>/. The final checkpoint is checkpoint_final.pth; the fully resolved configuration is saved beside it as config.yaml.

The training stages are explicit in the canonical YAML:

1. metric-depth Gaussian initialization;
2. three progressive-pruning warm-up rounds plus a final refinement stage;
3. main Gaussian optimization;
4. online Depth Enhancer updates every 20 steps during the configured window;
5. continued dense-depth supervision between enhancer updates.

--DC_step and --pretrain remain accepted by old scripts but are legacy no-ops. Use d2gs.progressive_pruning and d2gs.depth_enhancer overrides for schedule changes.

Evaluation

Render the training/test views and compute image metrics:

python tools/our_eval.py \
  --model_path outputs/d2gs/<scene-id>/checkpoint_final.pth \
  --sensor_list FrontCam02 SideFrontCam01 SideFrontCam02 \
  --disable_novel_view

Omit --disable_novel_view to run novel-view rendering. The default calibration directory is data/car_para/13878C_20241009; override it with --cam-para-path when needed.

For multiple completed runs:

python tools/our_eval_all.py --root-path outputs/d2gs --disable_novel_view

Depth metrics against held-out GT LiDAR:

python eval_depth.py \
  --gt-folder <gt-lidar-depth-directory> \
  --pred-folder outputs/d2gs/<scene-id>/images/original/depths

Add --pt for .pt predictions. Keep GT LiDAR outside the training scene directory so it cannot be accidentally consumed as initialization or supervision.

Utilities and checks

Export Gaussian nodes from a checkpoint:

python tools/export_ply.py <checkpoint.pth> <output.ply> \
  --gaussian-keys Background RoadNodes

Convert named camera calibration JSON for the novel-view evaluator:

python utils/convert_cam_json.py <cameras.json> <converted.json>

Run the lightweight CPU regression tests and a repository-wide syntax check:

python -m unittest discover -s tests -v
python -m compileall -q .

Acknowledgements

This code builds on OmniRe/DriveStudio, DepthSplat, Marigold-DC, gsplat, and the vendored third-party projects. See their repositories and license files for attribution and usage terms.

License

See LICENSE. Third-party and pretrained-model components may use different licenses.