Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?


Failed to load latest commit information.
Latest commit message
Commit time

ATLAS: End-to-End 3D Scene Reconstruction from Posed Images

Project Page | Paper | Video | Models | Sample Data

Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich


We provide a Colab Notebook to try inference.


We provide a docker image Docker/Dockerfile with all the dependencies.

Or you can install them yourself:

conda install -y pytorch=1.5.0 torchvision=0.6.0 cudatoolkit=10.2 -c pytorch
conda install opencv
pip install \
  open3d>= \
  trimesh>=3.7.6 \
  pyquaternion>=0.9.5 \
  pytorch-lightning>=0.8.5 \
python -m pip install detectron2 -f

For 16bit mixed precision (default training setting) you will also need NVIDIA apex

git clone
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex

For headless rendering with pyrender (used for evaluation) see installation instructions here.

For inference with COLMAP see installation instructions here.

(If you have problems running the code try using the exact versions specified... for example the pytorch-lightning API has not settled yet).

Data Preperation


We provide a small sample scene for easy download and rapid inference. Download and extract the data to DATAROOT. The directory structure should look like:

│   └───sample1
│       │   intrinsics.txt
│       └───color
│       │   │   00000001.jpg
│       │   │   00000002.jpg
│       │   │   ...
│       └───pose
│           │   00000001.txt
│           │   00000002.txt
│           │   ...

Next run our data preperation script which parses the raw data format into our common json format (more info here) (note that we store our derivered data in a seperate folder METAROOT to prevent pollution of the original data).

python --path DATAROOT --path_meta METAROOT --dataset sample


Download and extract Scannet by following the instructions provided at You also need to download the train/val/test splits and the label mapping from (Benchmark Tasks). The directory structure should look like:

│   └───scans
│   |   └───scene0000_00
│   |       └───color
│   |       │   │   0.jpg
│   |       │   │   1.jpg
│   |       │   │   ...
│   |       │   ...
│   └───scans_test
│   |       └───color
│   |       │   │   0.jpg
│   |       │   │   1.jpg
│   |       │   │   ...
│   |       │   ...
|   └───scannetv2-labels.combined.tsv
|   └───scannetv2_test.txt
|   └───scannetv2_train.txt
|   └───scannetv2_val.txt

Next run our data preperation script which parses the raw data format into our common json format (more info here) (note that we store our derivered data in a seperate folder METAROOT to prevent pollution of the original data). This script also generates the ground truth TSDFs using TSDF Fusion.

python --path DATAROOT --path_meta METAROOT --dataset scannet

This will take a while (a couple hours on 8 Quadro RTX 6000's)... if you have multiple gpus you can use the --i and --n flags to run in parallel

python --path DATAROOT --path_meta METAROOT --dataset scannet --i 0 --n 4 &
python --path DATAROOT --path_meta METAROOT --dataset scannet --i 1 --n 4 &
python --path DATAROOT --path_meta METAROOT --dataset scannet --i 2 --n 4 &
python --path DATAROOT --path_meta METAROOT --dataset scannet --i 3 --n 4 &

Note that if you do not plan to train you can prepare just the test set using the --test flag.

Your own data

To use your own data you will need to put it in the same format as the sample data, or implement your own version of something like After that you can modify to also prepare your data. Note that the pretrained models are trained with Z-up metric coordinates and do not generalize to other coordinates (this means that the scale and 2 axes of the orientation ambiguity of SFM must be resolved prior to using the poses).


Once you have downloaded and prepared the data (as described above) you can run inference using our pretrained model (download) or by training your own (see below).

To run on the sample scene use:

python --model results/release/semseg/final.ckpt --scenes METAROOT/sample/sample1/info.json

If your GPU does not have enough memory you can reduce voxel_dim (at the cost of possible clipping the scene)

python --model results/release/semseg/final.ckpt --scenes METAROOT/sample/sample1/info.json --voxel_dim 208 208 80

Note that the values of voxel_dim must be divisible by 8 using the default 3D network.

Results will be saved to:

results/release/semseg/test_final/sample1.ply // mesh
results/release/semseg/test_final/sample1.npz // tsdf
results/release/semseg/test_final/sample1_attributes.npz // vertex semseg

To run on the entire Scannet test set use:

python --model results/release/semseg/final.ckpt


After running inference on Scannet you can run evaluation using:

python --model results/release/semseg/test_final/

Note that uses pyrender to render depth maps from the predicted mesh for 2D evaluation. If you are using headless rendering you must also set the enviroment variable PYOPENGL_PLATFORM=osmesa (see pyrender for more details).

You can print the results of a previous evaluation run using

python --model results/release/semseg/test_final/


In addition to downloadinng and prepareing the data (as described above) you will also need to download our pretrained resnet50 weights (ported from detectron2) and unnzip it.

Then you can train your own models using

Configuration is controlled via a mix of config.yaml files and command line arguments. We provide a few sample config files used in the paper in configs/. Experiment names are specified by TRAINER.NAME and TRAINER.VERSION, which default to atlas and default. See for a full list of parameters.

python --config configs/base.yaml TRAINER.NAME atlas TRAINER.VERSION base
python --config configs/semseg.yaml TRAINER.NAME atlas TRAINER.VERSION semseg

To watch training progress use

tensorboard --logdir results/

COLMAP Baseline

We also provide scripts to run inference and evaluataion using COLMAP. Note that you must install COLMAP (which is included in our docker image).

For inference on the sample scene use

python --pathout results/colmap --scenes METAROOT/sample/sample1/info.json

and for Scannet

python --pathout results/colmap

To evaluate Scannet use

python --pathout results/colmap


  title={Atlas: End-to-End 3D Scene Reconstruction from Posed Images},
  author={Zak Murez and 
          Tarrence van As and 
          James Bartolozzi and 
          Ayan Sinha and 
          Vijay Badrinarayanan and 
          Andrew Rabinovich},
  booktitle = {ECCV},
  year      = {2020},
  url       = {}