Skip to content
Torch implementation for CVPR 18 paper: "LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image"
MATLAB HTML C C++ PostScript Lua Other
Branch: master
Clone or download
Latest commit 1b0f1a6 Oct 10, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data Added a mapping from the rectified pano name, to the image data array… Apr 27, 2018
figs update Mar 26, 2018
gt clarify validation set Apr 22, 2019
matlab update Dec 8, 2018
model update Mar 26, 2018
result update Dec 8, 2018
License.txt update Mar 28, 2018
README.md Update README.md Oct 10, 2019
driver_pano_box.lua update Mar 27, 2018
driver_pano_edg.lua update Mar 27, 2018
driver_pano_full.lua update Mar 27, 2018
driver_pano_joint.lua update Mar 27, 2018
driver_persp_joint_lsun.lua update Jun 14, 2018
driver_persp_joint_lsun_type.lua update Jun 14, 2018
driver_persp_lsun.lua update Jun 14, 2018
model_pano_box.lua update Mar 26, 2018
model_pano_edg.lua update Mar 26, 2018
model_pano_full.lua update Mar 26, 2018
model_pano_joint.lua update Mar 26, 2018
model_persp_joint_lsun.lua update Jul 5, 2018
model_persp_joint_lsun_type.lua update Jul 5, 2018
model_persp_lsun.lua update Jul 5, 2018
model_utils.lua update Mar 26, 2018
preProcess_pano.lua update Jul 18, 2018
testNet_pano_full.lua update Mar 26, 2018
testNet_persp_full_lsun.lua update Dec 8, 2018
train_pano_box.lua update Mar 27, 2018
train_pano_edg.lua update Mar 27, 2018
train_pano_full.lua update Mar 27, 2018
train_pano_joint.lua update Mar 27, 2018
train_persp_joint_lsun.lua update Jun 14, 2018
train_persp_joint_lsun_type.lua update Jun 14, 2018
train_persp_lsun.lua update Jul 5, 2018
weight-init.lua update Mar 26, 2018

README.md

LayoutNet

New: Please check our official PyTorch implementation for LayoutNet v2

Torch implementation for CVPR 18 paper: "LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image"

See sample video of 3D reconstruced layouts by our method.

Third party implementation

Prerequisites

  • Linux
  • NVIDIA GPU + CUDA CuDNN
  • Torch 7

matio: https://github.com/tbeu/matio

  • Matlab

Data

  • Download preprocessed (aligned to horizontal floor plane) training/validation/testing data to current folder

This includes the panoramas from both the panoContext dataset and our labeled stanford 2d-3d dataset.

  • Download groundtruth data to current folder

This includes the groundtruth 2D position of room corners in .mat format from the two dataset. We've corrected some wrong corner labels in PanoContext to match the layout boundaries.

  • Download preprocessed LSUN training/validation/testing data and related .t7 file under /data/LSUN_data/ folder. We've corrected 10% wrong corner labels.

  • We provide training/tesing .t7 samples (selected from PanoContext) to train for general Manhattan layouts prediction. A few sub-samples have non-cuboid room shape ground truth.

Pretrained model

  • Download our pretrained model to current folder. This includes:
  1. The pretrained full approach on the panoContext dataset, the joint boudary and corner prediction branch, the single boundary prediction branch and the 3D layout box regressor;

  2. The pretrained full approach on the LSUN dataset (we've corrected 10% wrong labels), the joint boudary and corner prediction branch and the single boundary prediction branch.

  • The pretrained model for non-cuboid room shape prediction on the panoContext dataset. Using our labeled non-cuboid and cuboid room shape data.

Image preprocess

We provide sample script to extract Manhattan lines and align the panorama in ./matlab/getManhattanAndAlign.m.

To get gt edge map, corner map and box parameters, see sample script ./matlab/preprocessPano.m

To convert gt data to .t7 file, see sample code preProcess_pano.lua

Train network

  • To train our full approach:
th driver_pano_full.lua

Note that this loads the pretrained joint prediction branch and the 3D layout box regressor.

  • To train the joint prediction branch of boudary and corner:
th driver_pano_joint.lua

Note that this loads the pretrained boundary prediction branch.

  • To train the boudary prediction branch:
th driver_pano_edg.lua
  • To train the layout box regressor:
th driver_pano_box.lua

Test network

  • To test on our full approach:
th testNet_pano_full.lua

This saves predicted boundary, corner and 3D layout parameter in "result/" folder.

Optimization

  • To Add Manhattan constraints and optimize for a better layout, open Matlab, then:
cd matlab
panoOptimization.m

This loads saved predictions from the network output and performs sampling.

Evaluation

We provide the Matlab evaluation code for 3D IoU (compute3dOcc_eval.m) and the generation of 2D layout label (getSegMask_eval.m) for evaluating layout pixel accuracy.

Extension to perspective images

  • To train our full approach:
th driver_persp_joint_lsun_type.lua

Note that this loads the pretrained joint corner and boundary prediction branch.

  • To train the joint prediction branch of boudary and corner:
th driver_persp_joint_lsun.lua

Note that this loads the pretrained boundary prediction branch.

  • To train the boudary prediction branch:
th driver_persp_lsun.lua
  • To test the trained network:
th testNet_persp_full_lsun.lua

Note that this saves predicted boundary, corner and room type in "result/" folder. To get the exact 2D corner position on the image, run the following using Matlab:

cd matlab
getLSUNRes.m

You need to download the LSUN data and the toolbox to run through the experiment.

Miscellaneous

  • To get reference for the labeling tool please check panoLabelTool.m.

Citation

Please cite our paper for any purpose of usage.

@inproceedings{zou2018layoutnet,
  title={LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image},
  author={Zou, Chuhang and Colburn, Alex and Shan, Qi and Hoiem, Derek},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2051--2059},
  year={2018}
}
You can’t perform that action at this time.