No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
datalist Create suncg_trainlist_withcam Jun 27, 2018
image first commit May 12, 2018
matlab_code update May 13, 2018
torch_code update May 13, 2018 update May 13, 2018 first commit May 12, 2018 first commit May 12, 2018

Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View

This repo contains training and testing code for our paper on Im2Pano3D. More information about the project can be found in our paper and project webset



The code and data are organized as follows:

         |-- matlab_code
         |-- torch_code
         |-- models
         |-- datalist
         |-- data
                |-- mpv3  
                |-- suncgpano  


  1. Requirements for torch (see: torch installation instructions)
  2. Install torch-hdf5
sudo apt-get install libhdf5-serial-dev hdf5-tools
git clone
cd torch-hdf5
luarocks make hdf5-0-0.rockspec LIBHDF5_LIBDIR="/usr/lib/x86_64-linux-gnu/"


  1. Dwonload training and testing panorama data: ./ [41 G]
  2. Dwonload models: ./ [1.7 G]


Export path

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cudnn/v5/lib64

Train on SUNCG data

cd torch_code/
name=suncg maskType=twoview   loss_xyz=1 loadOpt=rgbpns  Gtype_in=rgbpns   Gtype_out=pns  dataset=suncg  dataPath=../data/suncgpano/ DATA_ROOT=../datalist/trainlist_suncgroom8_10000.txt gpu=1 th train.lua  2>&1 | tee ./checkpoints/training_suncg.log

Train on Matterport3D data

cd torch_code/
name=mp maskType=twoview   loss_xyz=1 loadOpt=rgbpns  Gtype_in=rgbpns   Gtype_out=pns  dataset=mp  dataPath=../data/mpv3/  DATA_ROOT=../datalist/trainlist_mp3.txt gpu=1 th train.lua  2>&1 | tee ./checkpoints/training_suncg.log
  1. Supported input output modalities:
Gtype_in Gtype_out loadOpt
(input) (output) (load type)
pns pns pns
rgbpn pns rgbpns
rgbpns pns rgbpns
rgb pns rgbpns
  1. Supported mask types (maskType):
    twoview: Input two views of the panorama and predicts the other two. (a)
    1camera: one normal camera (b)
    3camera: three normal cameras (c)
    middlecamera: Input six RGB-D camera looking horizantoally forward (d)
    upcamera: Input six RGB-D camera looking upwards (e)
    random: random mask
    nomask: No mask applied in the input image. Example usage, input rgb panorama output pns.(f)\



name=mp_twoview_rgbpns  maskType=twoview dataset=suncg dataPath=../data/suncgpano/ checkpoints_dir=../checkpoints  DATA_ROOT=../datalist/testlist_suncgroom8.txt how_many=480 th test.lua

Testing with pre-trained models:

name=mpft_suncg_pns_pns_twoview  maskType=twoview dataset=suncg dataPath=../data/suncgpano/  checkpoints_dir=../models DATA_ROOT=../datalist/testlist_suncgroom8.txt how_many=480 th test.lua

Depth encoding conversion

See ./matlab_code/pano_io/pano_demo.m. It shows how to \

  1. Compute the 3D pointcloud from depth representation. \
  2. Compute the 3D pointcloud from plane equation representation. \


Code is inspired by DCGAN, DCGAN, Context-Encoder and pix2pix.