Implementation of the paper: Light field intrinsics with a deep encoder-decoder network
Switch branches/tags
Nothing to show
Clone or download
Latest commit b617a48 Aug 1, 2018
Permalink
Failed to load latest commit information.
DataRead_code Delete lf_tools.cpython-36.pyc Jul 23, 2018
lf_autoencoder_cvpr2018_code Add files via upload Jul 23, 2018
README.md Update README.md Aug 1, 2018

README.md

Light field intrinsics with a deep encoder-decoder network

We present a fully convolutional autoencoder for light fields, which jointly encodes stacks of horizontal and vertical epipolar plane images through a deep network of residual layers. The complex structure of the light field is thus reduced to a comparatively low-dimensional representation, which can be decoded in a variety of ways. The different pathways of upconvolution we currently support are for disparity estimation and separation of the lightfield into diffuse and specular intrinsic components. The key idea is that we can jointly perform unsupervised training for the autoencoder path of the network, and supervised training for the other decoders. This way, we find features which are both tailored to the respective tasks and generalize well to datasets for which only example light fields are available. We provide an extensive evaluation on synthetic light field data, and show that the network yields good results on previously unseen real world data captured by a Lytro Illum camera and various gantries. teaser

Project description

Our project consist of 2 steps:

  1. Divide input light fields into 3D patches and create network inputs with DataRead_code project
  2. Train and evaluate the network with lf_autoencoder_cvpr2018_code project

Prerequisites

  1. Python 3.5
  2. Tensorflow with GPU support

1. Creating the data

Depends on the type of data use separate scripts to create inputs (.hdf5 data container) for the network:

  • synthetic create_training_data_autoencoder.py
  • light field benchmark create_training_data_depth.py
  • real-world use separete script create_training_data_lytro
px = 48 # patch size
py = 48 
nviews = 9 # number of views
sx = 16 # block step size
sy = 16

training_data_dir = "./trainData/"
training_data_filename = 'lf_patch_autoencoder1.hdf5'
file = h5py.File( training_data_dir + training_data_filename, 'w' )

data_source = "./CNN_data/1"

Synthetic data used for training, is organized in 7 folders, for every folder we create a data container.

2. Run the network

To train the network you need to specify all training options in the config_autoencoder_v9_final.py Also, you need to specify patch size and minimum and maximum disparity values in the config_data_format.py In cnn_autoencoder.py you need to specify coordinates that are taken into account when the loss is computed. For example, if the input patch size is 48x48, and we select loss_min_coord_3D= 0, loss_max_coord_3D = 40, then the last 8 pixels will be omitted while computing loss.

To use the trained model, please download the model current_v9.zip and extract the archive to ./networks/ folder. We provide some test examples diffuse_specular.zip that shoulb be extracted to the ./test_data/ folder.

To evaluate on all test examples used in the paper, create the test data with the DataRead_code project.

References