Light Field Image Super-Resolution Network via Joint Spatio-Angular and Epipolar Information TCI-2023

This repository contains official pytorch implementation of "Light Field Image Super-Resolution Network via Joint Spatio-Angular and Epipolar Information" in IEEE Transactions on Computational Imaging 2023, by Vinh Van Duong, Thuc Nguyen Huu, Jonghoon Yim, and Byeungwoo Jeon.

News

[2023-03-20]: Our paper has been accepted in TCI, the early access version of this paper is avaiable here HLFSR. Moreover, we released our RR-HLFSR, which is participated in NTIRE2023. The RR-HLFSR is an enhanced version of our HLFSR.

[2023-01-16]: we added the "SAI24DLF" function into the "common.py" file.

[2022-12-26]: we have updated the pre-trained models and codes.

Results

We share the pre-trained models and the SR LF images generated by our HLFSR-C32 and HLFSR-C64 model on all the 5 datasets for 2x and 4x SR, which are avaliable at https://drive.google.com/drive/folders/1SaTT3iP4GruKcome8r97Y6y54Nrj4gc5

Code

Dependencies

• Python 3.6
• Pyorch 1.3.1 + torchvision 0.4.2 + cuda 92
• Matlab

Dataset

We use the processed data by LF-DFnet, including EPFL, HCInew, HCIold, INRIA and STFgantry datasets for training and testing. Please download the dataset in the official repository of LF-DFnet.

Prepare Training and Test Data

• To generate the training data, please first download the five datasets and run:

  GenerateTrainingData_HLFSR.m

• To generate the test data, run:

  GenerateDataForTest_HLFSR.m

Train

• Run:

  python train_HLFSR.py  --angRes 5 --upscale_factor 4 --channels 64  --crop_test_method 3

Test

• Run:

  python test_HLFSR.py --angRes 5 --upscale_factor 4 --channels 64  --crop_test_method 3 --model_path [pre-trained dir]

[Important note]: For our HLFSR method, the performance is following “the larger image patch size is the better”. For example, if we keep the whole image as an input of our network (i.e., crop_test_method is fixed equal to 1), it can be achieved the best performance. This is because our proposed network components require an adequate size of an input image to better exploit the pixel correlations in a larger receptive field. To get the same performance as reported in our paper, we need to set the default crop_test_method equal to 3.

Visual Results

• To merge the Y, Cb, Cr channels, run:

  GenerateResultImages.m
  
  

Citation

If you find this work helpful, please consider citing the following papers:

@Article{vinh2023-lfsr,
  author  = {Duong, V. V. and Nguyen, T. H. and Yim, J. and Jeon, B.},
  journal = {IEEE Trans. Compuational Imaging},
  title   = {Light Field Image Super-Resolution Network via Joint Spatial-Angular and Epipolar Information},
  year    = {2023},
}

@InProceedings{LF-InterNet,
  author    = {Wang, Yingqian and Wang, Longguang and Yang, Jungang and An, Wei and Yu, Jingyi and Guo, Yulan},
  title     = {Spatial-Angular Interaction for Light Field Image Super-Resolution},
  booktitle = {European Conference on Computer Vision (ECCV)},
  pages     = {290-308},
  year      = {2020},
}

```Citation
@article{LF-DFnet,
  author  = {Wang, Yingqian and Yang, Jungang and Wang, Longguang and Ying, Xinyi and Wu, Tianhao and An, Wei and Guo, Yulan},
  title   = {Light Field Image Super-Resolution Using Deformable Convolution},
  journal = {IEEE Transactions on Image Processing},
  volume  = {30),
  pages   = {1057-1071},
  year    = {2021},
}

Acknowledgement

Our work and implementations are inspired and based on the following projects:
LF-DFnet
LF-InterNet
We sincerely thank the authors for sharing their code and amazing research work!

Contact

if you have any question, please contact me through email duongvinh@skku.edu