Skip to content
Natural and Realistic Single Image Super-Resolution with Explicit Natural Manifold Discrimination (CVPR, 2019)
Branch: master
Clone or download
Latest commit 993e21d Jul 4, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Model Uploaded Jun 27, 2019
figure Uploaded Jun 27, 2019
README.md Fixed Problem Jul 4, 2019
test.py Fixed Problem Jul 4, 2019
utils.py Uploaded Jun 27, 2019

README.md

NatSR

Natural and Realistic Single Image Super-Resolution with Explicit Natural Manifold Discrimination (CVPR 2019)

Jae Woong Soh, Gu Yong Park, Junho Jo, and Nam Ik Cho

[Paper] [Supplementary]

Environments

Abstract

Recently, many convolutional neural networks for single image super-resolution (SISR) have been proposed, which focus on reconstructing the high-resolution images in terms of objective distortion measures. However, the networks trained with objective loss functions generally fail to reconstruct the realistic fine textures and details that are essential for better perceptual quality. Recovering the realistic details remains a challenging problem, and only a few works have been proposed which aim at increasing the perceptual quality by generating enhanced textures. However, the generated fake details often make undesirable artifacts and the overall image looks somewhat unnatural. Therefore, in this paper, we present a new approach to reconstructing realistic super-resolved images with high perceptual quality, while maintaining the naturalness of the result. In particular, we focus on the domain prior properties of SISR problem. Specifically, we define the naturalness prior in the low-level domain and constrain the output image in the natural manifold, which eventually generates more natural and realistic images. Our results show better naturalness compared to the recent super-resolution algorithms including perception-oriented ones.

Related Work

Distortion Oriented Single Image Super-Resolution

[EDSR (CVPRW 2017)] Enhanced Deep Residual Networks for Single Image Super-Resolution Link

Perception Oriented Single Image Super-Resolution

[SRGAN (CVPR 2017)] Photo-realistic Single Image Super-Resolution Using a Generative Adversarial Network Link

[EnhanceNet (ICCV 2017)] Enhancenet: Single Image Super-Resolution Through Automated Texture Synthesis Link

[SFT-GAN (CVPR 2018)] Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform Link



Brief Description of Our Proposed Method

Explicitly Modeling the SISR & Designing Natural Manifold

As inifinite number of high-resolution (HR) images can be correspond to one low-resolution (LR) image, SISR is one-to-many problem. Therefore, we first define HR space and we divided HR space into three subspaces based on our prior knowledge.

Natural Manifold Discrimination

The network architecture of Natural Manifold Discriminator (NMD) and the loss function for training NMD.

Natural and Realistic Single Image Super-Resolution (NatSR)

  

Left: The overall training scheme & Right: The SR network (Generator) architecture.

Experimental Results

(FR-IQA) Results of the average PSNR (dB) and SSIM for the benchmark

(NR-IQA) Results of the NIQE and NQSR for BSD100 (Left) Perception-Distortion Tradeoff [Ref] Plot for BSD100 (Right)

Visualized Results



Guidelines for Test Codes

Requisites should be installed beforehand.

Clone this repo.

git clone http://github.com/JWSoh/NatSR.git
cd NatSR/

Ready for input data (low-resolution).

Test

[Options]

python test.py --gpu [GPU_number] --ref [True/False] --datapath [LR path] --labelpath [HR path] --modelpath [pretrained model path] --model [NatSR/FRSR] --savepath [SR path] --save [True/False]

--gpu: If you have more than one gpu in your computer, the number designates the index of GPU which is going to be used. [Default 0]
--ref: [1/0] 1 if there exist reference images. (Reference images are just for PSNR measurements.) [Default 1]
--datapath: Path of input images.
--labelpath: Path of reference images. (Not required, only for PSNR.)
--modelpath: Path of pretrained models. (If you clone this repo., you don't need to specify.
--model: [NatSR/FRSR] The type of model. [Default: NatSR]
--savepath: Path for super-resolved images. [Default: result]
--save: [1/0] Flag whether to save SR images. [Default 1]

Examples

To generate super-resolved image and also to measure PSNR.

python test.py --gpu 0 --datapath LR/Set5 --labelpath HR/Set5 --model NatSR

To generate output images, only.

python test.py --gpu 0 --ref 0 --datapath LR/Set5 --model NatSR

Super-resolve with FRSR (Distortion oriented model)

python test.py --gpu 0 --ref 0 --datapath LR/Set5 --model FRSR

Citation

@InProceedings{Soh_2019_CVPR,
author = {Soh, Jae Woong and Park, Gu Yong and Jo, Junho and Cho, Nam Ik},
title = {Natural and Realistic Single Image Super-Resolution With Explicit Natural Manifold Discrimination},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}
You can’t perform that action at this time.