Real-ESRGAN | Implementation of Real-ESRGAN

For the building image above please click here to see the original image and please click here to see the resulting/inference image.

Note: Images on the left are the original ones. Images on the right are coming from this project's model inference.

Implementation of Real-ESRGAN

Implementation of the paper
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Explore the docs »

Report Bug Request Feature

Table of Contents

1. About The Project
   • Problem Statement
   • Key Features
   • Built With
2. Getting Started
   • Prerequisites
   • Setup
3. Usage
4. Demo
5. Application Graphical User Interface
6. Reports
7. Contributing
8. License
9. Contact

About The Project

• This project is the implementation of the Real-ESRGAN and the Real-ESRNet models from the paper "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data"

Problem Statement

Trying to improve the quality of blurry images without knowing how they got blurry in the first place. However, existing methods still struggle with fixing common issues in real-world pictures. In this project, a strong image enhancement tool called ESRGAN is adapted for practical use and it is now called Real-ESRGAN. Real-ESRGAN has been trained using computer-generated data to better imitate complex real-world image problems. Real-ESRGAN also accounts for some common visual problems that might occur during the enhancement process. Real-ESRGAN does a better job than other methods when fixing real images.

Key Features

1. Practical Application of ESRGAN: The researchers extended the use of ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) for practical image restoration applications.
2. Training with Synthetic Data: Real-ESRGAN is trained using exclusively synthetic data. This means the model learns from computer-generated examples rather than real-world images.
3. High-Order Degradation Modeling: The authors introduced a high-order degradation modeling process. This helps the model better simulate and understand complex real-world image degradations.
4. Consideration of Common Artifacts: Real-ESRGAN takes into account common visual artifacts like ringing and overshoot that can occur during the image synthesis process.
5. Improved Discriminator Capability: A U-Net discriminator with spectral normalization is employed to enhance the discriminator's ability to evaluate and improve the training process's stability.
6. Superior Visual Performance: Extensive comparisons with previous methods on various real datasets demonstrate that Real-ESRGAN outperforms them in terms of visual quality.

Back to top

Built With

Back to top

Getting Started

To get a local copy of the Real-ESRGAN up and running locally follow these simple example steps:

Prerequisites

NOTE: How to check if Python is installed and what is its version

  python --version

NOTE: How to check if Git is installed and what is its version

  git -v

1. Please make sure you have pyenv installed and use Python3 version 3.10.9:

   • You can use pyenv to switch between different Python versions:
     • Windows: https://www.youtube.com/watch?v=HTx18uyyHw8
     • Mac: https://www.youtube.com/watch?v=31WU0Dhw4sk
     • Linux: https://www.youtube.com/watch?v=1Zgo8M9yUtM

2. Please make sure you have git installed

   • Windows: https://git-scm.com/download/win
   • Mac: https://git-scm.com/download/mac
   • Linux: https://git-scm.com/download/linux

3. Please look at the setup folder found in this project.

Setup

1. Navigate to the directory where you want to clone/run/save the application:

   cd your_selected_directory

2. Clone this repository:

   git clone https://github.com/GeorgiosIoannouCoder/realesrgan.git

3. Navigate to the realesrgan git repository:

   cd realesrgan

4. Use Python3 3.10.9 version in the cloned repository folder:

   pyenv local 3.10.9

5. Create virtual environment in the cloned repository folder:

   python -m venv .realesrgan-venv

6. Activate the virtual environment (Windows OR Mac/Linux):

   1. Windows

   .\.realesrgan-venv\Scripts\activate

   2. Mac/Linux

   source .realesrgan/bin/activate

7. Install the dependencies listed in the requirements.txt file:

   pip install -r requirements.txt

8. Run Streamlit:

   streamlit run app.py

NOTE: You can also use the inference/inference.py script to test the model in the command line. More instructions can be found below in the Usage section.

Back to top

Usage

NOTE: For training, NERSC was used with four A100 GPUs due to the huge size of the datasets and network architectures. It took five days to train both the Real-ESRNet and Real-ESRGAN.

1. To get any of the model weights from this project please contact me at any time. They have not been uploaded to GitHub due to the large file size. Please see the README.md files of the directories model_needed_for_esrnet_training and models for the required file structure and names of the model needed for the first training and the models resulting from both the first and second training for this project respectively.

2. To train your own model please follow the following instructions:

   NOTE: Please make sure that you have installed the dependencies listed in the requirements.txt file as shown in the Setup section before proceeding.

   1. Download the datasets DF2K and OST Training from here. Please see the readme file of the directory datasets in this project for the required file structure.
   2. For the DF2K dataset, use a multi-scale strategy to downsample HR images to obtain several Ground-Truth images with different scales. Use the script multiscale_df2k.py to do this.

      python image_scaling/multiscale_df2k.py --input datasets/DF2K/DF2K_HR --output datasets/DF2K/DF2K_HR_multiscale

   3. Next, crop the DF2K images obtained from step 2 into sub-images. Use the script image_crop.py to do this.

      python image_cropping/image_crop.py --input datasets/DF2K/DF2K_HR_multiscale --output datasets/DF2K/DF2K_HR_multiscale_subimages

   4. Prepare a txt file containing the image paths. Use the script image_path.py to do this.

      python image_path_generator/image_path.py -input datasets/DF2K/DF2K_HR_multiscale_subimages datasets/OST/ANIMAL datasets/OST/BUILDING -root "" "" "" -path ./image_path_generator/gt_image_paths.txt

   5. Train Real-ESRNet using the script training.py.

      python training/training.py -opt training_parameters/real_esrnet_x4.yml

   6. Train Real-ESRGAN using the script training.py and the model obtained from step 5.

      python training/training.py -opt training_parameters/real_esrgan_x4.yml --auto_resume

3. The training process happens in two stages:

   1. First, train Real-ESRNet with L1 loss from the pre-trained model ESRGAN_SRx4_DF2KOST_official-ff704c30.pth.
   2. Next, use the trained Real-ESRNet model from the previous step as an initialization of the generator, to train the Real-ESRGAN with a combination of L1 loss, perceptual loss, and GAN loss.

4. To inference any of the models, use the script inference.py:

   python inference/inference.py -input "ccny.jpg" -model_name "REALSRGAN_x4" -output "inferences" -upscale 4 -model_path "./models/REALESRGAN_x4.pth" -extension "auto"

Back to top

Demo

1. Training Process

   

2. Inference Process: Command Line | inference.py

   

3. Inference Process: Streamlit Web Application | app.py

   

Back to top

Application Graphical User Interface

Back to top

Report

1. You can find the report for this project here where it explains this project in detail.

Back to top

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Back to top

License

Distributed under the MIT License. See LICENSE for more information.

MIT License

Copyright (c) 2023 Georgios Ioannou

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Back to top

Contact

Georgios Ioannou - @LinkedIn

Georgios Ioannou - @georgiosioannoucoder - Please contact me via the form in my portfolio.

Project Link: https://github.com/GeorgiosIoannouCoder/realesrgan

Back to top