Homepage | Paper | Google Drive | Baidu Cloud | 知乎 | MIPI Starting-Kit

📰News | 🔧Install | ✨Models Zoo | 📷Quick Demo | 🤖Benchmark | 🚧Contribute | 📜License | ❓FAQ

This repository contains the official implementation of the following papers:

Lighting Every Darkness in Two Pairs: A Calibration-Free Pipeline for RAW Denoising
Xin Jin^*, Jia-Wen Xiao^*, Ling-Hao Han, Chunle Guo^#, Ruixun Zhang, Xialei Liu, Chongyi Li
(* denotes equal contribution. # denotes the corresponding author.)
In ICCV 2023, [Paper Link], [Poster], [Slides], [Video]

Make Explicit Calibration Implicit: Calibrate Denoiser Instead of the Noise Model
Xin Jin, Jia-Wen Xiao, Ling-Hao Han, Chunle Guo^#, Xialei Liu, Chongyi Li, Ming-Ming Cheng^#
(# denotes corresponding authors.)
arxiv preprint, [Paper Link]

Comparaison with Calibration-Based Method

Some brief introduction on the process of calibration in [EN/CN].

LED is a Calibration-Free (or called implicit calibration) Pipeline for RAW Denoising (currently for extremely low-light conditions).

So tired of calibrating the noise model? Try our LED!
Achieveing SOTA performance in 2 paired data and training time less than 4mins!

More Teaser	More Teaser

• First of all, 🔧 Dependencies and Installation.
• For academic research, please refer to pretrained-models.md and 🤖 Training and Evaluation.
• For further development, please refer to 🚧 Further Development.
• For using LED on your own camera, please refer to ✨ Pretrained Models and 📷 Quick Demo.

📰 News

Future work can be found in todo.md.

• Jan 13, 2024: Release the starting-kit for MIPI@2024. Additionally, we release the pre-trained parameters of Restormer and NAFNet.
• Dec 27, 2023: Update an extension version of our ICCV 23 paper (Project Page/Paper).
• Dec 1-5, 2023: Add the related code/doc[EN/CN] from PR#14/PR#16, thanks to @HYX20011209
• Sep 27, 2023: Add the urls to our Poster, Slides, and Video.
• Aug 19, 2023: Release relevent files on Baidu Clould(pwd: iay5).

History

• Aug 15, 2023: For faster benchmark, we released the relevant files in commit fadffc7.
• Aug, 2023: We released a Chinese explanation of our paper on 知乎.
• Aug, 2023: Our code is publicly available!
• July, 2023: Our paper "Lighting Every Darkness in Two Pairs: A Calibration-Free Pipeline for RAW Denoising" has been accepted by ICCV 2023.

🔧 Dependencies and Installation

1. Clone and enter the repo:

   git clone https://github.com/Srameo/LED.git ICCV23-LED
   cd ICCV23-LED

2. Simply run the install.sh for installation! Or refer to install.md for more details.

   We use the customized rawpy package in ELD, if you don't want to use it or want to know more information, please move to install.md

   bash install.sh

3. Activate your env and start testing!

   conda activate LED-ICCV23

✨ Pretrained Models

If your requirement is for academic research and you would like to benchmark our method, please refer to pretrained-models.md, where we have a rich variety of models available across a diverse range of methods, training strategies, pre-training, and fine-tuning models.

We are currently dedicated to training an exceptionally capable network that can generalize well to various scenarios using only two data pairs! We will update this section once we achieve our goal. Stay tuned and look forward to it!
Or you can just use the following pretrained LED module for custumizing on your own cameras! (please follow the instruction in Quick Demo).

Method	Noise Model	Phase	Framework	Training Strategy	Additional Dgain (ratio)	Camera Model	Validation on	🔗 Download Links	Config File
LED	ELD (5 Virtual Cameras)	Pretrain	UNet	PMN	100-300	-	-	[Google Drive]	[options/LED/pretrain/MM22_PMN_Setting.yaml]
LED	ELD (5 Virtual Cameras)	Pretrain	UNet	ELD	100-300	-	-	[Google Drive]	[options/LED/pretrain/CVPR20_ELD_Setting.yaml]
LED	ELD (5 Virtual Cameras)	Pretrain	UNet	ELD	1-200	-	-	[Google Drive]	[options/LED/pretrain/CVPR20_ELD_Setting_Ratio1-200.yaml]
LED	ELD (5 Virtual Cameras)	Pretrain	Restormer	ELD	100-300	-	-	[Google Drive]	[options/LED/other_arch/Restormer/LED+Restormer_Pretrain.yaml]
LED	ELD (5 Virtual Cameras)	Pretrain	NAFNet	ELD	100-300	-	-	[Google Drive]	[options/LED/other_arch/NAFNet/LED+NAFNet_Pretrain.yaml]

📷 Quick Demo

Get Clean Images in the Dark!

We provide a script for testing your own RAW images in image_process.py.
You could run python scripts/image_process.py --help to get detailed information of this scripts.

If your camera model is one of {Sony A7S2, Nikon D850}, you can found our pretrained model in pretrained-models.md.

Notice that, if you wish to use the model from release v0.1.1, you need to add the -opt parameter: For NAFNet, add -opt options/base/network_g/nafnet.yaml. For Restormer, add -opt options/base/network_g/restormer.yaml.

usage: image_process.py [-h] -p PRETRAINED_NETWORK --data_path DATA_PATH [--save_path SAVE_PATH] [-opt NETWORK_OPTIONS] [--ratio RATIO] [--target_exposure TARGET_EXPOSURE] [--bps BPS] [--led]

optional arguments:
  -h, --help            show this help message and exit
  -p PRETRAINED_NETWORK, --pretrained_network PRETRAINED_NETWORK
                        the pretrained network path.
  --data_path DATA_PATH
                        the folder where contains only your raw images.
  --save_path SAVE_PATH
                        the folder where to save the processed images (in rgb), DEFAULT: 'inference/image_process'
  -opt NETWORK_OPTIONS, --network_options NETWORK_OPTIONS
                        the arch options of the pretrained network, DEFAULT: 'options/base/network_g/unet.yaml'
  --ratio RATIO, --dgain RATIO
                        the ratio/additional digital gain you would like to add on the image, DEFAULT: 1.0.
  --target_exposure TARGET_EXPOSURE
                        Target exposure, activate this will deactivate ratio.
  --bps BPS, --output_bps BPS
                        the bit depth for the output png file, DEFAULT: 16.
  --led                 if you are using a checkpoint fine-tuned by our led.

Fine-tune for Your Own Camera!

A detailed doc can be found in issue#8.

1. Collect noisy-clean image pairs for your camera model, please follow the insruction in demo.md.
2. Select a LED Pretrained model in our model zoo (based on the additional dgain you want to add on the image), and fine-tune it using your data!

   python scripts/cutomized_denoiser.py -t [TAG] \
                                        -p [PRETRAINED_LED_MODEL] \
                                        --dataroot your/path/to/the/pairs \
                                        --data_pair_list your/path/to/the/txt
   # Then the checkpoints can be found in experiments/[TAG]/models
   # If you are a seasoned user of BasicSR, you can use "--force_yml" to further fine-tune the details of the options.

3. Get ready and test your denoiser! (move to Get Clean Images in the Dark!).

🤖 Training and Evaluation

Please refer to benchmark.md to learn how to benchmark LED, how to train a new model from scratch.

🚧 Further Development

If you would like to develop/use LED in your projects, welcome to let us know. We will list your projects in this repository.
Also, we provide useful tools for your futher development, please refer to develop.md.

📖 Citation

If you find our repo useful for your research, please consider citing our paper:

@inproceedings{jiniccv23led,
    title={Lighting Every Darkness in Two Pairs: A Calibration-Free Pipeline for RAW Denoising},
    author={Jin, Xin and Xiao, Jia-Wen and Han, Ling-Hao and Guo, Chunle and Zhang, Ruixun and Liu, Xialei and Li, Chongyi},
    journal={Proceedings of the IEEE/CVF International Conference on Computer Vision},
    year={2023}
}

@inproceedings{jin2023make,
  title={Make Explict Calibration Implicit: "Calibrate" Denoiser Instead of The Noise Model},
  author={Jin, Xin and Xiao, Jia-Wen and Han, Ling-Hao and Guo, Chunle and Liu, Xialei and Li, Chongyi and Cheng, Ming-Ming},
  journal={Arxiv},
  year={2023}
}

📜 License

This code is licensed under the Creative Commons Attribution-NonCommercial 4.0 International for non-commercial use only. Please note that any commercial use of this code requires formal permission prior to use.

📮 Contact

For technical questions, please contact xjin[AT]mail.nankai.edu.cn and xiaojw[AT]mail.nankai.edu.cn.

For commercial licensing, please contact cmm[AT]nankai.edu.cn.

🤝 Acknowledgement

This repository borrows heavily from BasicSR, Learning-to-See-in-the-Dark and ELD.
We would like to extend heartfelt gratitude to Ms. Li Xinru for crafting the exquisite logo for our project.

We also thank all of our contributors.