DarkFeat

DarkFeat: Noise-Robust Feature Detector and Descriptor for Extremely Low-Light RAW Images (AAAI2023 Oral)

Installation

git clone git@github.com:THU-LYJ-Lab/DarkFeat.git
cd DarkFeat
pip install -r requirements.txt

Pytorch installation is machine dependent, please install the correct version for your machine.

Demo

python ./demo_darkfeat.py \
	--input /path/to/your/sequence \
	--output_dir ./output \
	--resize 960 640 \
	--model_path /path/to/pretrained/weights

Sample raw image sequences and pretrained weights can be downloaded from here.

Note that different pytorch and cuda versions may cause different model output results, and the output matches may differ from those shown in the gif. The results are tested in python 3.6, PyTorch 1.10.2 and cuda 10.2.

Evaluation

1. Download MID Dataset.

2. Preprocessing the data in MID dataset, you can choose whether to enable histogram equalization or not:

   python raw_preprocess.py --dataset_dir /path/to/MID/dataset

3. Extract the keypoints and descriptors, followed by a nearest neighborhood matching:

   python export_features.py \
     --model_path /path/to/pretrained/weights \
     --dataset_dir /path/to/MID/dataset

4. Estimate the pose through corresponding keypoint pairs:

   python pose_estimation.py --dataset_dir /path/to/MID/dataset

5. Finally collect the results of pose estimation errors:

   python read_error.py
   

Training from scratch

We use GL3D as our source training-use matching dataset. Please follow the instructions to download and unzip all the data (including GL3D group and tourism group).

Then using the preprocessing code provided by ASLFeat to generate matching informations:

git clone https://github.com/lzx551402/tfmatch
# please edit the GL3D path in the shell script before executing.
cd tfmatch
sh train_aslfeat_base.sh

To launch the training, configure your training hyperparameters inside ./configs and then run:

# stage1
python run.py --stage 1 --config ./configs/config_stage1.yaml \
	--dataset_dir /path/to/your/GL3D/dataset \
	--job_name YOUR_JOB_NAME
# stage2 
python run.py --stage 2 --config ./configs/config_stage1.yaml \
	--dataset_dir /path/to/your/GL3D/dataset \
	--job_name YOUR_JOB_NAME \
	--start_cnt 160000
# stage3
python run.py --stage 3 --config ./configs/config.yaml \
	--dataset_dir /path/to/your/GL3D/dataset \
	--job_name YOUR_JOB_NAME \
	--start_cnt 220000

Citation

@inproceedings{he2023darkfeat,
  title={DarkFeat: Noise-Robust Feature Detector and Descriptor for Extremely Low-Light RAW Images},
  author={He, Yuze and Hu, Yubin and Zhao, Wang and Li, Jisheng and Liu, Yong-Jin and Han, Yuxing and Wen, Jiangtao},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={37},
  number={1},
  pages={826--834},
  year={2023}
}

Acknowledgements

This project could not be possible without the open-source works from ASLFeat, R2D2, MID, GL3D, SuperGlue. We sincerely thank them all.