Seeing Motion at Nighttime with an Event Camera

Official repository for the CVPR 2024 paper, "Seeing Motion at Nighttime with an Event Camera”

Real Low-light Event Dataset (RLED)

Download

The publicly available RLED dataset includes 100 training sequences (about 171GB) and 12 test sequences (about 21GB), including 5 scenes such as city, suburbs, town, valley, and village. To reduce storage and transmission costs, all data is stored in .h5 format, and the images have been converted to grayscale. We offer both Baidu Cloud Drive and Microsoft OneDrive for data download:

Baidu Disk (code: cknc)

OneDrive

RLED provides spatially and temporally aligned image GTs and low-light events:

Data Format

The H5 file is based on h5_tools.

The structure of the h5 file is the following:

├── city_0000_25fps.h5
│   └── events
│       ├── xs
│       ├── ys
│       ├── ts
│       ├── ps
│   └── images
│       ├── image000000000
│       │   ├── event_idx
│       │   ├── size
│       │   └── timestamp
│       ├── image000000001
│       ├── image000000002
│       │   ...
│   └── event_indices
│
│   ─── attribute info
│       ├── duration
│       ├── num_events
│       ├── num_neg
│       ├── num_pos
│       ├── num_imgs
│       ├── sensor_resolution
│       ├── t0
│       └── tk

• events: xs(column idx), ys(row idx), ts(timestamp in microseconds), ps(polarity).
• images: the collection of all images.
  • event_idx: the index of the event in the entire event stream sequence corresponding to the end of exposure time for the current image.
  • size: image size (height * width).
  • timestamp: the timestamp of the end of image exposure in microseconds.
• event_indices(Optional): An [n, 2] array, where n is the number of images in the current h5 file. To ensure a strict correspondence between events and images, we extend 10 ms before and after the midpoint of each image's exposure time and calculate the indices of the events corresponding to the start and end times.
• attribute info:
  • duration: the duration of the current h5 file.
  • num_events: the total number of events.
  • num_neg: the total number of negtive events.
  • num_pos: the total number of positive events.
  • num_imgs: the total number of images.
  • sensor_resolution: image size (width, height).
  • t0: the timestamp of the first event.
  • tk: the timestamp of the last event.

Nighttime Event Reconstruction Network (NER-Net)

Requirements

Note: Higher versions of NumPy may cause errors in the ETS algorithm. It is recommended to use numpy==1.20.3.

conda create -n ner python=3.8
conda activate ner
pip install -r requirements.txt
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch

Inference

Download the pretrained model from Baidu Disk (code: cknc) or OneDrive, into "pretrained" folder.

Download RLED dataset to any /data_path.

Run the test (Set --save_gt and --save_evframe to save the GT frames and event frames).

python inference.py --checkpoint_path pretrained --model_name ner.pth --dataset_path /data_path/RLED/test --output_folder /data_path/RLED/results --save_gt --save_evframe

Training

The training code is based on e2vid+. First, convert the .h5 files in RLED to .npy files to improve data processing speed.

cd data_process
python h5_to_memmap.py --dataset_path /data_path/RLED

After completing the data conversion, you can adjust the training settings by modifying the configuration file (config/ner.json). The 'data_file' in 'data_loader' and 'valid_data_loader' need to be modified to the paths where train_memmap.txt and val_memmap.txt are located (generated by h5_to_memmap.py). Note that, since the optical flow results are too large to upload, we have removed the TC loss in the configuration file for easier execution. You can load the pre-trained LETC module parameters by setting --representation_model and freeze LETC in the reconstruction model training by setting --frozen_representation.

python train.py --config config/ner.json --device 0 --representation_model pretrained/representation_model/letc.pth --frozen_representation

Citation

Please cite us if our work is useful for your research, thank you!

@inproceedings{liu2024seeing,
  title={Seeing Motion at Nighttime with an Event Camera},
  author={Liu, Haoyue and Peng, Shihan and Zhu, Lin and Chang, Yi and Zhou, Hanyu and Yan, Luxin},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={25648--25658},
  year={2024}
}
@article{liu2025ner,
  title={NER-Net+: Seeing Motion at Nighttime with an Event Camera},
  author={Liu, Haoyue and Xu, Jinghan and Peng, Shihan and Chang, Yi and Zhou, Hanyu and Duan, Yuxing and Zhu, Lin and Tian, Yonghong and Yan, Luxin},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2025},
  publisher={IEEE}
}