Skip to content

Multispectral event-based vision for deep learning based face detection (WACV 2024)

License

Notifications You must be signed in to change notification settings

ms-evs/ms-evs.github.io

Repository files navigation

ms-evs

Overview

The MS-EVS Dataset is the first large-scale event-based dataset for face detection.

  • Regular images (APS) and simulated events (EVS)
  • Multispectral data (RGB and/or near IR)
  • Very diverse faces and data quality (for robust face detection)

It is divided in four subsets:

  • N-YoutubeFaces, 3k+ celebrity faces from Youtube videos, in color (simulated events).
  • N-MobiFace, 80 unedited mobile live color videos (simulated events).
  • N-SpectralFace, 69 multispectral videos captured in-house (simulated events).
  • Real-SpectralFace, a few sequences with real multispectral events (Grayscale and Infrared).
Dataset APS Simulated EVS Real EVS Data
N-YoutubeFaces x x Color (RGB)
N-MobiFace x x Color (RGB)
N-SpectralFace x x Multispectral (9 visible + 1 infrared)
Real-SpectralFace x x Multispectral (1 visible + 1 infrared)

How to download

As the MS-EVS Dataset is quite big (a few hundred GBs), we compressed all the h5 files and split the dataset into many different subsets.

General procedure:

  • Before: Make sure you have the ZStandard compression tool: sudo apt install zstd
  • Select the relevant datasets for your project
  • Download the selected files, e.g. with wget -np -R "index.html*" -c -nH -r http://ms-evs.aisoft.org/link/to/subset/ (change link)
  • Uncompress the h5 files (zstd -d ./path/to/compressed/file.h5.zst -o ./path/to/uncompressed/file.h5)
  • Enjoy!
  • Optional: have a look at our demo scripts for a programmatic way to download all files and some functions to inspect the dataset files.

MS-EVS Dataset: All channels (RGB, Multispectral, Grayscale)

LIGHT MS-EVS Dataset: Grayscale only (~3 times lighter)

DEMO MS-EVS Samples: A selection of files to quickly explore the data

BONUS: Some files (8.5G) were removed from N-SpectralFace but can be downloaded here (https://ms-evs.aisoft.org/N-SpectralFace/extra/) : it is either captured under regular office light (LED) so it does not have infrared illumination (not used for our experiments) AND/OR it contains obvious labelling/calibration/syncing issues.

Cite

If you find the MS-EVS Dataset useful, please consider citing:

@InProceedings{Himmi_2024_WACV,
    author    = {Himmi, Saad and Parret, Vincent and Chhatkuli, Ajad and Van Gool, Luc},
    title     = {MS-EVS: Multispectral Event-Based Vision for Deep Learning Based Face Detection},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2024},
    pages     = {616-625}
}

Saad Himmi, Vincent Parret, Ajad Chhatkuli, and Luc Van Gool. Ms-evs: Multispectral event-based vision for deep learning based face detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 616–625, January 2024.

This dataset is distributed under the permissive MIT License.

More information

[paper][supp][poster]

Have a look at the supplementary material for more information about the dataset composition, capture, cleaning and labelling.

⚠️ Disclaimer: All datasets have been automatically labeled (either from scratch or to complete existing manual labels) using an existing face detector (YOLOv5) and therefore contain some (relatively few) wrong/missing bounding boxes. Any contribution is greatly appreciated.

Useful scripts

Use this script (download_ms_evs_demo.sh) to automatically download and uncompress the DEMO MS-EVS Samples (~5.2G total). This script can be easily modified to download/uncompress the MS-EVS Dataset (or LIGHT version).

Use the following script to see the HDF5 data dictionary structure: inspect_h5_tree.py. The structure depends on the number of devices used to record the dataset (1 for N-MobiFace and YoutubeFaces, 2 for the SpectralFace datasets) or the number of channels (RGB, Multispectral, GS+IR...)

The images, events and labels are all aligned and temporally synced! One can use plot_sample.py to plot synchronized data. This can serve 1)as an example on how to handle HDF5 files and/or events, or 2)as a way to quickly inspect files' data.

About

Multispectral event-based vision for deep learning based face detection (WACV 2024)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published