[BMVC-23] SynBlink and BlinkFormer

This is the official code and data for paper SynBlink and BlinkFormer: A Synthetic Dataset and Transformer-Based Method for Video Blink Detection, accepted by BMVC 2023.

Introduction

Accurate blink detection algorithms have significant implications in numerous fields, including human-computer interaction, driving safety, cognitive science, and medical diagnostics. Despite considerable efforts, the dataset volume for blink detection remains relatively small due to the cost of data collection and annotation, and there is still room for improvement in the accuracy of current algorithms. In this paper, we introduce a workflow for synthesizing video data in Blender. Fully-rigged 3D human models are programmatically controlled, with variations in head movement, blinking, camera angles, background types, and lighting intensities. We used this workflow to create the SynBlink dataset, which includes 50,000 video clips and their corresponding annotations. Additionally, we present BlinkFormer, an innovative blink detection algorithm based on Transformer architecture that fully exploits temporal information from video clips. The model not only detects blinks for the entire input video but also estimates blink strength for each frame individually. Experimental results reveal that the BlinkFormer outperforms other state-of-the-art blink detection methods, achieving the highest F1-score on HUST-LEBW dataset. This accomplishment highlights the effectiveness of our approach in accurately detecting blinks and its potential for real-world applications.

---

SynBlink Dataset

• Download link: https://pan.baidu.com/s/1bJ0nj0SxfVCxRmICKz5p8A?pwd=synb (access code: synb)

• Folder Structure

    SynBlink-50K
    ├── F_01_0000_NB_TEB_HL
    │   ├── 01.png
    │   ├── 02.png
    │   ├── ...
    │   ├── 13.png
    │   └── annotations.json
    ├── ...
    ├── M_01_0000_BL_IMB_LL
    │   ├── 01.png
    │   ├── 02.png
    │   ├── ...
    │   ├── 13.png
    │   └── annotations.json
    ├── ...
    └── ...

Note: F_01_0000_NB_TEB_HL (F: Female (M: Male), id = 1, video id = 0000, NB: No Blink (BL: Blink), TEB: Texture Background (IMB: Image Background, HDB: HDRI Background), HL: High Light (LL: Low Light, ML: Medium Light))

• Annotations:

SynBlink-50K > F_01_0000_NB_TEB_HL > annotations.json

{
    "01": {
        "L_Eye_Left": [
            42,
            171
        ],
        "L_Eye_Right": [
            61,
            163
        ],
        "L_Eye_Center": [
            51,
            162
        ],
        "R_Eye_Left": [
            86,
            151
        ],
        "R_Eye_Right": [
            112,
            142
        ],
        "R_Eye_Center": [
            93,
            142
        ],
        "Blink_Strength": 0.06183916702866554
    },
}

Note: 01: frame id, L_Eye_Left/L_Eye_Right/L_Eye_Center: Left eye keypoints, R_Eye_Left/R_Eye_Right/R_Eye_Center: Right eye keypoints, Blink_Strength: The degree of eye closure is represented on a scale from 0 to 1, where 1 indicates complete eye closure and 0 represents fully open eyes.

• Generation Code: Please Refer to the SynBlink-data-generation folder.

---

BlinkFormer Model

Code: Please Refer to the BlinkFormer folder.

Citation

If you use SynBlink or BlinkFormer in your research, please consider the following BibTeX entry and giving us a star

@inproceedings{bo2023synblink,
  title={SynBlink and BlinkFormer: A Synthetic Dataset and Transformer-Based Method for Video Blink Detection},
  author={Bo Liu, Yang Xu, Feng Lu},
  booktitle={British Machine Vision Conference (BMVC)},
  year={2023}
}

Acknowledgement

This implementation is built upon VideoTransformer-pytorch, vit-pytorch and SyntheticHeadPose. We thank the authors for their released code.

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.