EEG2Video

This repository is the official implementation of our NeurIPS 24 paper: EEG2Video.

EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals
Xuan-Hao Liu, Yan-Kai Liu, Yansen Wang, Kan Ren, Hanwen Shi, Zilong Wang, Dongsheng Li, Bao-Liang Lu, Wei-Long Zheng

📣 News

• Dec. 14, 2024. Our SEED-DV Dataset release.
• Dec. 13, 2024. EEG2Video code release.
• Nov. 25, 2024. EEG-VP code release.
• Sep. 26, 2024. Accepted by NeurIPS 2024.
• April. 5, 2025. We notice all issues in Github, we will update the code base in two months.

Installation

1. Fill out the SEED-DV's License file and Apply the dataset.

2. Download this repository: git clone https://github.com/XuanhaoLiu/EEG2Video.git

3. Create a conda environment and install the packages necessary to run the code.

conda create -n eegvideo
conda activate eegvideo
pip install -r requirements.txt

🖼️ Reconstruction Demos

GT	Ours	GT	Ours	GT	Ours
GT	Ours	GT	Ours	GT	Ours
GT	Ours	GT	Ours	GT	Ours

😞 Fail Cases

We present some failure samples, these failures are typically caused by the inability of the model to infer either the semantic information or the low-level visual information correctly, resulting the irrelevantly generated videos.

GT	Ours	GT	Ours	GT	Ours

BibTeX

@inproceedings{liu2024eegvideo,
    title={{EEG}2Video: Towards Decoding Dynamic Visual Perception from {EEG} Signals},
    author={Liu, Xuan-Hao and Liu, Yan-Kai and Wang, Yansen and Ren, Kan and Shi, Hanwen and Wang, Zilong and Li, Dongsheng and Lu, Bao-Liang and Zheng, Wei-Long},
    booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS)},
    year={2024},
    url={https://openreview.net/forum?id=RfsfRn9OFd}
}

Acknowledgement

Huge thanks to the Stable Diffusion Team for opensourcing their high-quality AIGC models. Gratitude to the Tune-A-Video Team for their elegant text-to-video model. And kudos to the Mind-Video Team for their pioneering and excellent fMRI-to-video work.