Revealing Latent Information: A Physics-inspired Self-supervised Pre-training Framework for Noisy and Sparse Events (ACM MM 2025)
This study proposes a self-supervised pre-training framework to fully reveal latent information in event data, including edge information and texture cues. Our framework consists of three stages: Difference-guided Masked Modeling, inspired by the event physical sampling process, reconstructs temporal intensity difference maps to extract enhanced information from raw event data. Backbone-fixed Feature Transition contrasts event and image features without updating the backbone to preserve representations learned from masked modeling and stabilizing their effect on contrastive learning. Focus-aimed Contrastive Learning updates the entire model to improve semantic discrimination by focusing on high-value regions. Extensive experiments show our framework is robust and consistently outperforms state-of-the-art methods on various downstream tasks, including object recognition, semantic segmentation, and optical flow estimation.
- [2026-07-12] Checkpoints and datasets released.
- [2025-10-26] Code released.
- [2025-07-06] Paper accepted by ACM MM 2025.
- python
- pytorch
- timm
- clip
- ptflops
- hdf5plugin
- tensorboard
| Backbone | after MM | after MM-Trans-CL |
|---|---|---|
| ViT | OneDrive / BaiduDisk | OneDrive / BaiduDisk |
| ConvViT | OneDrive / BaiduDisk | OneDrive / BaiduDisk |
| Swin | OneDrive / BaiduDisk | OneDrive / BaiduDisk |
| Task | Datasets |
|---|---|
| Pre-training | Organized | Original |
| Object Recognition | N-Caltech101 | CIFAR10-DVS | N-Cars | N-ImageNet | ES-ImageNet |
| Semantic Segmentation | DDD17 | DSEC |
| Optical Flow Estimation | MVSEC |
Please refer to Tables S16–S18 in the paper's supplementary material, or contact me if you have any questions.
Thanks to the following open-source works for their help and inspiration: ConvMAE, MoCo v3, GreenMIM, ECDP, ECDDP, MEM, BEiT, ES-ImageNet, ESS, DECIFlow, Layer-Grafted, EvRepSL, v2e, CLIP, etc.