4D Panoptic Scene Graph Generation

           

4D Panoptic Scene Graph Generation
Jingkang Yang, Jun Cen, Wenxuan Peng, Shuai Liu,
Fangzhou Hong, Xiangtai Li, Kaiyang Zhou, Qifeng Chen, Ziwei Liu,
S-Lab, NTU & HKUST & BUPT & HKBU

---

What is PSG4D Task?

The PSG4D (4D Panoptic Scene Graph Generation) Task is a novel task that aims to bridge the gap between raw visual inputs in a dynamic 4D world and high-level visual understanding. It involves generating a comprehensive 4D scene graph from RGB-D video sequences or point cloud video sequences.

The PSG4D Dataset

We provide two dataset to facilitate PSG4D research. Each dataset is composed with RGB-D/3D videos. To access them, please checkout data/GTA and data/HOI. If you find downloading PSG4D-GTA dataset challenging, please email jingkang001@e.ntu.edu.sg for some useful tips.

PSG4D-GTA Dataset Demo	PSG4D-HOI Dataset Demo

How to Run

Illustration of the PSG4DFormer pipeline. The PSG4DFormer is a two stage pipeline. For Panoptic Segmentation part, please refer to rgbd_seg for RGB-D segmentation and pc_seg for point cloud segmentation. Then please refer to *_track. The relation modeling is identical to our previous work OpenPVSG. Each part can be considered as a standalone code, so please checkout the readme in each directory.

Citation

If you find our repository useful for your research, please consider citing our paper:

@inproceedings{yang2023psg4d,
    author = {Yang, Jingkang and Cen, Jun and Peng, Wenxuan and Liu, Shuai amd Hong, Fangzhou and Li, Xiangtai and Zhou, Kaiyang and Chen, Qifeng and Liu, Ziwei}
    title = {4D Panoptic Scene Graph Generation},
    booktitle = {NeurIPS},
    year = {2023},
}