Situational Awareness Matters in 3D Vision Language Reasoning

Yunze Man · Liang-Yan Gui · Yu-Xiong Wang

[CVPR 2024] [Project Page] [arXiv] [pdf] [BibTeX]

This repository contains the official PyTorch implementation of the paper "Situational Awareness Matters in 3D Vision Language Reasoning" (CVPR 2024). The paper is available on arXiv. The project page is online at here.

About

Previous methods perform direct 3D vision language reasoning without modeling the situation of an embodied agent in the 3D environment. Our method, SIG3D, grounds the situational description in the 3D space, and then re-encodes the visual tokens from the agent's intended perspective before vision-language fusion, resulting in a more comprehensive and generalized 3D vision language (3DVL) representation and reasoning pipeline.

News

• 06/11/2024: Paper is available on arXiv. Project page is online.

TODO

• Situation Estimation
• SQA3D-based VQA
• 3DLLM-based VQA
• Evaluation and Visualization

BibTeX

If you use our work in your research, please cite our publication:

@inproceedings{man2024situation3d,
      title={Situational Awareness Matters in 3D Vision Language Reasoning},
      author={Man, Yunze and Gui, Liang-Yan and Wang, Yu-Xiong},
      booktitle={CVPR},
      year={2024} 
      }

Acknowledgements

This repo is built based on the fantastic work SQA3D, ScanQA, and 3D-LLM. We thank the authors for their great work and open-sourcing their codebase.