3D Vision and Language Research ✨

This is a curated list of "3D Vision and Language" research which is maintained by haonan. Watch this repository for the latest updates!

🍃Table of Contents

• 3D Multimodal Pretraining
• 3D Visual Understanding
• 3D Dense Captioning
• 3D Cross-Modal Retrieval
• 3D Visual QA & Reasoning
• 3D Visual Grounding
• Multi-view Learning

Methods

3D Multimodal Pretraining

• Context-aware Alignment and Mutual Masking for 3D-Language Pre-training [CVPR 2023]
  Zhao Jin¹, Munawar Hayat², Yuwei Yang¹, Yulan Guo³, Yinjie Lei^{1†
  1}Sichuan University, ²Monash University, ³Sun Yat-sen University

3D Visual Understanding

• ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding [CVPR 2023]
  Le Xue^1†, Mingfei Gao¹, Chen Xing¹, Roberto Mart´ın-Mart´ın^1,2, Jiajun Wu³, Caiming Xiong¹, Ran Xu¹, Juan Carlos Niebles¹, Silvio Savarese^{1
  1}Salesforce Research, Palo Alto, USA, ²UT Austin, Texas, USA, ³Stanford University, Stanford, USA

• ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding [Arxiv 2023]
  Le Xue^1†, Ning Yu¹, Shu Zhang¹, Junnan Li¹, Roberto Martín-Martín³, Jiajun Wu², Caiming Xiong¹, Ran Xu¹, Juan Carlos > Niebles^1,2, Silvio Savarese^{1,2
  1}Salesforce AI, ²Stanford University, ³The University of Texas at Austin

• PLA: Language-Driven Open-Vocabulary 3D Scene Understanding [CVPR 2023]
  Runyu Ding^1*, Jihan Yang^1*, Chuhui Xue², Wenqing Zhang², Song Bai^2†, Xiaojuan Qi^{1†
  1}The University of Hong Kong, ²ByteDance

• Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning? [ICLR 2023]
  Runpei Dong¹, Zekun Qi¹, Linfeng Zhang², Junbo Zhang², Jianjian Sun³, Zheng Ge⁴, Li Yi^245†, Kaisheng Ma^{2†
  1}Xi’an Jiaotong University, ²Tsinghua University, ³MEGVII Technology, ⁴Shanghai Artificial Intelligence Laboratory, ⁵Shanghai Qi Zhi Institute

3D Dense Captioning

• End-to-End 3D Dense Captioning with Vote2Cap-DETR [CVPR 2023]
  Sijin Chen^1*, Hongyuan Zhu², Xin Chen³, Yinjie Lei⁴, Tao Chen^1†, Gang YU^{3
  1}Fudan University, ²Institute for Infocomm Research, A^*STAR, ³Tencent PCG, ⁴Sichuan University

• Scan2Cap: Context-aware Dense Captioning in RGB-D Scans [CVPR 2021]
  Dave Zhenyu Chen¹, Ali Gholami2², Matthias Nießner¹, Angel X. Chang^{2
  1}Technical University of Munich, ²Simon Fraser University

3D Cross-Modal Retrieval

• RONO: Robust Discriminative Learning with Noisy Labels for 2D-3D Cross-Modal Retrieval [CVPR 2023]
  Yanglin Feng¹, Hongyuan Zhu², Dezhong Peng^1,3,4, Xi Peng^{1</sup, Peng Hu1†
  1College of Computer Science, Sichuan University, 2Institute for Infocomm Research (I2R), A*STAR, 3Sichuan Zhiqian Technology, 4Chengdu Ruibei Yingte Information Technology}

3D Visual QA and Reasoning

• 3D Concept Learning and Reasoning from Multi-View Images [CVPR 2023]
  Yining Hong¹, Chunru Lin², Yilun Du³, Zhenfang Chen⁵, Joshua B. Tenenbaum³, Chuang Gan^{4,5
  1}UCLA, ²Shanghai Jiaotong University, ³MIT CSAIL, ⁴UMass Amherst, ⁵MIT-IBM Watson AI Lab

3D Visual Grounding

• EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding [CVPR 2023]
  Yanmin Wu¹, Xinhua Cheng¹, Renrui Zhang^2,3, Zesen Cheng¹, Jian Zhang^{1†
  1}Shenzhen Graduate School, Peking University, ²The Chinese University of Hong Kong, ³Shanghai AI Laboratory

Multi-view Learning

• Investigating and Mitigating the Side Effects of Noisy Views in Multi-view Clustering in Practical Scenarios [Arxiv 2023]
  Jie Xu, Gang Niu, Xiaolong Wang, Yazhou Ren, Lei Feng, Xiaoshuang Shi, Heng Tao Shen, Xiaofeng Zhu

• Deep Incomplete Multi-View Clustering via Mining Cluster Complementarity [AAAI 2022]
  Jie Xu¹, Chao Li¹, Yazhou Ren^1†, Liang Peng¹, Yujie Mo¹, Xiaoshuang Shi^1†, Xiaofeng Zhu^{12
  1}UESTC, ²Shenzhen Institute for Advanced Study, UESTC

• Multi-level Feature Learning for Contrastive Multi-view Clustering [CVPR 2022]
  Jie Xu^1†, Huayi Tang^1†, Yazhou Ren¹, Liang Peng¹, Xiaofeng Zhu¹², Lifang He^{3
  1}UESTC, ²Shenzhen Institute for Advanced Study, UESTC, ³Lehigh University

• Multi-VAE: Learning Disentangled View-common and View-peculiar Visual Representations for Multi-view Clustering [ICCV 2022]
  Jie Xu¹, Yazhou Ren^1†, Huayi Tang¹, Xiaorong Pu¹, Xiaofeng Zhu¹, Ming Zeng², Lifang He^{3
  1}UESTC, ²CMU, ³Lehigh University