Papers with UnrealCV

A curated list of papers using UnrealCV.

This is a list of papers that use UnrealCV, an open source project to help researchers build virtual worlds using Unreal Engine 4/5. The papers are categorized by their applications, such as semantic understanding, 3D vision, embodied vision, etc.

The following labels are used to indicate the usage of UnrealCV in each paper:

• 📊Dataset: generate synthetic dataset for training.
• 🔍Diagnosis: control the factor for model diagnosis.
• 🏃Interaction: train/test agent(s) by interacting with the virtual worlds (e.g., RL).

Table of Contentc

• Semantic Understanding
  • Segmentation
  • Text Detection
  • Person Re-identification
  • Activity Recognition
• 3D Vision
  • Depth Estimation
  • Scene Reconstruction
  • Pose Estimation
• Embodied Vision
  • General Simulation/Datasets
  • Visual Navigation
  • Active Object Tracking
  • Multi-agent Cooperation
  • Applications

Contributing

If you find any papers that used UnrealCV but are not included in this list, please feel free to send a PR or open an issue.

When sending PRs, please put the new paper at the correct chronological position as the following format:

- Usage Tags(📊🔍🏃) 
  **Paper Title** 
  *Author(s)* 
  Conference/Journal Year. [[Paper](link)] [[code](link)] [[Website](link)]

Semantic Understanding

Object Segmentation and Detection

• 📊 ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond. Siyuan Qiao, Wei Shen, Weichao Qiu, Chenxi Liu, Alan Yuille. ICCV 2017. [paper] [code]

• 📊 Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation. Tremblay, Jonathan, Thang To, and Stan Birchfield. CVPR 2018 Workshop. [paper] [code]

• 📊 A Framework for Self-Training Perceptual Agents in Simulated Photorealistic Environments. Patrick Mania, Michael Beetz. ICRA 2019. [paper] [code]

• 📊 Semantic Part Detection via Matching: Learning to Generalize to Novel Viewpoints from Limited Training Data. Yutong Bai, Qing Liu, Lingxi Xie, Weichao Qiu, Yan Zheng, Alan Yuille. ICCV 2019. [paper]

• 📊 AugPOD: Augmentation-oriented Probabilistic Object Detection. Chuan-Wei Wang*, Chin-An Cheng*, Ching-Ju Cheng*, Hou-Ning Hu, Hung-Kuo Chu, Min Sun. CVPR 2019 Workshop. [paper]

• 📊 OmniSCV: An Omnidirectional Synthetic Image Generator for Computer Vision. Berenguel-Baeta, Bruno, Jesus Bermudez-Cameo, and Jose J. Guerrero. Sensors 2020. [paper]

• 📊 UR-MAT: A Multimodal, Material-Aware Synthetic Dataset of Urban Scenarios. Debora Russo, Nicola Mazzocca, Valeria Vittorini. ACM Multimedia 2025. [paper]

Text Detection and Recognition

• 📊 UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World. Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, Xiang Bai. CVPR 2020. [paper] [code]

• 📊 Synthetic-to-Real Unsupervised Domain Adaptation for Scene Text Detection in the Wild. Weijia Wu, Ning Lu, Enze Xie, Yuxing Wang, Wenwen Yu, Cheng Yang, Hong Zhou. ACCV 2020. [paper] [code]

Person Re-identification

• 📊 UnrealPerson: An Adaptive Pipeline for Costless Person Re-identification. Tianyu Zhang, Lingxi Xie, Longhui Wei, Zijie Zhuang, Yongfei Zhang, Bo Li, Qi Tian. CVPR 2021. [paper] [code]

• 📊 Robust Person Re-Identification with Wireless Signals Xi, Dong, Wengang Zhou, and Houqiang Li. ICME 2023. [paper]

• 📊 InfinitePerson: Innovating Synthetic Data Creation for Generalization Person Re-Identification. YingGuoqing Zhang; Jin Li; Yuhui Zheng; Ruili Wang. IEEE Transactions on Circuits and Systems for Video Technology. [paper] [code]

Activity Recognition

• 🔍 Identity Preserve Transform: Understand What Activity Classification Models Have Learnt. Jialing Lyu, Weichao Qiu, Xinyue Wei, Yi Zhang, Alan Yuille, Zheng-Jun Zha. CVPR 2020 Workshop. [paper]
• 📊 Active shooter detection and robust tracking utilizing supplemental synthetic data. Joshua R. Waite, Jiale Feng, Riley Tavassoli, Laura Harris, Sin Yong Tan, Subhadeep Chakraborty, Soumik Sarkar. Arxiv 2023. [paper]

3D Vision

Depth Estimation

• 🔍 Unrealstereo: Controlling hazardous factors to analyze stereo vision. Yi Zhang, Weichao Qiu, Qi Chen, Xiaolin Hu, Alan Yuille. 3DV 2018. [paper] [code]

• 📊 Coupled Real-Synthetic Domain Adaptation for Real-World Deep Depth Enhancement. Xiao Gu, Yao Guo, Fani Deligianni, Guang-Zhong Yang. IEEE Transactions on Image Processing (TIP) 2020. [paper]

• 📊 🏃 Enhancing optical-flow-based control by learning visual appearance cues for flying robots. G. C. H. E. de Croon, C. De Wagter, T. Seidl. Nature Machine Intelligence 2021. [paper]

• 📊 SMD-Nets: Stereo Mixture Density Networks. Fabio Tosi, Yiyi Liao, Carolin Schmitt, Andreas Geiger. CVPR 2021. [paper][code]

• 📊 G2-MonoDepth: A General Framework of Generalized Depth Inference from Monocular RGB+ X Data. Haotian Wang, Meng Yang, Nanning Zheng. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023. [paper]

• 📊 Learning Occluded Branch Depth Maps in Forest Environments Using RGB-D Images Geckeler, C., Aucone, E., Schnider, Y., Simeon, A., von Bassewitz, J. P., Zhu, Y., & Mintchev, S. IEEE Robotics and Automation Letters (RA-L) 2024. [[paper](https://ieeexplore.ieee.org/document/10403997]

• 📊 Real-World Depth Recovery via Structure Uncertainty Modeling and Inaccurate GT Depth Fitting. Delong Suzhang, Meng Yang. Arxiv 2025. [paper]

Scene Reconstruction

• 📊 Submodular Trajectory Optimization for Aerial 3D Scanning. Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, Neel Joshi. ICCV 2017. [paper ] [project]
• 📊 GEN-SLAM: Generative Modeling for Monocular Simultaneous Localization and Mapping. Punarjay Chakravarty, Praveen Narayanan, Tom Roussel. ICRA 2019. [paper]
• 📊 QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAM. Lachlan Nicholson, Michael Milford, Niko Sünderhauf. IEEE Robotics and Automation Letters (RA-L) 2019. [paper]
• 🏃 Path Planning for Active V-Slam Based on Reinforcement Learning. Borui Li, Fuchun Sun, Huaping Liu, Bin Fang. International Conference on Cognitive Systems and Signal Processing 2019. [paper]
• 📊 UnrealNavigation: Simulation Software for testing SLAM in Virtual Reality. Anne M. Bettens, Benjamin Morrell, Mauricio Coen, Neil McHenry, Xiaofeng Wu, Peter Gibbens, Gregory Chamitoff. AIAA Scitech 2020 Forum. [paper] [code]
• 📊 An Efficient Sampling-based Method for Online Informative Path Planning in Unknown Environments. Lukas Schmid, Michael Pantic, Raghav Khanna, Lionel Ott, Roland Siegwart, Juan Nieto. IEEE Robotics and Automation Letters (RA-L) 2020. [paper] [code]
• 🏃 Next-Best View Policy for 3D Reconstruction. Peralta, D., Casimiro, J., Nilles, A.M., Aguilar, J.A., Atienza, R., and Cajote, R. ECCV Workshops 2020. [paper] [code]
• 📊 Flight Planning for Survey-Grade 3D Reconstruction of Truss Bridges. Zhexiong Shang, Zhigang Shen. Remote Sens. 2022. [paper]
• 📊 View Planning Using Discrete Optimization for 3D Reconstruction of Row Crops. Athanasios Bacharis, Henry J. Nelson and Nikolaos Papanikolopoulos. IROS 2022. [paper]
• 📊 YOWO: You Only Walk Once to Jointly Map an Indoor Scene and Register Ceiling-Mounted Cameras. Fan Yang, Sosuke Yamao, Ikuo Kusajima, Atsunori Moteki, Shoichi Masui, Shan Jiang IEEE Transactions on Circuits and Systems for Video Technology, 2025. [paper]
• 📊🏃 Semantic-Driven Informed Planning and 3D Reconstruction for the Quadrotor Unmanned Aerial Vehicle. Xiaotian Xu; Xuetao Zhang; Yisha Liu; Hanzhang Wang; Xuebo Zhang; Yan Zhuang. IEEE Transactions on Vehicular Technology, 2025. [paper]

Pose Estimation

• 📊 A Unified Framework for Multi-View Multi-Class Object Pose Estimation. Chi Li, Jin Bai, Gregory D. Hager. ECCV 2018. [paper]

• 📊 🏃 CRAVES: Controlling Robotic Arm with a Vision-based, Economic System. Yiming Zuo*, Weichao Qiu*, Lingxi Xie, Fangwei Zhong, Yizhou Wang, Alan Yuille. CVPR 2019. [paper] [code]

• 📊 Learning from Synthetic Animals. Jiteng Mu*, Weichao Qiu*, Gregory Hager, Alan Yuille. CVPR 2020 (Oral). [paper] [code]

• 📊 AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild. Zhe Zhang, Chunyu Wang, Weichao Qiu, Wenhu Qin, Wenjun Zeng. IJCV 2020. [paper] [code]

• 📊 Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering. Pedro F. Proença, Yang Gao. ICRA 2020. [paper] [project]

• 📊 Learning From Synthetic Vehicles. Tae Soo Kim, Bohoon Shim, Michael Peven, Weichao Qiu, Alan Yuille, Gregory D. Hager. WACV 2022. [paper] [dataset]

• 🏃 Proactive Multi-Camera Collaboration for 3D Human Pose Estimation. Hai Ci*, Mickel Liu*, Xuehai Pan*, Fangwei Zhong, Yizhou Wang. ICLR 2023. [paper] [project]

• 📊 SPADES: A Realistic Spacecraft Pose Estimation Dataset using Event Sensing. Arunkumar Rathinam, Haytam Qadadri, Djamila Aouada. ICRA 2024. [paper] [code]

• 📊 Monocular 3D Human Pose Estimation for Sports Broadcasts using Partial Sports Field Registration. Tobias Baumgartner, Stefanie Klatt. CVPR-W 2023. [paper ] [code]

• 📊 Hardware-accelerated Mars Sample Localization via Deep Transfer Learning from Photorealistic Simulations R. Castilla-Arquillo, C. J. P ́erez-del-Pulgar, G. J. Paz-Delgado and L. Gerdes IEEE Robotics and Automation Letters 2022. [paper] [code]

View Synthesis

• 📊 Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar. SIGGRAPH 2019. [paper][project]

• 📊 5D Light Field Synthesis from a Monocular Video. Kyuho Bae, Andre Ivan, Hajime Nagahara, In Kyu Park. Arxiv 2019. [paper]

Event Camera

• 📊 ESIM: an Open Event Camera Simulator. Henri Rebecq, Daniel Gehrig, Davide Scaramuzza. CoRL 2018. [paper][code]

• 📊 Insights into Batch Selection for Event-Camera Motion Estimation. Valerdi, Juan L., Chiara Bartolozzi, and Arren Glover. Sensors 2023. [paper]

Embodied Vision

General Simulation/Datasets

• 🏃 📊 🔍 UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI. Fangwei Zhong, Kui Wu, Churan Wang, Hao Chen, Hai Ci, Zhoujun Li, Yizhou Wang. ICCV 2025. [paper] [project]

• 📊 🏃 EmbRACE-3K: Embodied Reasoning and Action in Complex Environments. Mingxian Lin, Wei Huang, Yitang Li, Chengjie Jiang, Kui Wu, Fangwei Zhong, Shengju Qian, Xin Wang, Xiaojuan Qi. Arxiv 2025. [paper]

• 📊 🏃 SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds. Jiawei Ren, Xiaokang Ye, Lingjun Mao, Xuhong He, Jianzhi Shen, Yiming Liang, Ruixuan Zhang, Tianai Yue, Yiqing Yang, Ryan Wu, Kevin Benavente, Rajiv Mandya Nagaraju, Muhammad Faayez, Xiyan Zhang, Dhruv Vivek Sharma, Xianrui Zhong, Ziqiao Ma, Tianmin Shu, Zhiting Hu, Lianhui Qin. Arxiv 2025. [paper] [project] [code]

• 📊 🏃 SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration. Jiawei Ren, Yiming Liang, Xiaokang Ye, Lingjun Mao, Xuhong He, Tianai Yue, Yiqing Yang, Ryan Wu, Zhiting Hu, Lianhui Qin. NeurIPS 2025. [paper] [project]

• 📊 🏃 SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning. Haoqiang Kang, Xiaokang Ye, Yuhan Liu, Siddhant Hitesh Mantri, Lingjun Mao, James Fleming, Drishti Regmi, Lianhui Qin. Arxiv 2026. [paper] [code]

• 📊 🏃 🔍 LychSim: A Controllable and Interactive Simulation Framework for Vision Research. Wufei Ma, Chloe Wang, Siyi Chen, Jiawei Peng, Patrick Li, Alan Yuille. Arxiv 2026. [paper] [project] [code]

• 📊 🏃 🔍 RescueBench: Can Embodied Agents Save Lives in the Wild ?. Kui Wu, Beiyu Guo, Hao Chen, ShuHang Xu, Yuling Li, Yongdan Zeng, Zhoujun Li, Yizhou Wang, Fangwei Zhong. Arxiv 2026. [paper] [code]

Visual Navigation

• 🏃 An Improved Method Based on Deep Reinforcement Learning for Target Searching. Xiao Long Wei, Xiang Lin Huang, Tao Lu, Ge Ge Song. International Conference on Robotics and Automation Engineering (ICRAE) 2019. [paper]

• 🏃 Training an Agent to Find and Reach an Object in Different Environments using Visual Reinforcement Learning and Transfer Learning. Evelyn Conceição Santos Batista, Wouter Caarls, Leonardo A. Forero, Marco Aurélio C. Pacheco. ICAART 2021. [paper]

• 📊 🏃 Enhancing optical-flow-based control by learning visual appearance cues for flying robots. G. C. H. E. de Croon, C. De Wagter, T. Seidl. Nature Machine Intelligence 2021. [paper]

• 📊 🏃 Simultaneous localization and mapping architecture for small bodies and space exploration.. Bettens, A., Morrell, B., Coen, M., Wu, X., Gibbens, P., & Chamitoff, G.. Advances in Space Research 2024, 73(1), 1185-1197. [paper]

• 📊 🏃 OpenFly: A Versatile Toolchain and Large-scale Benchmark for Aerial Vision-Language Navigation. Yunpeng Gao, Chenhui Li, Zhongrui You, Junli Liu, Zhen Li, ..., Bin Zhao, Xuelong Li. Arxiv 2025. [paper] [project]

• 📊 🏃 UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning. Xiangyu Wang, Donglin Yang, Yue Liao, Wenhao Zheng, wenjun wu, Bin Dai, Hongsheng Li, Si Liu. Arxiv 2025. [paper]

• 📊 🏃 DeliveryBench: Can Agents Earn Profit in Real World?. Lingjun Mao, Jiawei Ren, Kun Zhou, Jixuan Chen, Ziqiao Ma, Lianhui Qin. Arxiv 2025. [paper] [project] [code]

• 📊 🏃 Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents. Shaofei Cai, Zhancun Mu, Haiwen Xia, Bowei Zhang, Anji Liu, Yitao Liang. Arxiv 2025. [paper]

• 🏃 🔍 WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation. Baining Zhao, Jiacheng Xu, Weicheng Feng, Xin Zhang, Zhaolu Wang, Haoyang Wang, Shilong Ji, Ziyou Wang, Jianjie Fang, Zhiheng Zheng, Weichen Zhang, Yu Shang, Wei Wu, Chen Gao, Xinlei Chen, Yong Li. Arxiv 2026. [paper] [code] [project]

• 📊 🏃 Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation. Xiangyi Zheng, Xiangyu Wang, Qinan Liao, Zimu Tang, Yue Liao, Dongyue Lyu, Guodong Wang, Junjie Liu, Si Liu. Arxiv 2026. [paper]

Active Object Tracking

• 🏃 End-to-end Active Object Tracking via Reinforcement Learning. Wenhan Luo*, Peng Sun*, Fangwei Zhong, Wei Liu, Tong Zhang, Yizhou Wang. ICML 2018. [paper] [project]

• 🏃 End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning. Wenhan Luo*, Peng Sun*, Fangwei Zhong*, Wei Liu, Tong Zhang, Yizhou Wang. IEEE TPAMI 2019. [paper] [project]

• 🏃 AD-VAT: An Asymmetric Dueling mechanism for learning Visual Active Tracking. Fangwei Zhong, Wenhan Luo, Peng Sun, Tingyun Yan, Yizhou Wang. ICLR 2019. [paper] [code]

• 🏃 🔍 AD-VAT+: An Asymmetric Dueling Mechanism for Learning and Understanding Visual Active Tracking. Fangwei Zhong, Wenhan Luo, Peng Sun, Tingyun Yan, Yizhou Wang. IEEE TPAMI 2019. [paper] [code]

• 🏃 Pose-Assisted Multi-Camera Collaboration for Active Object Tracking. Jing Li*, Jing Xu*, Fangwei Zhong*, Xiangyu Kong, Yu Qiao, Yizhou Wang. AAAI 2020. [paper] [code]

• 🏃 🔍 Towards Distraction-Robust Active Visual Tracking. Fangwei Zhong, Wenhan Luo, Peng Sun, Tingyun Yan, Yizhou Wang. ICML 2021. [paper] [code] [project]

• 🏃 Anti-Distractor Active Object Tracking in 3D Environments. Mao Xi, Yun Zhou, Zheng Chen, Wengang Zhou, Houqiang Li. IEEE TCSVT 2022. [paper]

• 🏃 RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking. Fangwei Zhong*, Xiao Bi*, Yudi Zhang, Wei Zhang, Yizhou Wang. AAAI 2023. [paper] [project]

• 📊🏃 Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL. Fangwei Zhong, Kui Wu, Hai Ci, Churan Wang, Hao Chen*. ECCV 2024. [paper] [code]

• 🏃 VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models. Kui Wu, Shuhang Xu, Hao Chen, Churan Wang, Zhoujun Li, Yizhou Wang, Fangwei Zhong. IROS 2025. [paper]

• 🏃 TrackVLA: Embodied Visual Tracking in the Wild. Shaoan Wang, Jiazhao Zhang, Minghan Li, Jiahang Liu, Anqi Li, Kui Wu, Fangwei Zhong, Junzhi Yu, Zhizheng Zhang, He Wang. CoRL 2025. [paper]

• 🏃📊 Hierarchical Instruction-aware Embodied Visual Tracking. Kui Wu, Hao Chen, Churan Wang, Fakhri Karray, Zhoujun Li, Yizhou Wang, Fangwei Zhong. Arxiv 2025. [paper]

• 🏃 🔍 CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models. Youzhi Liu, Li Gao, Liu Liu, Mingyang Lv, Yang Cai. Arxiv 2026. [paper] [code]

• 🏃 AdaTracker: Learning Adaptive In-Context Policy for Cross-Embodiment Active Visual Tracking. Kui Wu, Hao Chen, Jinzhu Han, Haijun Liu, Churan Wang, Yizhou Wang, Zhoujun Li, Si Liu, Fangwei Zhong. IEEE RA-L 2026. [paper]

• 🏃 Instance-level Visual Active Tracking with Occlusion-Aware Planning. Haowei Sun, Kai Zhou, Hao Gao, Shiteng Zhang, Jinwu Hu, Xutao Wen, Qixiang Ye, Mingkui Tan. CVPR 2026 Poster. [paper] [code]

• 📊 🏃 CosFly: Plan in the Matrix, Fly in the World. Hanxuan Chen, Xiangyue Wang, Songsheng Cheng, Ruilong Ren, Jie Zheng, Shuai Yuan, Tianle Zeng, Hanzhong Guo, Binbo Li, Kangli Wang, Ji Pei. Arxiv 2026. [paper]

• 🏃📊 TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Tracking. Shaoan Wang, Jiazhao Zhang, Minghan Li, Jiahang Liu, Anqi Li, Kui Wu, Fangwei Zhong, Junzhi Yu, Zhizheng Zhang, He Wang. ICRA 2026. [paper] [project]

Multi-Agent Cooperation

• 🏃 Pose-Assisted Multi-Camera Collaboration for Active Object Tracking. Jing Li*, Jing Xu*, Fangwei Zhong*, Xiangyu Kong, Yu Qiao, Yizhou Wang. AAAI 2020. [paper] [code]

• 🏃 Proactive Multi-Camera Collaboration for 3D Human Pose Estimation. Hai Ci*, Mickel Liu*, Xuehai Pan*, Fangwei Zhong, Yizhou Wang. ICLR 2023. [paper] [project]

Applications

• 🏃 iFANnpp: Nuclear Power Plant Digital Twin for Robots and Autonomous Intelligence. Youndo Do, Marc Zebrowitz, Jackson Stahl, Fan Zhang. Arxiv 2024. [paper]

• 📊 A scalable pipeline to create synthetic datasets from functional–structural plant models for deep learning. Dirk Norbert Baker, Felix Maximilian Bauer, Mona Giraud, Andrea Schnepf, Jens Henrik Göbbert, Hanno Scharr, Ebba Hvannberg, Morris Riedel. in silico Plants, 6(1), diad022. [paper]

• 📊 A Hybrid Co-Finetuning Approach for Visual Bug Detection in Video Games. Faliu Yi, Sherif Abdelfattah, Wei Huang, A. J. Brown. AIIDE 2025. [paper]

• 📊 Adapting Agricultural Virtual Environments in Game Engines to Improve HPC Accessibility. Dirk Norbert Baker, Felix Bauer, Andrea Schnepf, Hanno Scharr, Morris Riedel, Jens Henrik Göbbert, Ebba Hvannberg. [paper]