Monocular Depth estimation on UE4 and UE5

Example UE project for depth estimation using a single RGB camera.
See ue5 branch to use with UE5.
(日本語の説明はこちら)

NNEngine-demo-depth-estimation-github.mp4

(DPT model demo)

DPT.mp4

Environment

• OS: Windows 10 64bit
• Unreal Engine: 4.26.2
• NNEngine plugin

Download

Please download from the release page.

Run the demo

1. Extract the downloaded zip file and double-click MonoDepthEstimation.uproject.
2. After launching, click Play on the editor to start the demo that performs AI estimation for the pre-recorded video.
3. To run on your webcam, specify the webcam you want to use in /Content/Common/MediaPlayer_webcam.uasset.

Step-by-step guide to implementation

Additional models

Large model

If you want to use the large model (whose input image size is 384x384), download from here (GitHub Release page) or here (Google Drive) and place it under Source\ThirdParty\Models.

DPT model

To use DPT model, download model from here (GitHub Release page) or here (Google Drive) and place it under Source\ThirdParty\Models. Then, open /Content/DepthEstimation/DPT/test_DPT.map and click Play.

TCMonoDepth model

To use TCMonoDepth model, download model from here (GitHub Release page) or here (Google Drive) and place it under Source\ThirdParty\Models. Then, open /Content/DepthEstimation/TCMonoDepth/test_TCMonoDepth.map and click Play.

Display camera image to the preview mesh

By switching the base color pin of /Content/DepthEstimation/Grayscale_WPO.uasset, you can display camera image to the preview mesh.

Model details

See the following pages for the details of the model used in this project.

Default and large models

• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
• GitHub

DPT model

• Vision Transformers for Dense Prediction
• GitHub

Converted to ONNX by

• PINTO_model_zoo

TCMonoDepth model

• Enforcing Temporal Consistency in Video Depth Estimation
• GitHub