Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to eval on MiDaS? #129

Closed
guangkaixu opened this issue Sep 27, 2021 · 7 comments
Closed

How to eval on MiDaS? #129

guangkaixu opened this issue Sep 27, 2021 · 7 comments

Comments

@guangkaixu
Copy link

guangkaixu commented Sep 27, 2021

Hi, is there any evaluation code for MiDaS?

MiDaS can predict a robust inverse-depth of a single image, but how can I eval on datasets with ground truth depth like KITTI? Should I convert predict disparity to depth and evaluate in the depth space, or convert ground truth depth to disparity and evaluate in the disparity space?

I downloaded the official validation set of KITTI, and convert gt_depth to gt_disparity with:

gt_disparity = 1 / (gt_depth + 1e-8)
gt_disparity[gt_depth==0]=0

after performing alignment in disparity space and evaluating on midas_v3.0_dpt-large on KITTI, I got the performance:

KITTI AbsRel : 10.0
KITTI delta > 1.25 : 10.1

It seems that my evaluation code is not so accurate. Could you please provide your evaluation code for MiDas? Thank you so much.

@AlexeyAB
Copy link
Contributor

AlexeyAB commented Sep 29, 2021

@guangkaixu Hi,
To evaluate MiDaS v3.0 (DPT-Hybrid) on NYU and Kitti depth datasets using Eigen-Split, use this manual and this repository: https://github.com/isl-org/DPT/blob/main/EVALUATION.md

@guangkaixu
Copy link
Author

@AlexeyAB Thank you for your reply.
The evaluation code of DPT is only suitable for metric depth prediction, but MiDaS can predict monocular disparity without scale and shift. What's more, the disparity can't recover scale and shift with datasets with gt_depth like KITTI directly, so we would like to know that how can disparity prediction recover scale and shift with gt_depth:

  1. Convert pred_disp to pred_depth.
pred_depth = 1 / (pred_disp + 1e-8) 
pred_depth[pred_disp <= 0] = 0
pred_depth_metric = recover_scale_shift(pred_depth, gt_depth)
error = evaluate_error(pred_depth_metric, gt_depth)
  1. Convert gt_depth to gt_disp.
gt_disp = 1 / (gt_depth + 1e-8) 
gt_disp[gt_depth <= 0] = 0
pred_disp_metirc = recover_scale_shift(pred_disp, gt_disp)
error = evaluate_error(pred_disp_metirc, gt_disp)
  1. Other evaluation methods...

@ranftlr
Copy link
Collaborator

ranftlr commented Sep 29, 2021

The gists linked in the section "General-Purpose models" in https://github.com/isl-org/DPT/blob/main/EVALUATION.md show how this evaluation is done.

@guangkaixu
Copy link
Author

Oh I see, thanks a lot!

@ghost
Copy link

ghost commented Oct 14, 2021

I have a question here:
what is the disparity unit?

@dreamlychina
Copy link

Oh I see, thanks a lot!

你好,大佬问下,最后模型的输出要怎么转换才能变成米制单位的??模型最后的输出范围太大了。

@guangkaixu
Copy link
Author

Oh I see, thanks a lot!

你好,大佬问下,最后模型的输出要怎么转换才能变成米制单位的??模型最后的输出范围太大了。

The MiDaS can only predict affine-invariant disparity, which contains an unknown scale and an unknown shift value in the disparity area. You can align scale-shift values with np.polyfit(pred_disp, gt_disp), where gt_disp = 1 / gt_depth. Remember mask the invalid gt_depth :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants