Poor point cloud quality with RGB-D input

[hanyangyu1021]

Hi,

Thanks for your excellent work on this project!

I'm currently testing mapanything with RGB-D images. When I input only the RGB image, the results look great (as shown in the first image). However, when I also provide the depth map and intrinsics following the multimodal code in the README, the resulting point cloud quality is poor and seems to be hugely influenced by imperfections in the depth map.
Is the reconstructed quality largely impacted by the depth map?

[Nik-V9]

Hi, Thanks for trying it out! I think you forgot to share the screenshots, could you do that?

Also can you share one visual of the RGB-D frame or maybe some sample data?

[MutimirStajdohar]

Hi, I also tried giving depth images from an outside source. I generated the view's dictionary in this format.

    {
        "img": images, # (H, W, 3) - [0, 255]
        "depth_z": depth_z, # (H, W)
        "is_metric_scale": torch.tensor([False], device=device), # (1,)
    }

I used

pred = model.forward(views) 
outputs = postprocess_model_outputs_for_inference(
        raw_outputs=preds,
        input_views=views,
        apply_mask=True,
        mask_edges=True,
        edge_normal_threshold=10.0,         #5.0 default
        edge_depth_threshold=0.06,       #0.03 default
        apply_confidence_mask=False,
        confidence_percentile=20,     #10 default
    )

Since I didn't want to provide any ray directions or camera intrinsics and the read me suggested using the forward call instead.
But i noticed that the result of the forward call doesn't differ much when the depth isn't provided like the model is ignoring it internally.

Is that the expected behaviour and does it make sense to provide depth information without camera intrinsics?

[Nik-V9]

Hi, there are two things which are causing the provided depth to be not used:

1. The default configuration for the model is set to only consider image inputs. As done in our infer call, you need to update this configuration for the model to start considering additional inputs. See for more details:

   map-anything/mapanything/models/mapanything/model.py

   Line 2082 in 5ffcb61

   # Set the model input probabilities based on input args for ignoring inputs

2. The forward call expects depth_along_ray as input. Please see the docs for more details:

   map-anything/mapanything/models/mapanything/model.py

   Line 1490 in 5ffcb61

   Args:

We believe that a depth image doesn't really make sense with out a corresponding camera projection model. For example, this would be the case for any RGB-D sensor or even if you project LiDAR points to an image plane (can also be a dummy virtual camera). Furthermore, even all the recent monodepth predictors, predict both camera calibration and Z depth, which can be used to construct depth along the ray.