How to use DepthCleaner class with Python

[andreaceruti]

System information (version)

• OpenCV => 4.5.5 contrib
• Operating System / Platform => Ubuntu 18.04

Detailed description

I would like to use the class in order to clean my deoth image that has some black pixels, the problem is that since documentation is lacking I am unable to use it properly.

Steps to reproduce

import cv2
import numpy as np
image = cv2.imread("depth_image_path")
depthCleaner = cv2.rgbd.DepthCleaner_create(cv2.CV_16U, 7, cv2.rgbd.DepthCleaner_DEPTH_CLEANER_NIL )
output_image = np.copy(image)
depthCleaner.apply(image, output_image)

The problem is then that my output image looks exactly as the input image, so how would I use DepthCleaner class?

If from the snippet above I run

print(dir(depthCleaner))

I see all of its method but I can't understand their connection with the official .cpp implementation since I am not confident in using cpp.
The avaliable methods are
[ 'apply', 'clear', 'create', 'empty', 'getDefaultName', 'getDepth', 'getMethod', 'getWindowSize', 'initialize', 'read', 'save', 'setDepth', 'setMethod', 'setWindowSize', 'write']

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found any solution

• I updated to the latest OpenCV version and the issue is still there

• There is reproducer code and related data files: videos, images, onnx, etc

[berak]

as it is now, it only accepts CV_16U, CV_32F or CV_64F single channel depth images, but you are reading in a 3channel bgr , it falls through here, and just does nothing.
so, asuming, your "depth_image_path" leads to a 16bit, single channel depth image, it would be:

image = cv2.imread("depth_image_path", cv2.IMREAD_UNCHANGED) # preserve 16bit
depthCleaner = cv2.rgbd.DepthCleaner_create(cv2.CV_16U, 7, cv2.rgbd.DepthCleaner_DEPTH_CLEANER_NIL )
output_image = depthCleaner.apply(image)

however, the docs / comments are indeed faulty and misleading (most of it seems copypasted from the rgbdNormals class):

opencv_contrib/modules/rgbd/include/opencv2/rgbd/depth.hpp

Line 210 in 98f6a2e

* @param method one of the methods to use: RGBD_NORMALS_METHOD_SRI, RGBD_NORMALS_METHOD_FALS

DepthCleaner::DEPTH_CLEANER_NIL is the only implemented method here

opencv_contrib/modules/rgbd/include/opencv2/rgbd/depth.hpp

Lines 218 to 219 in 98f6a2e

	/** Given a set of 3d points in a depth image, compute the normals at each point.
	* @param points a rows x cols x 3 matrix of CV_32F/CV64F or a rows x cols x 1 CV_U16S

no, it does not accept 3d point clouds, just single channel depth maps

opencv_contrib/modules/rgbd/src/depth_cleaner.cpp

Line 47 in 98f6a2e

/** Given a depth image, compute the normals as detailed in the LINEMOD paper

wrong paper

opencv_contrib/modules/rgbd/src/depth_cleaner.cpp

Line 78 in 98f6a2e

switch (depth_in.depth())

there should be a default case, throwing an error for wrong input type (maybe even check type(), not depth() ?)

opencv_contrib/modules/rgbd/src/depth_cleaner.cpp

Lines 55 to 56 in 98f6a2e

	typedef Vec<T, 3> Vec3T;
	typedef Matx<T, 3, 3> Mat33T;

hmmm, unused

opencv_contrib/modules/rgbd/src/depth_cleaner.cpp

Lines 240 to 245 in 98f6a2e

	DepthCleaner::initialize() const
	{
	if (depth_cleaner_impl_ == 0)
	initialize_cleaner_impl();
	else if (!reinterpret_cast<DepthCleanerImpl *>(depth_cleaner_impl_)->validate(depth_, window_size_, method_))
	initialize_cleaner_impl();

it does not delete an existing instance, memleak here

[andreaceruti]

@berak Thank you for your interest, I rushed a bit in writing the issue, I was reading the image depth with this flag cv2.IMREAD_ANYDEPTH, so it was reading a 16 bit single channel image.
Also looking at image values with this command from imagemagick, I saw that there was a very slight difference between original image and cleaned image. The difference remains also If I run the methods 100 times on the same image.
Thank you for the help, I will wait for any other experience with this method in this issue

[berak]

The difference remains also If I run the methods 100 times on the same image.

see Listing 1 in the paper for the idea behind it
and indeed, the difference_threshold value is hardcoded to 10, that's why subsequent 99 more iterations wont improve it