New datatype - RGBDImage #42

syncle · 2017-06-14T22:29:18Z

No description provided.

syncle · 2017-06-14T22:30:18Z

src/Core/Geometry/Image.cpp

-	}
-	return pyramid_image;
-}
-


Moving this function to ImageFactory.cpp

syncle · 2017-06-14T22:33:29Z

src/Core/Geometry/ImageFactory.cpp

+std::shared_ptr<RGBDImage> CreateRGBDImageFromTUMFormat(
+		const Image& color, const Image& depth) {
+	return CreateRGBDImageFromColorAndDepth(color, depth, 5000.0);
+}


I am not clear what CreateRGBDImageFromMITFormat should be. For MIT dataset (presumably SUN3D), The raw depth is divided by 1000, right? Then, we can use CreateRGBDImageFromColorAndDepth

No. I think they have a very special format.
If you get the short depth pixel as d.
Then the float depth is: (( d >> 3 ) | ( d << 13 )) / 1000.0.
See ln 357 - 369 in this file (my implementation):
https://bitbucket.org/qianyizh/stanfordcode/src/e10b458923689794e72d6fad5b92f379cba242b4/DepthParser/SlamApp.cpp?at=master&fileviewer=file-view-default
Or from the SUN3D toolbox, ln 196-210:
https://github.com/PrincetonVision/SUN3DCppReader/blob/master/src/sun3d_reader.cpp

There is another very strange depth format. It is the NYU data.
http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html
I forgot where I got this magic, but the correct way to parse it is in ln 333-355 this file:
https://bitbucket.org/qianyizh/stanfordcode/src/e10b458923689794e72d6fad5b92f379cba242b4/DepthParser/SlamApp.cpp?at=master&fileviewer=file-view-default
In mathematics, you first swap the high byte and the low byte of d, then you compute: 351.3 / ( 1092.5 - *d )

I confirm the magic of the NYU data.
Its matlab toolbox has this function:

% Projects the given depth image to world coordinates. Note that this 3D % coordinate space is defined by a horizontal plane made from the X and Z % axes and the Y axis points up. % % Args: % imgDepthOrig - 480x640 raw depth image from the Kinect. Note that the % bytes of the original uint16 image must have already % been swapped via swapbytes.m % % Returns: % imgDepthAbs - 480x640 depth image whose values are in meters, rather % than the internal depth values used by the kinect. function imgDepthAbs = depth_rel2depth_abs(imgDepthOrig) assert(isa(imgDepthOrig, 'double')); [H, W] = size(imgDepthOrig); assert(H == 480); assert(W == 640); camera_params; imgDepthAbs = depthParam1 ./ (depthParam2 - imgDepthOrig); imgDepthAbs(imgDepthAbs > maxDepth) = maxDepth; imgDepthAbs(imgDepthAbs < 0) = 0; end

The bytes were swapped by swapbytes.m (in its document). And the two parameters are:

depthParam1 = 351.3 depthParam2 = 1092.5

Another comment:

I think just a toolbox that can read different RGB-D image formats would already be very very helpful. I have been thinking of it for a long time.

Let's make it a function in Open3D.

There is a new issue related to this direction. See #43

I commented in #43
I think we should have a function to parse NYU format, but the image IO can be handled by Python.

syncle · 2017-06-14T22:35:58Z

src/Core/Geometry/RGBDImage.cpp

+
+Eigen::Vector2d RGBDImage::GetMaxBound() const
+{
+	return Eigen::Vector2d(color.width_, color.height_);


Not quite sure about this line. As factory function CreateRGBDImageFromColorAndDepth checks that color and depth are same in size, this function assume the size of color and depth are the same.

syncle · 2017-06-14T22:37:29Z

src/Core/Geometry/RGBDImage.h

+	bool IsEmpty() const override;
+	Eigen::Vector2d GetMinBound() const override;
+	Eigen::Vector2d GetMaxBound() const override;
+


I tried to mimic how class Image is defined. Are these lines are reasonable?

Umm, I think it should not be a Geometry2D. It is a collection of Images.
So just remove the inheritance, and remove these virtual functions.

syncle · 2017-06-15T18:55:09Z

src/Core/Geometry/RGBDImage.cpp

+
+namespace three{
+
+}	// namespace three


RGBDImage.cpp has nothing as I removed a few virtual functions. Should we keep this?

Yes, remove this file. If there is nothing.

syncle · 2017-06-15T18:56:58Z

src/Python/Core/py3d_image.cpp

+			std::to_string(rgbd_image.depth_.num_of_channels_) +
+			std::string(" channels.\n") + 
+			std::string("Use numpy.asarray to access buffer data.");
+		});


I put these interface functions to Python/Core/py3d_image.cpp for now. Do you think we need to make Python/Core/py3d_rgbdimage.cpp?

No, I think it is okay to put it under py3d_image.cpp.

syncle · 2017-06-15T19:01:39Z

src/Python/Tutorial/Basic/read_rgbd_images.py

+		count=int(width)*int(height),
+		offset=len(header)).reshape((int(height), int(width)))
+	img_out = img.astype('u2')
+	return img_out


This function is necessary for this python tutorial but it is bit hacky.

qianyizh

There are some minor formatting issues. Others are good.

I have an additional question: what if the input color image is of different size of depth image? There are some use cases:

We use Primesense to capture a higher resolution image stream (my color mapping paper).
We use RealSense DS4 to capture a higher resolution image stream.
We use occipital sensor with an iPhone camera.

We don't need to handle them immediately. But I think we should leave a note at the class definition of RGBDImage, saying that it is for a pair of registered color and depth images, viewed from the same view, of the same resolution. If you have other format, convert it first.

In the future, we can provide similar helper functions for other kinds of input (as long as the input resolution and the calibrated extrinsic matrix is given). But it is future work.

qianyizh · 2017-06-15T21:10:32Z

src/Core/Geometry/ImageFactory.cpp

+
+std::shared_ptr<RGBDImage> CreateRGBDImageFromColorAndDepth(
+		const Image& color, const Image& depth, 
+		const double& depth_scale/* = 1000.0*/, double depth_trunc/* = 3.0*/) {


Add a line break before {.
Just want to make this consistent.

qianyizh · 2017-06-15T21:11:30Z

src/Core/Geometry/ImageFactory.cpp

+}
+
+std::shared_ptr<RGBDImage> CreateRGBDImageFromTUMFormat(
+		const Image& color, const Image& depth) {


Add a line break before {.
Also, put the link to TUM dataset here.
If there is a link presenting the data format, add it too.

qianyizh · 2017-06-15T21:12:06Z

src/Core/Geometry/ImageFactory.cpp

+}
+
+std::shared_ptr<RGBDImage> CreateRGBDImageFromSUNFormat(
+		const Image& color, const Image& depth) {


Add a line break before {.

Add link to SUN3D dataset, and the link to the depth file format if there is any.

qianyizh · 2017-06-15T21:12:40Z

src/Core/Geometry/ImageFactory.cpp

+}
+
+std::shared_ptr<RGBDImage> CreateRGBDImageFromNYUFormat(
+		const Image& color, const Image& depth) {


Add a line break before {.

Add a link to NYU dataset, and the link to the depth file format if there is any.

qianyizh · 2017-06-15T21:14:50Z

src/Python/Core/py3d_image.cpp

@@ -110,6 +111,27 @@ void pybind_image(py::module &m)
 					std::to_string(img.num_of_channels_) +
 					std::string(" channels.\nUse numpy.asarray to access buffer data.");
 		});
+
+	py::class_<RGBDImage, std::shared_ptr<RGBDImage>>
+			rgbd_image(m, "RGBDImage", py::buffer_protocol());


Remove py::buffer_protocal(). It is for class bindings that require raw data access.

syncle added 5 commits June 12, 2017 17:16

same as the reviewed code

bf3278c

Merge remote-tracking branch 'upstream/master'

f912ecc

Major modifications on RGBDImage

73cd714

minor

bcb2d01

minor

3b77cdc

syncle commented Jun 14, 2017

View reviewed changes

src/Core/Geometry/Image.cpp

}

return pyramid_image;

}

Copy link

Contributor Author

syncle Jun 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving this function to ImageFactory.cpp

syncle commented Jun 14, 2017

View reviewed changes

adding examples for other datasets

3f5ffc0

syncle commented Jun 15, 2017

View reviewed changes

removing rgbd_odometry.py

b0effba

syncle commented Jun 15, 2017

View reviewed changes

removing RGBDImage.cpp

601158d

qianyizh reviewed Jun 15, 2017

View reviewed changes

fixed ver. based on review

337c534

qianyizh merged commit 2c27dde into isl-org:master Jun 15, 2017

qianyizh deleted the RGBDImage branch June 15, 2017 22:17

This was referenced Jun 16, 2017

PPM/PGM file support #43

Closed

Add functions to accept more RGB-D input formats (different resolution, different view, etc) #44

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New datatype - RGBDImage #42

New datatype - RGBDImage #42

syncle commented Jun 14, 2017

syncle Jun 14, 2017

syncle Jun 14, 2017

qianyizh Jun 14, 2017

qianyizh Jun 14, 2017

qianyizh Jun 14, 2017

syncle Jun 15, 2017

qianyizh Jun 15, 2017

syncle Jun 14, 2017

syncle Jun 14, 2017

qianyizh Jun 14, 2017

syncle Jun 15, 2017

qianyizh Jun 15, 2017

syncle Jun 15, 2017 •

edited

qianyizh Jun 15, 2017

syncle Jun 15, 2017

qianyizh left a comment

qianyizh Jun 15, 2017

qianyizh Jun 15, 2017

qianyizh Jun 15, 2017

qianyizh Jun 15, 2017

qianyizh Jun 15, 2017

New datatype - RGBDImage #42

New datatype - RGBDImage #42

Conversation

syncle commented Jun 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

syncle Jun 15, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qianyizh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

syncle Jun 15, 2017 •

edited