Getting depth from the camera/aligned_depth_to_color/image_raw/compressed topic

[rakshith95]

Hello,
I've seen this issue or some variant of this pop up multiple times, but I don't think any of them have a solution. Is it possible to get the depth from the compressed topic, or is it possible only from the raw topic?

[MartyG-RealSense]

Hi @rakshith95 Once a depth topic (including compressed ones) is published then typically a script would be used to subscribe to it and retrieve the distance values, like in #1790 and in #2140 (which demonstrates subscribing to /camera/depth/image_rect_raw/compressed to obtain distance).

[rakshith95]

Hello @MartyG-RealSense , issue #1790 is for the raw depth topic, which is trivial, but I'm trying to get it to work with the camera/aligned_depth_to_color/image_raw/compressed topic, in C++. #2140 seems to be dealing with a similar issue, but provides no solution.
I use image_transport for subscribing to the topic, and perform the following (which works for camera/aligned_depth_to_color/image_raw):

cv_bridge::CvImageConstPtr cv_ptrD; try { cv_ptrD = cv_bridge::toCvShare(msgD); } catch (cv_bridge::Exception& e) { ROS_ERROR("cv_bridge exception: %s", e.what()); return; }

But I get the error: cv_bridge exception: Image is wrongly formed: step < width * byte_depth * num_channels or 848 != 848 * 2 * 1. Is the topic just unusable then?

[MartyG-RealSense]

My knowledge of using ROS compressed depth topics is limited, unfortunately. A discussion in the link below suggests using image_transport when accessing compressed depth with OpenCV, which you are already doing.

https://answers.ros.org/question/276352/how-to-understand-data-from-the-image_rawcompressed-topic/?answer=276385#post-id-276385

It is notable that the advice in that case links to compressed_image_transport, a plugin package for image_transport, as Intel provide official ROS compressed topic documentation at the link below that also recommends having compressed_image_transport installed.

https://dev.intelrealsense.com/docs/ros-wrapper#section-compression-packages

[rakshith95]

Hello, I do have the compressed_image_transport plugin installed, so I don't think that's the problem. Can you link me to the file where the compressed depth image is created and published? Maybe I can find or work with something there.

[MartyG-RealSense]

Are you using ROS1 or ROS2 please? #2140 (comment) mentions that there have been problems with accessing compressed topics in ROS2 Foxy in general, not just with RealSense.

I'm not involved in ROS wrapper development but I cannot see any code in the main wrapper files (base_realsense.node.cpp and realsense_node_factory.cpp) that is involved in compression. It is likely handled by ROS itself or the transport plugins.

[rakshith95]

@MartyG-RealSense I'm using ROS1. I recorded some data with the compressed topics, and I've been trying to get it to work ever since. If you have any suggestions, it'd be super helpful

[MartyG-RealSense]

The developer of the ROS1 wrapper posted at #714 (comment) a Python node script that they created called my_depth_subscriber.py that retrieves the distance by subscribing to the aligned topic /camera/aligned_depth_to_color/image_raw

The script was posted as a text file, so I have put the script below in case you cannot access the file.

---

#!/usr/bin/env python`
import rospy
from cv_bridge import CvBridge, CvBridgeError
from sensor_msgs.msg import Image
import numpy as np

def convert_depth_image(ros_image):
    bridge = CvBridge()
     # Use cv_bridge() to convert the ROS image to OpenCV format
    try:
     #Convert the depth image using the default passthrough encoding
        depth_image = bridge.imgmsg_to_cv2(ros_image, desired_encoding="passthrough")
        depth_array = np.array(depth_image, dtype=np.float32)
        center_idx = np.array(depth_array.shape) / 2
        print ('center depth:', depth_array[center_idx[0], center_idx[1]])

    except CvBridgeError, e:
        print e
     #Convert the depth image to a Numpy array


def pixel2depth():
	rospy.init_node('pixel2depth',anonymous=True)
	rospy.Subscriber("/camera/aligned_depth_to_color/image_raw", Image,callback=convert_depth_image, queue_size=1)
	rospy.spin()

if __name__ == '__main__':
	pixel2depth()

---

Could you try changing the subscribed topic in that script to /camera/aligned_depth_to_color/image_raw/compressed/ to see whether it produces the desired distance values, please.

[rakshith95]

I think I tried this script before with the compressed topic and it didn't work, but I don't remember exactly. Will try it again tomorrow and let you know

[rakshith95]

But even my program works perfectly for the /camera/aligned_depth_to_color/image_raw but not for the /camera/aligned_depth_to_color/image_raw/compressed with image transport. Is there any record of either any developer in Intel, or anyone online having successfully worked with the /camera/aligned_depth_to_color/image_raw/compressed topic?

[rakshith95]

Hello @MartyG-RealSense , I tried the script with the compressed image topic, and compressedimage message type, with bridge.compressed_imgmsg_to_cv2, but the depth values are capped at 255 (0.255 m) which suggests that it only contains 8 bit values.

[MartyG-RealSense]

In another case where a RealSense user was using cv2 to get depth values and was restricted to 255, a RealSense team member suggested code at IntelRealSense/librealsense#6101 (comment) for converting to CV_32FC1

You could try editing that script to convert to CV_16UC1 instead. Though the linked to script was C++ so it would need to be converted to Python for use in a Python node script in ROS.

[rakshith95]

@MartyG-RealSense I think I figured out the issue. The format of the compressed topic is: "16UC1; jpeg compressed " and for some reason 16 bit jpeg does not work as intended with Cv bridge.
For eg, if I run the following script:

from cv_bridge import CvBridge
import numpy as np

im0 = np.empty(shape=(100, 100), dtype=np.uint16)
im0[:] = 2500 # 2.5m
print("original:", np.max(im0), im0.dtype)

msg = CvBridge().cv2_to_compressed_imgmsg(im0, dst_format = "png")

im1 = CvBridge().compressed_imgmsg_to_cv2(msg)
print("converted:", np.max(im1), im1.dtype)
print("match?", np.all(im0==im1))

I get the output:

original: 2500 uint16
converted: 2500 uint16
match? True

But if I change the dst_format to "jpeg" instead of "png", I get:

original: 2500 uint16
converted: 255 uint8
match? False

[MartyG-RealSense]

Great to hear that you made significant progress. Thanks so much for sharing your code!

[rakshith95]

Unfortunately, this progress only leaves me with more questions xD. Do you have any idea on whether compressed depth data saved in the jpeg format is useable?

[rakshith95]

On this note, why is it that the depth/compressed topic messages are stored in the jpeg format by realsense-ros, and not the png format?

[MartyG-RealSense]

My understanding of the difference between jpg and png in compression is that jpg has 'lossy' compression (where you lose some quality due to loss of data in exchange for the benefit of smaller data size) whilst png is 'lossless' and so has higher quality but at the cost of a larger data size.

In regard to RealSense specifically, I am not familiar with the principles of how compression is handled in the RealSense ROS wrapper though, so cannot offer advice about its format.

[rakshith95]

Is there someone who can comment on it? AFAIK there is no support for 16-bit jpeg compression, so this is all the more confusing.

[rakshith95]

@MartyG-RealSense any information on where and how the compressed image topics are published would also be super helpful in understanding this further, if you can point me to it.

[MartyG-RealSense]

As far as I am aware, compressed depth topics are generated when ROS compressed transport plugins are installed and compression is not generated by code in the RealSense ROS wrapper itself. The section of a transport tutorial linked to below may be helpful in explaining how compression is produced in ROS outside of the wrapper.

https://github.com/ros-perception/image_transport_tutorials#adding-new-transports

[MartyG-RealSense]

Hi @rakshith95 Do you require further assistance with this case, please? Thanks!

[rakshith95]

Hello
There is still no solution to this, and it seems the topic is unuseable; however there is no confirmation whether or not it is so.

[MartyG-RealSense]

There is not any further information that I can add to the references provided above, unfortunately.

[rakshith95]

Ok can you keep the issue open then, since this seems to be still an open question on whether or not it could work?

[MartyG-RealSense]

I added a wait to close label to the case as a reminder to keep it open.

[zhangbaozhe]

I encountered the same issue. I am using C++, ROS noetic, and RealSense D435. In my program, I use a synchronizer to subscribe 4 topics camera/color/image_raw/compressed, camera/color/camera_info, camera/aligned_depth_to_color/image_raw/compressed and camera/aligned_depth_to_color/camera_info.

When I tried to get the distance from topic,

1. I used a pointer DEPTH_IMAGE with the type of sensor_msgs::CompressedImagePtr to store the image;
2. I passed DEPTH_IMAGE to cv_bridge using the code like cv_ptr = cv_bridge::toCvCopy(DEPTH_IMAGE, "16UC1");
3. I tried to get the distance by using the code like cv_ptr->image.at<ushort>(cv::Point(x, y)) where x and y are the pixels.

Most of the cases, I get the value 65535 (which is the max number of ushort). I don't know how can I get the correct depth from this topic.

Thanks

[rakshith95]

@zhangbaozhe I was doing something very similar. I was using a synchronizer for the compressed image, and compressed depth. Unfortunately, I think (90% sure) that the 16 bit jpeg compression cannot be used, and the best you can get is 8 bit values which have a max value of 255 corresponding to 0.255m .
The alternative is to use the /camera/aligned_depth_to_color/image_raw/compressedDepth topic which uses 16 bit png compression which works fine, but is at a much lower fps. It doesn't help me since I already recorded data with the compressed topic, but if you haven't maybe it helps you.

[MartyG-RealSense]

Thanks very much @rakshith95 for your advice to @zhangbaozhe :)

[zhangbaozhe]

@rakshith95 Thanks for the information. I tried the 8-bit way but it is too small. I haven't tried the /camera/aligned_depth_to_color/image_raw/compressedDepth topic yet, but I think I won't, because my program requires higher fps. Now I just choose the raw image topic ... Thanks again for the help.

[MartyG-RealSense]

Hi @zhangbaozhe Do you require further assistance with your compression problem, please? Thanks!

[MartyG-RealSense]

Hi @rakshith95 Did you make any progress with your compression problems, please?

[rakshith95]

@MartyG-RealSense No, unfortunately. I had to give up on using the data, and will have to re-capture with the /.../compressedDepth topic rather than the /.../compressed topic.

[MartyG-RealSense]

Thanks very much @rakshith95 for the update.

[MartyG-RealSense]

Case closed due to change of approach.

[Szy20000721]

我遇到了同样的问题。我正在使用C++，ROS noetic和RealSense D435。在我的程序中，我使用同步器订阅 4 个主题、 和 。camera/color/image_raw/compressed``camera/color/camera_info``camera/aligned_depth_to_color/image_raw/compressed``camera/aligned_depth_to_color/camera_info

当我试图与主题保持距离时，

1. 我使用了一个类型的指针来存储图像;DEPTH_IMAGE``sensor_msgs::CompressedImagePtr
2. 我使用这样的代码传递给cv_bridgeDEPTH_IMAGE``cv_ptr = cv_bridge::toCvCopy(DEPTH_IMAGE, "16UC1");
3. 我试图通过使用诸如像素的位置和位置之类的代码来获取距离。cv_ptr->image.at<ushort>(cv::Point(x, y))``x``y

大多数情况下，我得到的值（这是最大数量）。我不知道如何从这个主题中获得正确的深度。65535``ushort

谢谢
i also meet this problem ,i want get distance from camera/aligned_depth_to_color/image_raw/compressedthis topic,and i use cv_ptr->image.at<ushort>(cv::Point(x, y))``x``y/1000 to get the distacne ,but always give me 1.00000.
however when i use the topic ``camera/aligned_depth_to_color/image_raw" ,it can give me the correct distance

[rakshith95]

@zhangbaozhe I was doing something very similar. I was using a synchronizer for the compressed image, and compressed depth. Unfortunately, I think (90% sure) that the 16 bit jpeg compression cannot be used, and the best you can get is 8 bit values which have a max value of 255 corresponding to 0.255m . The alternative is to use the /camera/aligned_depth_to_color/image_raw/compressedDepth topic which uses 16 bit png compression which works fine, but is at a much lower fps. It doesn't help me since I already recorded data with the compressed topic, but if you haven't maybe it helps you.

@Szy20000721 maybe the above helps.