cv_bridge: CompressedImage conversions with 16bit grayscale images

[NikolausDemmel]

I have 16bit grayscale images (mono16) and would like to convert to and from CompressedImage. It seems cv_bridge is not properly handling this case in most instances.

Possibly related: image_transport also has issues with decoding 16bit grayscale: ros-perception/image_common#77

Python

It looks like cv2_to_compressed_imgmsg works correctly and creates 16bit grayscale png images.

However, compressed_imgmsg_to_cv2 has issues. Firstly, imdecode is given only IMREAD_ANYCOLOR, which means the image is always converted to 8 bit. I think IMREAD_UNCHANGED is what we want here, or at least IMREAD_ANYCOLOR | IMREAD_ANYDEPTH (not sure what the difference is between these two cases). Secondly, if a specific encoding is asked for, it later assumes that the result from imdecode is bgr8 encoded. Not sure about the color encoding you can expect from imdecode, but at least for grayscale it seems wrong, and the actual bitdepth of the input would ideally also be respected here.

C++

I didn't try it for C++, but looking at the code, it seems that for neither conversion 16bit is supported.

vision_opencv/cv_bridge/src/cv_bridge.cpp

Lines 480 to 506 in 9d49971

	void CvImage::toCompressedImageMsg(sensor_msgs::CompressedImage& ros_image, const Format dst_format) const
	{
	ros_image.header = header;
	cv::Mat image;
	if (encoding == enc::BGR8 || encoding == enc::BGRA8)
	{
	image = this->image;
	}
	else
	{
	CvImagePtr tempThis = boost::make_shared<CvImage>(*this);
	CvImagePtr temp;
	if (enc::hasAlpha(encoding))
	{
	temp = cvtColor(tempThis, enc::BGRA8);
	}
	else
	{
	temp = cvtColor(tempThis, enc::BGR8);
	}
	image = temp->image;
	}
	std::string format = getFormat(dst_format);
	ros_image.format = format;
	cv::imencode("." + format, image, ros_image.data);
	}

vision_opencv/cv_bridge/src/cv_bridge.cpp

Lines 515 to 536 in 9d49971

	CvImagePtr toCvCopy(const sensor_msgs::CompressedImage& source, const std::string& encoding)
	{
	// Construct matrix pointing to source data
	const cv::Mat_<uchar> in(1, source.data.size(), const_cast<uchar*>(&source.data[0]));
	// Loads as BGR or BGRA.
	const cv::Mat rgb_a = cv::imdecode(in, cv::IMREAD_UNCHANGED);
	switch (rgb_a.channels())
	{
	case 4:
	return toCvCopyImpl(rgb_a, source.header, enc::BGRA8, encoding);
	break;
	case 3:
	return toCvCopyImpl(rgb_a, source.header, enc::BGR8, encoding);
	break;
	case 1:
	return toCvCopyImpl(rgb_a, source.header, enc::MONO8, encoding);
	break;
	default:
	return CvImagePtr();
	}
	}

[NikolausDemmel]

Just for the record:

Firstly, imdecode is given only IMREAD_ANYCOLOR, which means the image is always converted to 8 bit. I think IMREAD_UNCHANGED is what we want here, or at least IMREAD_ANYCOLOR | IMREAD_ANYDEPTH (not sure what the difference is between these two cases).

Still not sure, but the difference might be that ANYDEPTH | ANYCOLOR possibly flattens the alpha channel.

Secondly, if a specific encoding is asked for, it later assumes that the result from imdecode is bgr8 encoded. Not sure about the color encoding you can expect from imdecode, but at least for grayscale it seems wrong, and the actual bitdepth of the input would ideally also be respected here.

According to the documentation, imdecode returns either single channel for grayscale or 3-channel BGR for color images (or possibly BGRA for png with alpha channel)

[awesomebytes]

I needed to use IMREAD_UNCHANGED for a 16bit CompressedImage from compressedDepth topic.

[NikolausDemmel]

What happens with IMREAD_ANYCOLOR | IMREAD_ANYDEPTH instead?

[awesomebytes]

Dunno, I don't have the code handy to try. I'll try to remember to give it a try whenever I cross paths with this again. It was quite a pain to get it right.

[NikolausDemmel]

Ah ok, don't worry about it. I thought you did this just now. Anyway, thanks for the input.

[samarth-robo]

What happens with IMREAD_ANYCOLOR | IMREAD_ANYDEPTH instead?

@NikolausDemmel I just tried it. IMREAD_ANYCOLOR | IMREAD_ANYDEPTH works.
So do IMREAD_GRAYSCALE | IMREAD_ANYDEPTH, and IMREAD_UNCHANGED.