ROS2 rectified images don't seem to have the correct intrinsic values

[mpickett-bdai]

Hi,

I'm using the ROS2 version of the driver for an N36 Kit IR. I've noticed that the rectified image topics have a camera info that has default values for all the camera info variables except for the P matrix, but it seems that even the intrinsic values for the P matrix are incorrect. When used for target detection the poses are significantly off in terms of distance. I did some more experimenting and pulled off images using the built in ROS2 image viewer and it seems that the width and height of the rectified images are downsampled images of
580x364 which differs from the camera info that says it's 1935x1215.

I rerun my own intrinsic calibration on these downsampled images and found that I get a intrinsics calibration that is accurate enough for target pose detection. Is there something I'm doing that is causing this downsampling or wrong camera info? Also can you confirm that the pointclouds coming from the driver are using the rectified left sensor as the origin?

Thank You!

[saierd]

Unfortunately this is hard to test for me, because I don't have a working ROS setup at the moment and my colleague who usually does the ROS topics is on vacation. I will try to answer some of your questions theoretically for now.

Also can you confirm that the pointclouds coming from the driver are using the rectified left sensor as the origin?

Yes, that is correct.

downsampled images of 580x364 which differs from the camera info that says it's 1935x1215.

You probably have binning or an AOI enabled? It is expected and according to the documentation of sensor_msgs/CameraInfo that the resolution (as well as the projection matrices) are always the full one without binning or AOI. Those are specified separately in the binning and roi fields.

The actual projection matrices to use for a specific image are then computed by image_geometry::PinholeCameraModel using all of this information.

default values for all the camera info variables except for the P matrix

The code filling in the info is in StereoCamera::fillCameraInfoFromNxLib. This is on purpose for the rectified images. It makes sense to me on a first glance, since D, K and R should only be used to rectify the images and P then contains the calibration of the rectified image.

[mpickett-bdai]

Thanks for the quick response, @saierd. I currently have the binning set to as low as I think I can. Is there a specific setting in NXView for AOI? The CameraInfo shows no ROI change or binning. Below is the camera info for one of the rectified streams.

header:
  stamp:
    sec: 1718059918
    nanosec: 82046509
  frame_id: optical_frame_232508
height: 1216
width: 1936
distortion_model: plumb_bob
d:
- 0.0
- 0.0
- 0.0
- 0.0
- 0.0
k:
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
r:
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
p:
- 1670.5164794921875
- 0.0
- 657.8764038085938
- 0.0
- 0.0
- 1670.5164794921875
- 613.2181396484375
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
binning_x: 1
binning_y: 1
roi:
  x_offset: 0
  y_offset: 0
  height: 1215
  width: 1935
  do_rectify: true

When looking at the camera settings json that I am loading in, I don't see anything in particular in the camera settings that specifies the AOI of the 580x364 that I've been seeing. For both left and right the current settings are:

      "ForcedRawImageSize": [
		2147483647,
		2147483647
	],
	"ForcedRectifiedImageSize": [
		2147483647,
		2147483647
	],
	"RawAoiIncrements": {
		"Height": 1,
		"Left": 1,
		"Top": 1,
		"Width": 1
	},
	"RectifiedAoiIncrements": {
		"Height": 1,
		"Left": 1,
		"Top": 1,
		"Width": 1
	},

[mpickett-bdai]

Hi!

I just found what is causing my issue. We were dropping the stereo matching volume scale down to help with frame rate. Reading the tool tip it seems that the rectified image is down sampled for the stereo matching, which make sense from a scaling/speed point of view. It seems that the returned rectified image is this down sampled image.

Is there anyway to return the original rectified image rather than this downsampled version so the camera info properly matches? Or can the camera info be set to match the downsampled rectified image?

Thank you

[saierd]

Ah that makes sense. Scaling scales the rectified images during rectification already. That's how it reduces the data.

This information is missing from the CameraInfo and it seems like there is no place to put it. It has the same effect on the camera matrix as binning (just inverted), but the binning fields are integers. We might have to apply it to the P matrix before sending it out.

Getting the full size image is possible in the NxLib API by rectifying twice, but not so easy in ROS, since each RequestData action captures a new image. Maybe you can try to improve the frame rate in some other way as a workaround? Binning and AOI also reduce the computation time and should be properly accounted for in the calibration. In contrast to scaling they also improve the image transmission times in addition to the computation, so using those instead of scaling might actually be faster.