how to aligen Y16 infrared stream to z16 depth stream

[cdb0y511]

Stream Profiles supported by Stereo Module
Supported modes:
stream resolution fps format
Infrared 2 1280x800 @ 30Hz Y8
Infrared 1 1280x800 @ 30Hz Y8
Infrared 2 1280x800 @ 25Hz Y16
Infrared 1 1280x800 @ 25Hz Y16
Infrared 2 1280x800 @ 15Hz Y16
Infrared 1 1280x800 @ 15Hz Y16
Infrared 1 1280x800 @ 15Hz Y8
Infrared 2 1280x800 @ 15Hz Y8
Infrared 2 1280x720 @ 30Hz Y8
Infrared 1 1280x720 @ 30Hz Y8
Infrared 2 1280x720 @ 15Hz Y8
Infrared 1 1280x720 @ 15Hz Y8
Infrared 2 1280x720 @ 6Hz Y8
Infrared 1 1280x720 @ 6Hz Y8
Infrared 2 848x480 @ 90Hz Y8
Infrared 1 848x480 @ 90Hz Y8
Infrared 1 848x480 @ 60Hz Y8
Infrared 2 848x480 @ 60Hz Y8
Infrared 2 848x480 @ 30Hz Y8
Infrared 1 848x480 @ 30Hz Y8
Infrared 1 848x480 @ 15Hz Y8
Infrared 2 848x480 @ 15Hz Y8
Infrared 2 848x480 @ 6Hz Y8
Infrared 1 848x480 @ 6Hz Y8
Infrared 2 848x100 @ 100Hz Y8
Infrared 1 848x100 @ 100Hz Y8
Infrared 1 640x480 @ 90Hz Y8
Infrared 2 640x480 @ 90Hz Y8
Infrared 2 640x480 @ 60Hz Y8
Infrared 1 640x480 @ 60Hz Y8
Infrared 1 640x480 @ 30Hz Y8
Infrared 2 640x480 @ 30Hz Y8
Infrared 1 640x480 @ 15Hz Y8
Infrared 2 640x480 @ 15Hz Y8
Infrared 1 640x480 @ 6Hz Y8
Infrared 2 640x480 @ 6Hz Y8
Infrared 2 640x400 @ 25Hz Y16
Infrared 1 640x400 @ 25Hz Y16
Infrared 1 640x400 @ 15Hz Y16
Infrared 2 640x400 @ 15Hz Y16
Infrared 2 640x360 @ 90Hz Y8
Infrared 1 640x360 @ 90Hz Y8
Infrared 1 640x360 @ 60Hz Y8
Infrared 2 640x360 @ 60Hz Y8
Infrared 1 640x360 @ 30Hz Y8
Infrared 2 640x360 @ 30Hz Y8
Infrared 1 640x360 @ 15Hz Y8
Infrared 2 640x360 @ 15Hz Y8
Infrared 1 640x360 @ 6Hz Y8
Infrared 2 640x360 @ 6Hz Y8
Infrared 1 480x270 @ 90Hz Y8
Infrared 2 480x270 @ 90Hz Y8
Infrared 1 480x270 @ 60Hz Y8
Infrared 2 480x270 @ 60Hz Y8
Infrared 2 480x270 @ 30Hz Y8
Infrared 1 480x270 @ 30Hz Y8
Infrared 1 480x270 @ 15Hz Y8
Infrared 2 480x270 @ 15Hz Y8
Infrared 1 480x270 @ 6Hz Y8
Infrared 2 480x270 @ 6Hz Y8
Infrared 1 424x240 @ 90Hz Y8
Infrared 2 424x240 @ 90Hz Y8
Infrared 1 424x240 @ 60Hz Y8
Infrared 2 424x240 @ 60Hz Y8
Infrared 1 424x240 @ 30Hz Y8
Infrared 2 424x240 @ 30Hz Y8
Infrared 1 424x240 @ 15Hz Y8
Infrared 2 424x240 @ 15Hz Y8
Infrared 1 424x240 @ 6Hz Y8
Infrared 2 424x240 @ 6Hz Y8
Depth 1280x720 @ 30Hz Z16
Depth 1280x720 @ 15Hz Z16
Depth 1280x720 @ 6Hz Z16
Depth 848x480 @ 90Hz Z16
Depth 848x480 @ 60Hz Z16
Depth 848x480 @ 30Hz Z16
Depth 848x480 @ 15Hz Z16
Depth 848x480 @ 6Hz Z16
Depth 848x100 @ 100Hz Z16
Depth 640x480 @ 90Hz Z16
Depth 640x480 @ 60Hz Z16
Depth 640x480 @ 30Hz Z16
Depth 640x480 @ 15Hz Z16
Depth 640x480 @ 6Hz Z16
Depth 640x360 @ 90Hz Z16
Depth 640x360 @ 60Hz Z16
Depth 640x360 @ 30Hz Z16
Depth 640x360 @ 15Hz Z16
Depth 640x360 @ 6Hz Z16
Depth 480x270 @ 90Hz Z16
Depth 480x270 @ 60Hz Z16
Depth 480x270 @ 30Hz Z16
Depth 480x270 @ 15Hz Z16
Depth 480x270 @ 6Hz Z16
Depth 424x240 @ 90Hz Z16
Depth 424x240 @ 60Hz Z16
Depth 424x240 @ 30Hz Z16
Depth 424x240 @ 15Hz Z16
Depth 424x240 @ 6Hz Z16

---

Required Info
Camera Model	{ D435 }
Firmware Version	(05.11.15.00)
Operating System & Version	{Win (10) / Linux (Ubuntu 18)
Kernel Version (Linux Only)	(4.15.13)
Platform	PC/.
SDK Version	{ 2.. }
Language	{C/C#/labview/nodejs/opencv/pcl/python/unity }
Segment	{Robot }

Issue Description

how to aligen Y16 infrared stream to z16 depth stream
I need Y16 infrared stream cause the Y8 can not reach my needs. However I realize the D435i only support the 1280x800 Y16 infrared, and the depth stream does not support this resolution. I think the match algorithm can not work on this resolution, it is written on the VPU. So how to match the 1280x800 infrared stream to 1280x700 depth stream, I need them both.
Any suggestion?

[cdb0y511]

I mean match, each left infrared stream pixel have a known depth from depth stream.

[MartyG-RealSense]

I hope that the link below will be helpful in explaining why Y16 does not operate when a depth stream is used.

https://forums.intel.com/s/question/0D70P0000068efSSAQ/stream-depth-and-y16-simultaneously?language=en_US

[cdb0y511]

I hope that the link below will be helpful in explaining why Y16 does not operate when a depth stream is used.

https://forums.intel.com/s/question/0D70P0000068efSSAQ/stream-depth-and-y16-simultaneously?language=en_US

thanks, I think this post describe almost the same question.
we do need higher IR stream sampling rate (Y16).
But simultaneous depth stream is not necessary. We want to capture the depth from 1280x720. Then capture the 1280x800 IR from same scene. And align IR 1280x800 to 1280x720. Is there any method or suggestion to make the alignment good enough?

[dorodnic]

This functionality is not provided by the SDK, but if I had to do it, I'd be looking at the calibration data (inside librealsense device) - ds5-private.h:340-347

[ev-mp]

@cdb0y511 , additionally - while the requested mode is not supported by D435 it is feasible with D415.
With v2.21 release a new 10bit infrared streaming format (Y10BPACK) was introduced for D415 device that allows to stream 1920X1080 rectified IR frames of the left IR sensor and 720p Depth frames simultaneously. The 10 bits are the native sensitivity of the IR sensor, so this streaming format will exhibit the sensor at its full capacity.

[cdb0y511]

@ev-mp is it possible to let D435i get upgrades too? Or I just give up, the Y16 720p can be used as a single stream for IR signal measurement . But as mentioned before, this model may need extra calibration. Any suggestion for the calibration? @dorodnic @MartyG-RealSense
Thanks.

[MartyG-RealSense]

@cdb0y511 D415 and D435 have different IR imager components with differing capabilities. For example, D415 can stream color from the left IR imager but the left and right IR imagers on D435 are monochrome. The D435 IR imager is also wider and lets more light into it.

If you believe your camera needs calibrating, you can use the 'Dynamic Calibrator' tool to do so.

https://downloadcenter.intel.com/download/28517/Intel-RealSense-D400-Series-Calibration-Tools-and-API

If you are using Linux, please see page 14 of the tool's user guide for installation instructions.

https://www.intel.co.uk/content/www/uk/en/support/articles/000026723/emerging-technologies/intel-realsense-technology.html

[cdb0y511]

@MartyG-RealSense
thanks,
I wonder if the Dynamic Calibrator tool is only useful to calibrate the depth calculation.
"Intel advises against using Y16 in an application because it is unrectified and intended for use in camera calibration." In the post, what do you mean Y16 model is not rectified?
I guess

1. it is not align with the depth map.
2. Y16 model has serious distortion.
3. the intensity is not correct in this Y16 model due to the ADC sampling rate.

Some of them can be calibrated. But some of them can not.

[MartyG-RealSense]

When data is captured by the camera, a circuit board inside it called the Vision Processor D4 (a separate component from the camera board) will usually do 'rectification' calculations on the data that alter it before sending it onward through the USB cable. Y16 mode does not perform these changes though.

Information about the processes that the Vision Processor D4 performs on the data can be found here:

https://www.stemmer-imaging.com/media/uploads/cameras/13/136603-Intel-RealSense-D4-Vision-Processor-Brochure.pdf

[RealSenseCustomerSupport]

---

@cdb0y511 Any other questions about this? Looking forward to your update. Thanks!

@MartyG-RealSense Thanks very much for your input which helps us a lot!

[RealSenseCustomerSupport]

---

@@cdb0y511 If no other questions about this, will close it accordingly. Thanks!