About Infrared camera of RealSense D435 and OpenCV

[Vermilion016]

I am learning Intel RealSense D435 & Visual Studio 2017.
Like the image, I would like to reproduce it with an infrared camera instead of a color camera in a program that recognizes faces with a RealSense camera and OpenCV. Is there a way?
If possible, I would appreciate it if you could also describe the program.

[MartyG-RealSense]

Hi @Vermilion016 I note that in the other case at the link below that you posted the question at, you wished to adjust the rs-dnn OpenCV example program to use infrared instead of RGB color.

https://dev.intelrealsense.com/docs/rs-dnn?_ga=2.48086116.2144105902.1660733096-1041004708.1589641371

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/dnn

Whilst it is sometimes possible to change a program to use the infrared stream instead of RGB (especially for RealSense camera module boards that do not have an RGB sensor), it may not be practical to do so for a DNN detection program. The stream type could be changed from RGB to infrared, but a DNN program relies on identification of specific types of visual elements on an RGB image using a dataset that has been trained to recognize them.

rs-dnn would also not be a suitable example program for face detection, since its dataset is not trained for face recognition. The example program rs-dlib-face which uses the 'Dlib' face recognition library is more in line with what you have in mind.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/dlib/face

There is also a RealSense face recognition example program for the OpenVINO Toolkit.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/openvino/face

---

If you would like to test making changes to rs-dnn to see whether it works with the infrared image, I would recommend the following changes to the script:

1. On line 33, change RS2_STREAM_COLOR to RS2_STREAM_INFRARED

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/opencv/dnn/rs-dnn.cpp#L33

2. Make this change again to the align instruction on line 35.

rs2::align align_to(RS2_STREAM_INFRARED);

Although depth and infrared are already aligned as they both originate from the left infrared sensor, we will change this line since the rs-dnn program calls for alignment to be performed.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/opencv/dnn/rs-dnn.cpp#L35

3. Change line 63 to auto color_frame = data.get_infrared_frame();

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/opencv/dnn/rs-dnn.cpp#L63

Ideally, we would use auto infrared_frame = data.get_infrared_frame(); but since the rs-dnn script relies on the 'color_frame' name in multiple lines, it is easier for test purposes to just change the second half of the line so that it accesses the infrared frame.

[Vermilion016]

Thank you very much.
But sorry. It was not possible.
Few days ago, it was possible to manipulate the infrared stream in the past.
I referred to the site below.
Somehow, I want to apply this to OpenCV.
Is there a good way?

[MartyG-RealSense]

There is an OpenCV example for the infrared stream at the link below.

https://github.com/IntelRealSense/librealsense/blob/master/doc/stepbystep/getting_started_with_openCV.md#displaying-infrared-frame

[Vermilion016]

Thank you very much.
It could be possible.
But sorry. That's not what I want to do.
Created with reference to the following site, I want to apply an infrared stream like an image to OpenCV.
Please give me a good link.

#1140

[MartyG-RealSense]

Has the problem in this case been solved by the resolution mentioned at #11116 (comment) please or do you still require assistance? Thanks!

[Vermilion016]

Excuse me.
I have a new problem, please help me.
This time I succeeded with the following code, but there is a big lag every 1 frame.
(The movement of the camera is absurdly heavy, and it moves only about 1 frame per second)
If anyone knows the reason, please let me know.

#include "opencv2/opencv.hpp" #include "opencv2/highgui.hpp" #include <vector> #include <librealsense2/rs.hpp> #include <iostream> #include <stdio.h> using namespace cv; using namespace std; int main(int argc, char * argv[]) { int width = 1280; int height = 720; int fps = 30; rs2::config config; config.enable_stream(RS2_STREAM_INFRARED, width, height, RS2_FORMAT_Y8, fps); rs2::pipeline pipeline; rs2::pipeline_profile pipeline_profile = pipeline.start(config); CascadeClassifier cascade; cascade.load("C:/opencv/sources/data/haarcascades/haarcascade_frontalface_alt.xml"); vector<Rect> faces; while (1) { rs2::frameset frameset = pipeline.wait_for_frames(); rs2::video_frame ir_frame = frameset.get_infrared_frame(); cv::Mat dMat_left = cv::Mat(cv::Size(width, height), CV_8UC1, (void*)ir_frame.get_data()); cascade.detectMultiScale(dMat_left, faces, 1.1, 3, 0, Size(20, 20)); for (int i = 0; i < faces.size(); i++) { rectangle(dMat_left, Point(faces[i].x, faces[i].y), Point(faces[i].x + faces[i].width, faces[i].y + faces[i].height), Scalar(0, 0, 0), 3); } imshow("img", dMat_left); const int c = cv::waitKey(1); } return EXIT_SUCCESS; }

[MartyG-RealSense]

Does performance improve if you change the FPS value from 30 to 15?

[Vermilion016]

I see. I tried lowering the fps and image quality.
Significantly reduced lag.
Thank you for your cooperation.

[MartyG-RealSense]

You are very welcome. I'm pleased that I could help. Thanks very much for the update!

[Vermilion016]

Excuse me.
Due to various circumstances, I want to reduce the time lag a little more.
Is there a better way?

[MartyG-RealSense]

If you are still using the Haar Cascade in the script above at #11113 (comment) then a cascade will tend to run slowly. Googling for the term opencv slow haar cascade provides information about this.

[Vermilion016]

I'see. I will try.
Thank you very much.