get_infrared_frame python wrapper

[TheMikeyR]

---

Required Info
Camera Model	D415
Firmware Version	05.08.15.00
Operating System & Version	Linux (Ubuntu 16.04.4)
Kernel Version (Linux Only)	4.14.13-041413-generic
Platform	PC
SDK Version	v2.10.0
Language	Python

Issue Description

Any plan on adding get_infrared_frame to the python wrapper or there any workaround to get both infrared streams? Currently I'm able to get one of the infrared stream using the frameset.first(rs.stream.infrared) call.

I suspected that it should be available in the python wrapper but can't seem to find it anywhere https://github.com/IntelRealSense/librealsense/blob/development/wrappers/python/python.cpp#L419

Currently you are able to call get_infrared_frame(index) in cpp according to #1140 (comment)
I can see the function is defined here

librealsense/include/librealsense2/hpp/rs_frame.hpp

Line 691 in 9437bc2

video_frame get_infrared_frame(const size_t index = 0) const

It would be nice with the same functionality to python.

[zivsha]

Hi @TheMikeyR ,
Yes we can add it, in fact you can also add it :)
Currently what you can do is simply iterate over the framset:

for frame in pipe.wait_for_frames():
    print frame.profile

EDIT:
maybe just to make it a bit clearer:

for frame in pipe.wait_for_frames():
    print frame.profile.stream_type(), frame.profile.stream_index()

[TheMikeyR]

@zivsha thanks, i've just realised when recording data from realsense-viewer that it only record one infrared stream even though both is activated. Not sure if that is intended or bug? I've tried to enable both infrared and no depth, record, load bag file and post index like above and it only shows index 2.

If I enable both streams from camera and use the above, I get both index 1 and 2.

I might add get_infrared_stream if I can figure out how, I'm not the strongest in cpp yet.

[zivsha]

i've just realised when recording data from realsense-viewer that it only record one infrared stream even though both is activated. Not sure if that is intended or bug?

Can you please elaborate on this? The viewer should (and as far as I can see, does) record both infrared streams. This works for me with or without depth stream. Can you please make sure that if you open the file with the viewer, all streams are available?

Regarding infrared index - The index might also be 0 sometimes, depending on which streams are active. If you run rs-enumarate-devices you can see that some stream profiles have index 0 (no index will appear) and some have 1 or 2.

Regarding adding the function to python, pybind11 did a truly amazing job to make it simple to bind cpp code to python, so to add binding to that function you just have to add the following line

        .def("get_infrared_frame", &rs2::frameset::get_infrared_frame, "index"_a = 0)

next to:

librealsense/wrappers/python/python.cpp

Line 419 in 5121ab0

.def("get_color_frame", &rs2::frameset::get_color_frame)

[TheMikeyR]

@zivsha oh yeah my bad, they are both being recorded (when I open it in the viewer all streams are available).

That seems extremely easy! I will try it out, thanks 👍

[TheMikeyR]

It worked wonders, thanks @zivsha I made a PR to get it added. For others who are coming across this, here is an example on how to get each IR stream.

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.infrared, 1, 1280, 720, rs.format.y8, 30)
config.enable_stream(rs.stream.infrared, 2, 1280, 720, rs.format.y8, 30)
pipeline.start(config)
while True:
    frames = pipeline.wait_for_frames()
    ir1_frame = frames.get_infrared_frame(1) # Left IR Camera, it allows 0, 1 or no input
    ir2_frame = frames.get_infrared_frame(2) # Right IR camera

[zivsha]

Awesome, glad to hear it all works.

[Woodstock94]

Hi all, I just came up into this issue.
Now that we know how to access ir frames, how do we display them (using cv2, matplotlib or whatever)?
right now i tried:

import pyrealsense2 as rs
import numpy as np
import cv2

points = rs.points()
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.infrared, 1, 1280, 720, rs.format.y8, 30)
profile = pipeline.start(config)

try:
    while True:
        frames = pipeline.wait_for_frames()
        ir1_frame = frames.get_infrared_frame(1) 
        image = np.asanyarray(ir1_frame)
        cv2.namedWindow('IR Example', cv2.WINDOW_AUTOSIZE)
        cv2.imshow('IR Example', image)
        key = cv2.waitKey(1)
        # Press esc or 'q' to close the image window
        if key & 0xFF == ord('q') or key == 27:
            cv2.destroyAllWindows()
            break
finally:
    pipeline.stop()

the problem here is on the "cv2.imshow('IR Example', image)" line, because the dtype of image is an array of pyrealsense2.pyrealsense2.video_frame.

Do you know how to solve this?
Thanks in advance

[Woodstock94]

I just solved my issue by substituting
image = np.asanyarray(ir1_frame)
with
image = np.asanyarray(ir1_frame.get_frame())

[pedrombmachado]

@Woodstock94 you mean:
image = np.asanyarray(ir1_frame.get_data())
because there aren't any get_frame() method. :p

import pyrealsense2 as rs
import numpy as np
import cv2

points = rs.points()
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.infrared, 1, 640, 480, rs.format.y8, 30)
profile = pipeline.start(config)

try:
    while True:
        frames = pipeline.wait_for_frames()
        ir1_frame = frames.get_infrared_frame(1)
        if not ir1_frame:
            continue
        image = np.asanyarray(ir1_frame.get_data())
        cv2.namedWindow('IR Example', cv2.WINDOW_AUTOSIZE)
        cv2.imshow('IR Example', image)
        key = cv2.waitKey(1)
        # Press esc or 'q' to close the image window
        if key & 0xFF == ord('q') or key == 27:
            cv2.destroyAllWindows()
            break
finally:
    pipeline.stop()

[Woodstock94]

yes, sure. I messed it up!^^

[neocubist]

I believe this may be the solution I am looking for. I am a newbie to Python. How do I use the above code to extract the raw images?

[neocubist]

Update: I have installed the necessary packages and managed to get the program running. This was able to stream the IR and also view the dots. However I was unable to capture an image. It would be great if someone could show me how to do that.
Also once the above program begins running, I cannot close the window to exit. Needed to break routine using ^c. Was there a fix for that?

[pedrombmachado]

Hi @neocubist,
replace line cv2.imshow('IR Example', image) by cv2.imwrite( "<filename>", image) for saving the image.
replace while True by while flag and set the flag=False to stop. The answer to your next question "How do I change the flag?" is google it.
I hope it helps.
Bw
Pedro