D435 projector fails to turn on using set_option

[panjekm]

Required Info
Camera Model	D435
Firmware Version	05.08.15.00
Operating System & Version	Ubuntu 16.04
Kernel Version (Linux Only)
Platform	PC
SDK Version	v2.10.1

Issue Description

We are having issues with turning on the D435 projector using the set_option command. This command succeeds but the images have no pattern on them. Doing the same thing works perfectly on D415 device. Below is a code sample of how we start the streams and enable the projector. We request depth resolution of 1280x720 which is max that is supported for depth images and 1920x1080 for rgb.

rs2::config config;
config.enable_device(params.serial_no);
config.enable_stream(RS2_STREAM_INFRARED, 1, params.depth_width,
                       params.depth_height, RS2_FORMAT_Y8, params.depth_fps);
config.enable_stream(RS2_STREAM_INFRARED, 2, params.depth_width,
                       params.depth_height, RS2_FORMAT_Y8, params.depth_fps);
config.enable_stream(RS2_STREAM_DEPTH, -1, params.depth_width,
                       params.depth_height, RS2_FORMAT_ANY, params.depth_fps);
config.enable_stream(RS2_STREAM_COLOR, -1, params.rgb_width,
                       params.rgb_height, RS2_FORMAT_BGR8, params.rgb_fps);

rs2::pipeline pipe;
if (!config.can_resolve(pipe)) {
  LOG(INFO) << "Cant resolve this config.";
}

rs2::pipeline_profile pipe_profile = config.resolve(pipe);
rs2::device device = pipe_profile.get_device();
std::vector<rs2::sensor> sensors = device.query_sensors();

for (rs2::sensor element : sensors) {
  std::string module_name = element.get_info(RS2_CAMERA_INFO_NAME);
  if ("Stereo Module" == module_name) {
    constexpr size_t kEmitterEnabled = 1u;
    element.set_option(rs2_option::RS2_OPTION_EMITTER_ENABLED,
                       kEmitterEnabled);
  }
}
pipe.start(config);

cv::Mat image( params.depth_height,  params.depth_width, CV_16UC1);
// Wait for several frames for stable stream.
for (size_t i = 0u; i < 20u; ++i) {
  pipe->wait_for_frames();
}
rs2::frameset frames = pipe->wait_for_frames();
constexpr int kStreamIndex = 1u;
rs2::frame frame = frames.get_infrared_frame(kStreamIndex);

if (!frame) continue;
for (size_t x = 0u; x < params.depth_width; ++x) {
    for (size_t y = 0u; y < params.depth_height; ++y) {
      image.at<uint16_t>(y, x) =
              ((uint16_t*)frame.get_data())[y * params.depth_width + x];
    }
}
  
pipe.stop();

The strange thing is that if I call the get_option right after turning on the emitter or after the actual frame capture it says it is on while it actually isnt. After this we stop the streams and set them up again, but this time only the IR streams (no depth or color) and at full resolution (1280x800) and in this case we get images that include the projection pattern. I tried adding additional wait times after setting the projector, capturing multiple frames but nothing seems to help. However when using the official ROS driver this works. Is there some other setting I need to set?

Unrelated question: If we want to maximize accuracy and only need depth ranges between 0.2 and 1m, do you have any recommended settings we should use? I read somewhere that you can get close range depth at an expense of resolution, is this correct?

[dorodnic]

Hi @panjekm
What about the realsense-viewer? Are you able to see the effect of Emitter Enabled control in the UI? The reason I'm asking is because the viewer also listens to firmware errors and will show a notification if something is wrong at that level.

[dorodnic]

Also, I think this can help:
D400-Series-Visual-Presets / related-discussion

[panjekm]

I tried it in realsense-viewer. There the emitter seems to work, no errors are thrown. But as I start the realsense-viewer i get several warnings:

 10/04 14:52:26,700 WARNING [140668020262656] (sensor.cpp:313) Unregistered Media formats : [ UYVY ]; Supported: [ ]
 10/04 14:52:32,012 WARNING [140668020262656] (types.cpp:57) set_pu(id=3) failed! Last Error: Input/output error
 10/04 14:52:32,012 WARNING [140668020262656] (sensor.cpp:691) Exception was thrown when inspecting properties of a sensor
 10/04 14:52:37,107 WARNING [140668020262656] (types.cpp:57) set_pu(id=9) failed! Last Error: Input/output error
 10/04 14:52:37,107 WARNING [140668020262656] (sensor.cpp:691) Exception was thrown when inspecting properties of a sensor

I noticed some flickering on the borders of IR image, maybe because of the emitter frequency? Do you know how that frequency is set?

Also, I use the wait_for_frames() function to grab both IR images and sometimes one of them has the IR pattern and the other one doesnt. I wonder how this is possible since this function is supposed to give synced frames?

Thanks!

[dorodnic]

Also, I use the wait_for_frames() function to grab both IR images and sometimes one of them has the IR pattern and the other one doesnt. I wonder how this is possible since this function is supposed to give synced frames?

This can happen if Kernel patches were not successfully applied. Frame number attribute required to synchronise between left and right frames is passed from firmware via UVC metadata. For now, this utility of the UVC1.5 protocol requires custom kernel patch.

So, just to summarise, with D435 you can see the effect of emitter enabled control in the Viewer but not in code, and with D415 both seem to work. I will try to reproduce this using the code you submitted, but this is not something I have previously encountered.

[panjekm]

How can I check if I have the proper kernel patch? Can I somehow update it?

[dorodnic]

In the viewer, you can click on the (i) icon above any of the streams to see the FPS, timestamp, etc...
If the patch is unavailable the timestamp domain will be marked red with relevant error in the tool-tip.
If the patch is working, the timestamp domain should be hardware clock.

[panjekm]

Hm... I see those timestamps on all streams.

[dorodnic]

Hi @panjekm
I had some time to test your code. With my D435 I can clearly see the effect of kEmitterEnabled on the image. My firmware is slightly different (05.09.08.00), but otherwise everything should be equal. Are you sure the defect only reproduces through code and not via the viewer? As it is, this sounds like a hardware issue (perhaps this specific D435 unit overheats and turns off the laser? It can be an unlikely production flaw)
Regarding synchronisation, are you saying you see hardware clock? Looking at the code, I think that what happens is that there are frame-drops on all streams (streaming HD x 3 + Full-HD is putting a lot of pressure on the USB subsystem, if you have somewhat slow PC or it is sufficiently utilised by other processes, the USB driver will cause frame drops). As a result, the pipeline can't find any synchronised IR pairs and falls back on older frames.

[panjekm]

I once observed it also in the viewer but its harder to reproduce. I think it is somehow connected to emitter frequency. As a side test I tried using D435 emitter and seeing it in IR images of D415. Because the devices are not synced you could for some time see no pattern, then you would see it blinking, or full on...

Yes, i see hardware clock. Frames definitely start dropping if I increase frame rates to 25 or 30fps, thats why im keeping it at 6 normally. Yes, it could be that frames are not synchronised because of USB overload, but arent also these older frames that you mention synchronized to emitter frequency so they should contain the pattern?

Thanks for your help!

[dorodnic]

You should certainly see the emitter pattern if the hardware is operating correctly. That said, the camera is allowed to turn-off the emitter in an event of overheating or electrical failure. In the viewer are you enabling all streams at max resolution like in the test App?

My comments on left-right IR synchronization got somewhat off-topic and are not related to the main issue you are reporting. All IR frames on a specific device are synchronized to its emitter frequency.

[dorodnic]

Just to clarify:

Also, I use the wait_for_frames() function to grab both IR images and sometimes one of them has the IR pattern and the other one doesn't

On the D435 in your test App you sometimes get one infrared frame within a frame-set with the pattern and another one without? This is indeed very strange. My comment was that these frames may sometimes not be perfectly synchronized at the software level. But at the hardware level, they should be.

[panjekm]

In the viewer are you enabling all streams at max resolution like in the test App

No, then I get a lot of dropped frames. Missing IR pattern occasionally appears on 1280x720 Y8 images at 6fps.

On the D435 in your test App you sometimes get one infrared frame within a frame-set with the pattern and another one without?

Here it is a bug on my side, was calling wait_for_frames() twice in a row, once for each IR image. Ill try extracting both image from the same frameset. But I would assume the missing pattern is still going to appear but on both images.

[panjekm]

I tested by using IR images from the same frameset and now I can see the pattern in all images. I also tested using D415 and there some images have a weaker pattern (look below). What could be the issue?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@panjekm
Can you try to use realsense viewer and see if the pattern looked different at the same laser power with various application launch test. I didn't see the situation from my side.

[panjekm]

Im not sure I understand what you mean by "various application launch test"?
It doesnt happen all the time. So far Ive seen anything between every 5 and 40 frames.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@panjekmS
Sorry that didn't describe clearly. I mean can you try to close and open the realsense-viewer application multiple times and check if there is weaker IR pattern happens. If you don't change the IR projector power, the IR pattern should be the same no matter how many times you launch the application.

Can you also help to clarify : the sentence "I also tested using D415 and there some images have a weaker pattern", does that mean you run the application multiple times and see the pattern with different power level? or you tested with different D415 device and see the difference?

[panjekm]

Reopening realsense-viewer seems to always give the same image with same strength of the pattern.

So in the code i start the pipe with both IR streams configuration (1280x720, 6fps, Y8), then enable the projector with power set to 360, then call pipe.wait_for_frames(), save the IR images, disable the projector and stop the pipe. Afterwards, the same process (starting the pipe, ...) is repeated. And images from these repetitions sometimes have slightly less strong pattern in them. My guess is that maybe the projector is not perfectly synchronized to image capture frequency and the weaker pattern might be from the period its still turning on.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@panjekmW
Why do you need to start and stop the laser power so frequently? Can you post your code where you looping open and close project and save images?

[panjekm]

Since I am using both D415 and D435 cameras consecutively.

I have fixed the problem now, but I still think there is a bug somewhere in device firmware. Problem occurred when I first started the pipe with 1920x1080 IR stream at 15fps, then stopped the stream, turned off the laser and started up the pipe and turned on the laser again but this time with 1280x720 IR stream at 6fps. The problem with missing pattern was appearing at this second lower resolution stream, but using 15fps for both streams seems to solve it.