D435 frame rate changes with respect to ambient lighting

[anug7]

---

Required Info
Camera Model	D435
Firmware Version	05.11.15.00
Operating System & Version	Ubuntu 16.04
Kernel Version (Linux Only)	4.15
Platform	Lattepanda Alpha
SDK Version	2.22.0
Language	C++
Segment	Robot

Issue Description

The frame rate of D435 stream (both color & depth) are not constant when the external lighting varies. We have a external strobe LED to illuminate the scene and when the intensity is high, we see that frame rate goes high (6 fps) compared to 3.5 fps when illumination is very low. See the attachment for more details. For our requirement, we need constant fps on both the streams. Any pointers related to this issue would be really helpful.

Thank you in advance!

D435 settings:

1. Color: 1920x1080 @ 6 fps
2. Depth: 1280x720 @ 6 fps
3. Using recorder to record the stream in bag file.
4. I have tried disabling both auto exposure and auto white balancing. But no improvements.
5. We record this streams when the system is under full load. e.g 400% cpu consumption. The recorded bag file also has this fps issue.

Code:

1. Delay is calculating in the code by following piece of code, prev_time is 0.167 when the lighting intensity is high and it does to 0.367 - 0.500 when the intensity is low

if (boost::iequals(name, "Stereo Module")){
         sen.open(depth_stream);
         sen.start([&](rs2::frame f){ 
              printf("Time: %f\n", f.get_timestamp() /1000 - prev_time);
              prev_time = f.get_timestamp() / 1000;
          });
 }

2. The auto exposure and white balancing are disabled by using the following code,

void set_controls(rs2::device dev, ros::NodeHandle n){
        int expo = 400, gain = 128, wb = 3700;
        bool  auto_expo = false, auto_wb = false;
        for (auto sen: dev.query_sensors()){
                auto name = sen.get_info(RS2_CAMERA_INFO_NAME);
                if(boost::iequals(name, "RGB Camera")){
                        sen.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, auto_expo);
                        sen.set_option(RS2_OPTION_EXPOSURE, expo);
                        sen.set_option(RS2_OPTION_GAIN, gain);
                        sen.set_option(RS2_OPTION_ENABLE_AUTO_WHITE_BALANCE, auto_wb);
                        sen.set_option(RS2_OPTION_WHITE_BALANCE, wb);
                }else if (boost::iequals(name, "Stereo Module")){
                        sen.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, false);
                        sen.set_option(RS2_OPTION_EXPOSURE, 5000);
                }
        }
}

Attachment details:

1. First two images corresponds to high intensity (as per setup) scenario of color & depth images.
2. Second two images corresponds to low intensity.

[MartyG-RealSense]

You can enforce a constant FPS speed by leaving auto-exposure enabled but disabling the 'Auto-Exposure Priority' option.

#5290

[anug7]

@MartyG-RealSense ,
Thank you for the suggestion.

Will try the option and let you know. Just curious, Wont disabling the auto-exposure give constant fps?

Also for our requirement, we need the exposure to be manual.

[MartyG-RealSense]

You could disable auto-exposure and set your own values manually, though it may affect depth performance for the reasons discussed in the link below,

#1957 (comment)

[anug7]

I tried setting it for depth stream. But it fails.

try{
    sen.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, false);
}
catch(...){
    std::cout << "Setting failed"<< "\n";
}

And with manual exposure, the problem still persists.

[anug7]

@MartyG-RealSense

Also I tried for color stream for which I was able to set this value using the same code. But the problem still persists. I used auto exposure and disabled RS2_OPTION_AUTO_EXPOSURE_PRIORITY.

[MartyG-RealSense]

Generally, RS2_OPTION settings are set to false or true with a 0 or 1, where 0 = false and 1 = true. For example, to disable:

sen.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0.f);

[anug7]

@MartyG-RealSense , I have tried your suggestion. Still the problem persists.
And also tried updating librealsense to 2.31.0 version.

[MartyG-RealSense]

If you have turned off auto-exposure, did you set a manual exposure value that is appropriate for the FPS speed that you are using, as described in the advice linked to earlier please?

#1957 (comment)

[anug7]

Yes. I set the value of exposure to 400 (~25fps).

[MartyG-RealSense]

I wonder if the sudden changes in light conditions caused by the LED might be saturating the IR imager, in the same way that a stream may degrade or freeze if it captures a burst of sunlight (by looking up at the sky or experiencing a bright ray reflected off a surface such as a glass door).

[anug7]

Is there a way we could handle this? such that we can get consistent fps.

[MartyG-RealSense]

You could apply a physical IR filter to the camera. This could take the form of a bulky external filter or a micro-thin filter film placed over the IR lens on the front of the camera.

Another approach may be to program the camera to reset automatically if an event that degrades performance occurs. Assuming that the LED will be strobing between low and high constantly though, prevention (filtering) is probably better in this particular case, otherwise you might have to reset the camera repeatedly.

The subject of physical filtering with cover materials is discussed in detail in the link below.

https://forums.intel.com/s/question/0D50P00004O1gOtSAJ/where-can-we-obtain-cover-material-and-gaskets-for-custom-mounting-of-the-d415-stereo-depth-module

[anug7]

@MartyG-RealSense , Thank you for your suggestion. Will try!