How to reduce frame drops while recording to a bag file?

[mahir1010]

We have a recording setup where we record multiple hardware-synced D435s to files without processing framesets. Latency is not an issue; however, frame drop is. Our goal is to achieve stable and constant framerates. Following is a snippet of our code, and for each camera, we fork a separate process.

    try:
        slave_pipeline=rs.pipeline()
        slave_cfg=rs.config()
        slave_cfg.enable_device(slave)
        slave_cfg.enable_stream(rs.stream.depth, *(resolution), rs.format.z16, NATIVE_FRAME_RATE)
        slave_cfg.enable_stream(rs.stream.color, *(resolution), rs.format.bgr8, 60) # Does format have any impact on frame drops?
        slave_cfg.enable_record_to_file(f"./{time_stamp}/{slave}.bag") # Does frame_queue impact recording to file?
        slave_pipeline.start(slave_cfg)# Would adding frame_queue as a parameter here work?
        while True:
            pass
    except:
        print(slave,"Stopped") 
    finally:
        slave_pipeline.stop()

Not shown here, but we also disable auto-exposure and set it to 15.6 ms to get 60Hz, however, our effective framerate is always ~59.3Hz on the RGB sensor and ~59.8Hz(acceptable) on the depth.

In one of the discussions, I read that increasing frame queue size will increase latency but reduce frame drops. The performance is even better when setting the frame queue size separately for both streams. What would be the best way of doing it?

EDIT: Added Camera configuration

        if cfg["master"]==system.devices[i].get_info(rs.camera_info.serial_number):
            print("Master Camera Found:",rs.camera_info.serial_number)
            master=system.devices[i]
            sensor=master.first_depth_sensor()
            c_sensor=master.first_color_sensor()
            c_sensor.set_option(rs.option.enable_auto_exposure, 0)
            sensor.set_option(rs.option.enable_auto_exposure, 0)
            sensor.set_option(rs.option.exposure, 1000) # Ignore these values, only used as a signal generator.
            sensor.set_option(rs.option.inter_cam_sync_mode,1)
        else:
            print("Slave Camera Found:",rs.camera_info.serial_number)
            slaves.append(system.devices[i])
            sensor=slaves[-1].first_depth_sensor()
            c_sensor=slaves[-1].first_color_sensor()
            c_sensor.set_option(rs.option.exposure, 156)
            c_sensor.set_option(rs.option.gain,128)
            c_sensor.set_option(rs.option.gamma,380)
            c_sensor.set_option(rs.option.sharpness,70)
            sensor.set_option(rs.option.exposure, 15000)
            sensor.set_option(rs.option.inter_cam_sync_mode,2)

[MartyG-RealSense]

Hi @mahir1010 In a discussion at #1957 (comment) about the maths of forcing a constant FPS when using manual exposure, a RealSense team member suggests increasing the gain value to improve depth performance.

In regard to applying settings for individual streams, you could set up a separate pipeline for each stream. There is a Python example at #5628 (comment) that creates two pipelines and places depth + color on one pipeline and IMU on the other. So it may be possible to adapt the script to have depth on one pipeline and color on the other.

[mahir1010]

Hi Marty, thank you for the information. I have updated the post with the camera configuration we have been using for reference.

I had some questions written as comments in the code snippet so you might have missed them.

1. Does setting frame_queue affect the code that handles recording to file? In that, do they share the pipeline configuration?
2. Do stream formats have any impact on frame drops? (RGB,BGR,RAW....)
3. Does having separate pipeline/config for both streams affect writing data to the bag file and HW synchronization?

[MartyG-RealSense]

1. Setting a larger frame queue size than the default of '1' should affect the recording positively in terms of a reduction in the number of frames that are dropped during the recording. '2' would be a safe queue size to use for two streams (depth and color). Other RealSense users have had success with '50'.

The larger the frame queue size, the more of the computer's available memory capacity that will be consumed as frames will not be being released from the pipeline as often.

2. There can be FPS lag when both depth and color are enabled, though your manual exposure settings should be helping to ensure a constant frame rate and so that does not seem to be a concern in this case.

3. My recollection is that bag recording or hardware sync has not been previously attempted with a two-pipeline setup, so I do not have information about how multiple pipelines would affect those functions.

[mahir1010]

I tried several different configuration (even with absurdly high buffer sizes) for 30 second clips however it does not seem to affect the drop rate. I even tried increasing frame_queue_size value to 30 (default max) on each sensors.
I reduced exposure to 7.8 ms on the color sensor and it still drops a lot of frames.

[MartyG-RealSense]

I read this issue again from the beginning. You mention that you are setting 60 FPS for depth and color but actually getting ~59.3Hz on the RGB sensor and ~59.8Hz. That is actually an excellent performance, as it is rare for the actual FPS to be exactly the value that was set. For example, if 30 FPS is set in the RealSense Viewer tool and the real-time information overlay is activated, the displayed actual FPS during streaming will typically be around 29.8.

When talking about 'frame drops', this is typically meant as image frames that have been lost, and are therefore missing from a bag file recording. Is that what is happening to you, or was your question about dropped frames related to the lower than expected FPS speed, please?

[mahir1010]

Yes, I understand that we cannot get constant 60Hz on these cameras but there are some sessions where we get ~59.7 Hz on RGB which is still manageable for synchronization, however, 59.3Hz is not . The videos drift apart after a while. I just want to know what can be done to ensure that FPS stay as close as possible to 60Hz on the RGB sensor.

Yes, by frame drops I mean actual missing data from the bag files. The FPS I was talking about was average frame rate of the whole bag file. $\frac{total frames}{expected frames}$. Hope this clears up what I am asking.

[MartyG-RealSense]

Instead of enforcing a constant FPS using a manual exposure value, the alternative method of FPS enforcement to have auto-exposure enabled and an RGB option called auto-exposure priority disabled.

The hardware specification of the computer can have a bearing on dropped frames. For example, if the storage drive that the data is being written to has a slow access speed then it can create a 'bottleneck' delay in writing frames. Having recording compression enabled may also cause dropped frames on some computers. A RealSense team member provides advice about this at #2102 (comment)

At #9022 (comment) a RealSense user provides a very detailed analysis of frame queue size based on their research.

[mahir1010]

The hardware spec should be more than enough. We use i-7 10850H, 128GB 2900MHz Ram, 4TB NVME SSD . Also, as I said, this happens randomly, which is even worse.

As for RS2_OPTION_FRAMES_QUEUE_SIZE, I tried the rs.option.frames_queue_size to change the default queue size (16) to 30 for both RGB and Depth sensors. There seems to be some kind of bug. The metadata from the bag file shows that the queue size does not change from 16 for both sensors whether we set it to 30 or 10. I believe fixing this should improve the results. FYI we are using 5.14.0 firmware with 2.53.1.4623 librealsense.

[MartyG-RealSense]

Is the queue size set before recording begins? If changes to settings are made after recording starts then those changes are not applied to the recording and it only uses the settings that were configured before recording began.

[mahir1010]

Yes, even before enabling the stream if that matters.

sensor=slaves[-1].first_depth_sensor()
c_sensor=slaves[-1].first_color_sensor()
c_sensor.set_option(rs.option.exposure, 78)
c_sensor.set_option(rs.option.gain,128)
c_sensor.set_option(rs.option.gamma,427)
c_sensor.set_option(rs.option.sharpness,70)
c_sensor.set_option(rs.option.frames_queue_size,30)
sensor.set_option(rs.option.frames_queue_size,30)
sensor.set_option(rs.option.exposure, 12000)
sensor.set_option(rs.option.inter_cam_sync_mode,2)

then

slave_pipeline=rs.pipeline()
slave_cfg=rs.config()
slave_cfg.enable_device(slave)
slave_cfg.enable_stream(rs.stream.depth, *(resolution), rs.format.z16, NATIVE_FRAME_RATE)
slave_cfg.enable_stream(rs.stream.color, *(resolution), rs.format.yuyv, TARGET_FRAME_RATE)
queue = rs.frame_queue(600)
slave_cfg.enable_record_to_file(f"./{out_folder}/{time_stamp}_{slave}.bag")
while not flag.value:
    pass
print(slave,"Started",time.time())
slave_pipeline.start(slave_cfg,queue)

[MartyG-RealSense]

As you are using both depth and color, I would not rule out the possibility of the color stream being a factor in your dropped frames. At #2637 (comment) a RealSense team member lists the reasons why using color in a multiple-camera hardware sync setup where depth is synced and color is unsynced can cause problems.

[mahir1010]

This still does not answer my questions. Yes, in all of my replies I have mentioned that we are using both streams. Is there a way to fix the frame_queue_size issue? If not, respectfully, please close the discussion from your end as this is not going anywhere. These "quirks" in the cameras add up, making them very unreliable.

[MartyG-RealSense]

The intention of my suggestion was to explore reasons why frame drops might be occurring in a multi-camera setup.

As you are using manual exposure, setting RGB to 6 fps instead of 60 and setting RGB exposure to 78 instead of 156 may result in less lag on the RGB stream. 60 FPS RGB will work best when auto-exposure is enabled. These characterstics are due to the 'rolling' shutter on the D435's RGB sensor being slower than the fast 'global' shutter on its depth sensor.

I do not have any further advice to offer about the frame queue size unfortunately, so if changing the RGB settings is not possible for your multicam system or the changes do not make a difference to frame drop then regretfully, closing the issue may be the best course of action.