-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Frame Queue drops frames when more than 1 stream is enabled. #9022
Comments
Creating a separate frame queue for each sensor seems to work.
It drops a few depth frames right at the start, but then it runs fine. I was able to run this script for an hour with a D455 on a Raspberry Pi 4 with no dropped frames. There really aren't any examples or documentation for creating separate frame queues in python, so this is a bit of a clunky workaround. One would expect that a pipeline with a queue would be able to handle multiple sensor streams without dropping frames, so I am leaving this issue open for now. |
Hi @sam598 Thanks very much for sharing the Python code of your workaround! The Latency vs Performance section of the frame buffering documentation suggests setting the frame queue size to '2' instead of the default of '1' if two streams are being used. Changing the frame queue size to a custom value may though carry the risk of breaking the streams if the wrong value is chosen (i.e a value other than the 1 or 2 suggested in the buffering documentation). As an alternative to changing the frame queue size, dividing streams into separate pipelines may improve performance. In the Python script in the link below with a 2-pipeline setup, the IMU is placed on its own on one pipeline and depth & color on the other pipeline. |
Thanks @MartyG-RealSense ! I think some of my confusion comes from the intended purpose of the This example makes it seem like the purpose of the queue is to create an asynchronous buffer so that frames don't get dropped if some other processing takes longer than usual. The example does explicitly say:
But with multiple streams it doesn't sound like it can be used for that use case at all. Having a buffer of 2 frames doesn't sound any different from waiting for frames from the synchronous pipeline. If the queue can only hold 1 frame per stream, does the queue have any purpose? If not is there a way to warn the the developer or return an error if they try to pass multiple streams into a single queue? If the queue is really only intended for one stream at a time, it would be great to make that explicit in the documentation and examples. This caused a bug that I ended up spending over a week trying to track down.
I did not know if this was possible, so that may be a more elegant solution than my case. |
For the vast majority of RealSense users, changing the frame queue size is a method that they don't need to think about or use even if they have more than one stream active and the default of '1' is sufficient. It is just a feature that is available if the user decides it is useful for their project, as a tool for changing the balance between performance and latency (with a possible risk of dropped frames when weighted towards performance). |
Hmm, the official frame_queue_example.py example I’ve referred to several times in this issue uses a Are you saying that this example will actually cause dropped frames? Something here does not make sense. |
Whilst the advice was that using a custom frame queue size may introduce stream-breaking, it does not state which values do so. Therefore some values may benefit a project whilst other values may harm. There is a risk to using custom values for the frame queue size without documentation about which values are useful or harmful. If you are aware of the possibility though then it may be safe to experiment. Otherwise, someone who does not know of the risk could unknowingly break their program and then spend a long time trying to debug it without being aware of the true cause. of the problem. |
I have to ask and clarify, is there a difference between the
Is my understanding of this correct? |
So to recap.
|
https://dev.intelrealsense.com/docs/frame-management#section-frames-and-threads |
To be absolutely clear; From #5041 (comment) :
|
The
These two things are extremely confusingly named but it seems that they are different. I just want to make sure we are on the same page, because understanding what |
The documentation defines RS2_OPTION_FRAMES_QUEUE_SIZE with this description: "Number of frames the user is allowed to keep per stream. Trying to hold-on to more frames will cause frame-drops". My understanding is that frame queue size is the same thing as frame queue capacity, which can be used to set a maximum latency of a defined number of frames, as described in the SDK example script 'Do processing on a background thread' (see the section of this link headed 'librealsense2'). https://github.com/IntelRealSense/librealsense/wiki/API-How-To#do-processing-on-a-background-thread I do not think that there is much else that I can add about the subject that has not already been covered in this discussion however. |
I've tried my best to explain how I understand @dorodnic's previous advice, but I agree there is only so much the two of us can try and parse that comment without an official statement. |
Hi @sam598 Do you agree that this case can be closed now, please? Thanks very much. |
I need to know there is a way I can use the I appreciate your assistance, but the comments in this issue have thrown that entire use case into doubt. From the documentation and comment it seems clear to me that the If it is as you suggest, that contradicts the example and means You have also expressed that you do not know this 100% for sure, and I do not claim too either. At this point I need assistance from someone who can speak with authority on the matter. Thank you for your understanding. |
I will seek advice from Intel on this subject. Thanks very much for your patience. |
@sam598 , in this case the frame queue is only symptomatic to the actual difference - So I believe now the real question is to establish how many frame drops are registered to ensure that the syncer throughput is within the spec. |
Thank you @ev-mp. That information clears up a lot and is extremely helpful. (1) To answer your question first:
When using the Even when not artificially stalling the thread when using the (2) As for preventing dropped frames: The most critical thing for my use case is that I capture every frame from both streams without dropping any. Based on what you have said it looks like I have no need for the Should I be able to prevent dropped frames by providing a large Finally, is providing a new |
@sam598 hello, thank you for the inputs as it also clarifies both the application and the constrains.
I'll clarify the on the 'Pipeline' internal mode - the As for preventing frame drops - imo the As for the last question - the |
Hi @sam598 Do you require further assistance with this case, please? Thanks! |
@MartyG-RealSense I think that clears everything up. So to recap:
I would propose that for this issue to be properly closed the frame_queue_example.py script should caveat that the given Pipeline example only works for a single stream. I would hate for someone to spend days on the same issue, and lengthy GitHub issues like this one are not a substitute for proper documentation. |
@sam598 I have added a Documentation tag to this case to highlight your documentation request described in the comment above. This case should be kept open whilst that documentation request is active, but there is nothing else that you need to do. Thanks very much! |
I wanted to comment on this after doing a bunch of research into the code. I'm not sure everything I think I know is right, but this is what I understand so far. Take everything written here with a grain of salt. First, on the issue of RS2_OPTION_FRAMES_QUEUE_SIZE. This is option is, in my humble opinion, poorly named. Inside the library, a pool of empty frames is maintained. From this pool, unused frames are claimed and filled with color, infrared, and depth information as it arrives from the physical sensor. (I don't think framesets themselves draw from this pool, only the frames inside the frameset, but I'm not sure about that.) No matter who has the frame --- library, queue, or you --- or what it is being used for, so long as the frame is being used by somebody and must not be erased, it has been "published." When the frame is .released() (c++) or .closed() (java), the frame is returned to this pool. If the pool ever becomes empty, libRealSense will start dropping frames --- not because your queue is full, but because it has effectively "run out of memory" to allocate new frames. The difference between this situation and a full queue is that when a queue becomes full the queue will drop the oldest frame and hold onto the newest frames; but, when the frame pool is exhausted, the library cannot acquire a new frame and thus it must drop the incoming sensor image data. If you fail to unpublish your frames for any reason, the library will stop storing new frames as the arrive from the sensor. This is the, "I got 15 frames and then didn't get any more," complaint. When the library is closed, a message will be (sometimes) be printed telling you how many frames you were "holding" when the library was closed. This is really how many frames were taken out of the frame pool, but not returned to the frame pool, for each stream. This is usually 1 for each stream (if 0 nothing prints), because the library itself may have been holding frames for you internally. But if the number is high, like 12, or 14, then you have failed to release your frames and framesets when you were done with them (and probably you had a frozen stream.) (Note: This paragraph applies to Java and may or not apply to other languages.) One important key thing to note is that only the Frame object owns the underlying frame-pool frame. If you cast the Frame using .as(...) to a DepthFrame or a VideoFrame, this "recasted" frame does NOT own the underlying frame-pool object and "releasing"/"closing" it does nothing. You MUST close/release the Frame from which the VideoFrame/DepthFrame/Whatever was created, or the frame will not be returned to the pool, but you must NOT close/release it until you are done using any casted DepthFrames/VideoFrames/etc. (Example: if you cache a VideoFrame, you must also cache the underlying Frame so it can be closed later.) Closing a FrameSet closes all UNREMOVED frames in that FrameSet. If you get a new FrameSet after processing a FrameSet (with, say, align), I BELIEVE you must close them both and can close the unprocessed set before you use the frames in the processed set, but I'm NOT SURE ABOUT THAT. I have, so far, been unable to get align to work properly in my own code. In order to not drop frames, the size of the pool must be sufficient for all users of the pool -- the library for incoming image data, the syncher, the queue, frames created by filters, and you -- to hold on to whatever frames they require. Setting the pool to 1 will certainly cause problems! Setting it too large could cause you to run out of memory. What this setting does NOT do is control how many frames are queued, or in what way they are queued, or how many streams you can use. RS2_OPTION_FRAMES_QUEUE_SIZE sets the size of this pool, in frames, and because it represents nothing more than a frame-based memory pool, this is why you are told not to change the value. By default, it starts at 16 and will grow as high as 32, and that should be enough for most programs. However, there are two situations where you might want to alter this number. One is if you wish to grab frames and hold onto them for a while because you are processing multiple frames at a time, because you need a big queue, or because you are holding multiple frames generated by multiple filters. (Remember, even if the Queue has your frames, they're still published.) In this case, you might need to raise the number. The other is if you are on a low memory platform, you may wish to decrease this number so that if your software hits a brief pause the library does not drain all system memory resources trying to buffer past frames excessively. My current working theory is the minimum value for the frame pool setting is 4 times the number of streams you wish to use minus one. You need 1 frame per stream for what you are currently processing, 1 frame per stream minus one for the syncher to hold while it waits for a coherent set of frames, 1 frame per stream sitting in the queue as a FrameSet waiting for you to remove, and 1 more frame per stream so that arriving data from the camera can be stored by the library. Thus, for 2 streams, a minimum pool size of 7 is required. This assumes you are not using any filters, as filters generate additional frames. Of course, the default is 16 with a maximum of 32, and if you wish to have a queue of size greater than 1 or make use of the available filters (such as align), you must increase the size of the frame pool above 7, and potentially above 32. As for FrameQueue, it appears to be a simple FIFO "frame" queue. If I understand the code correctly, the internal queue has the ability optionally to block on enqueue until room is available, but as far as I know this option is always turned off in the library and the queue instead always responds to overcapacity situations by dropping the oldest unclaimed data and accepting new data. FrameQueue, despite its name, can hold either Frames or FrameSets, and if you are supplying it with FrameSets (from, say, pipline or syncer) then a queue depth of 1 will hold 1 FrameSet containing both streams. If, however, you are suppling it with frames from sensors directly (not as framesets), you can run into trouble: you need a minimum depth of 2 (for 2 streams) so that the two streams do not overwrite each other's frames --- but, if one sensor for any reason gives two frames before the second sensor gives one frame (say, because they are at different FPS or are physically different sensors with slight timing variations), then the queue will sometimes contain two frames from one sensor and none from the other. Having separate queues for each sensor eliminates this potential problem. Pipeline uses a queue internally if you are not using callbacks, but queue is set to a size of 1. That is, it always drops any unclaimed old frameset and replaces it with the new incoming frameset. If you want a queue that is larger than 1, you would pass in the FrameQueue::enqueue method to the pipeline (or modify the source code, but that seems unncessary). I don't believe there is any difference at all between using the pipeline directly and redirecting the output to a FrameQueue with a length 1: If you want to poll the pipeline, just do a Pipeline.waitForFrames(0). If on the other hand, you want to queue up frames and not drop them, you must create a FrameQueue with a size larger than 1. Obviously, you must process the frames before the queue fills up, or you will start dropping the oldest frames --- or the newest if RS2_OPTION_FRAMES_QUEUE_SIZE is set improperly. If you are getting the same Frame multiple times from one or more streams in sequential FrameSets or dropping every-other-frame, this is likely because the syncer stopped getting frames from one or more of your streams. The most likely cause of this is that the library could not obtain a frame from the frame pool, and as a result the syncher reused an old frame that it had stored previously. This can happen when failed to release one stream's frames but are correctly releasing other stream's frames. If you get intermittent drops but are certain that you are closing your frames properly, you may have a real out-of-memory condition where frames from the frame pool are getting memory-swapped to disk causing the library thread to hang and lose data: try reducing the value of RS2_OPTION_FRAMES_QUEUE_SIZE (and the Queue, if appropriate) to reduce your memory footprint. If you get "couldn't allocate a composite frame", you are not freeing your Frames and the pool is completely exhausted. The bottom line is, if you want to not drop any frames, do these four things:
|
Thanks so much @cwm9cwm9 for sharing your experience in such detail! |
Tested with this modified version of
frame_queue_example.py
This script will drop frames whenever
time.sleep()
is called. It does not drop frames when only 1 stream is enabled.The amount of dropped frames varies depending on camera and platform. But I have seen this behavior on every combination of D435, D455, Raspberry Pi OS, and Windows 10.
SDK 2.45.0 and Firmware 5.12.13.50
The text was updated successfully, but these errors were encountered: