Playback from bag files - direct access to frames by index

[andreabnd]

Required Info
Camera Model	D400
Firmware Version	05.08.15.00
Operating System & Version	Win 10
Platform	PC
SDK Version	2.10.2

Hello,
I am recording the color and depth streams for offline processing. To reproduce the recorded sequence, I am following the rs-record-playback sample. https://github.com/IntelRealSense/librealsense/tree/master/examples/record-playback
However, my objective is to process every frame of the sequence, and using the below code, I am experiencing a drop of frames (especially if the processing takes a while).

rs2::frameset frames;
rs2::config cfg;
auto pipe = std::make_shared<rs2::pipeline>();

cfg.enable_device_from_file("Test.bag");
pipe->start(cfg);

rs2::device device = pipe->get_active_profile().get_device();
rs2::playback playback = device.as<rs2::playback>();

if (pipe->poll_for_frames(&frames))
{
       rs2::frame depth = frames.get_depth_frame();
       rs2::frame color = frames.get_color_frame();
       // Do processing on color and depth frames
       // ...
}

Thus, I was wondering if there is a way to access the frames by index, knowing in advance the number of frames recorded in the sequence.
I have tried to fix the drop of frames by pausing the playback ( playback.pause() ) before to get the frames and doing the processing, and then resuming it ( playback.resume() ) after the processing. However, I would definitely prefer to have direct access to the frames by index.
I know that using the playback::seek method I can set the playback to a specified timepoint. So, in case I cannot access the frames by index, is there another way to know the timestamp of every frame recorded in the sequence, to get access through the seek function?

Thanks in advance!

[dorodnic]

Hi @andreabnd
Thank you for your feedback. We will integrate it into our backlog, either in form of seek overload, or some other API.
At the moment, pausing the playback is the best solution we offer. Alternatively, you can look into the underlying rosbag API. We have a sample doing something similar under tools/rosbag_inspector.

[andreabnd]

Hi @dorodnic
Thanks for your reply! It would be really helpful to have that possibility.

Thanks

[blackccpie]

Hi @dorodnic & @andreabnd,

I was looking for the same feature in the API, in order to have frame-by-frame replay without drop, but I first thought this could be handled using the set_real_time method of the rs2::playback device interface. I made some quick tests with it but I saw no real difference in the replay behavior, did I miss something about this feature?

Thx for your help

[lramati]

The set_real_time function is currently broken, but I am working on a potential fix.
As @dorodnic said above, for now using rs2::playback::pause and rs2::playback::resume in your callback is the best we can offer

[blackccpie]

@lramati : thx for your answer, I'll stick with pause/resume for now, and wait for the set_real_time fix or the index based seek.

[visez]

it seems like repeated calls to rs2::playback::resume and rs2::playback::pause crash the program. Here is the minimal code to replicate the issue. My bag file contains 91 frames and yet the code crashes after a variable number of frames < 91. (xcode on macos, librealsense 2.2). Are there any easy ways to allow non-realtime frame-by-frame processing from bag files?

    rs2::frameset frames;    
    auto pipe = std::make_shared<rs2::pipeline>();
    rs2::config cfg;
    cfg.enable_device_from_file("test.bag");
    pipe->start(cfg); //File will be opened in read mode at this point
    rs2::device device = pipe->get_active_profile().get_device();
    rs2::playback playback = device.as<rs2::playback>();
    
    int count = 0;
    while(true) {
        if (pipe->poll_for_frames(&frames))
        {            
            rs2::frame depth = frames.get_depth_frame();
            rs2::frame color = frames.get_color_frame();
            playback.pause();
            std::cout << count << std::endl;
            waitKey(500);
            playback.resume();
            count++;
        }
    }

[andreabnd]

Hi there,

I am following up on the previous messages. Did you guys implement any solution to access the frames of a recording sequence by index. Anything helpful in the latest release of the SDK?

Thanks in advance for your help!

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@andreabnd The team is looking into the issue. Will update when get resolved.

[valkjsaaa]

+1 I'm also looking for indexed access, or fixed set_real_time function.

[apoorva-sriv]

See https://communities.intel.com/message/553535#553535.

[valkjsaaa]

Thanks for the information! However, I just really want the set_real_time property to work, so that I can keep the same processing pipeline for online usage and offline debug.

[dorodnic]

@valkjsaaa @visez @andreabnd - set_real_time should now work properly, and you should be able to fetch frames one frameset at a time using 2.16.1

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@andreabnd Did you get chance to try v2.16.1? Please let us know if it works for you. Thanks!

[visez]

sorry unfortunately I have been assigned to a new project and I do not have access to the realsense anymore. Thanks for all the support

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@andreabnd Thanks for your update! We will close this ticket accordingly.

[andreabnd]

Hi there, thanks for following up! Actually I still have to try, and I'll let you know as soon as possible

Thanks

[blackccpie]

Hi,

Following my previous question about pipeline::poll_for_frames behavior in non-realtime mode, in the #2472 issue report, I think it would be great to have a code sample in the sdk showing a simple example
of frame-by-frame bag file replay, with stop condition management. For now I've come up with this sample code:

int main( int argc, char** argv ) try
{
	rs2::config config;
	rs2::device device;
	rs2::pipeline pipe;
	rs2::pipeline_profile pipeline_profile;

	// enable file playback with playback repeat disabled
	config.enable_device_from_file( "test.bag", false );

	// start pipeline and get device
	pipeline_profile = pipe.start( config );
	device = pipeline_profile.get_device();
	
	// get playback device and disable realtime mode
	auto playback = device.as<rs2::playback>();
	playback.set_real_time( false );

	rs2::frameset frames;

	size_t frame_index = 0;

	bool stop = false;
	bool playing = false;

	while( !stop )
	{
		if( pipe.poll_for_frames( &frames ) )
		{
			playing = true;

			// Add some frames processing here...

			std::cout << "\rsuccessfully retrieved frame #" << ++frame_index << " (" << playback.current_status() << ")";
		}
		else
		{
			using namespace std::chrono_literals;
			std::this_thread::sleep_for( 10ms );
		}

		stop = playing && ( playback.current_status() == RS2_PLAYBACK_STATUS_STOPPED );
	}

	std::cout << std::endl << "successfully ended file playback" << std::endl;

	pipe.stop();

	return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}
catch (const std::exception& e)
{
    std::cerr << e.what() << std::endl;
    return EXIT_FAILURE;
}

First thing is I'm not really happy with the stop condition, which I find a bit awkward because it seems like the status is set to RS2_PLAYBACK_STATUS_PLAYING only after the first successful frameset retrieval. Do you have a better recommendation on that?

Second thing is poll_for_frames happens to periodically fail and return false even if the stream has not ended, and it's a strange behavior for a query mode not supposed to depend on frame readiness, don't you think?

Regards

[blackccpie]

Hi @MartyG-RealSense, thanks for referencing my script. The thing is I'm not really satisfied with its current functioning, as the two questions from my last comment remain unanswered. I know you might receive a lot of user support requests, but as theses questions are just questions regarding how the API is supposed to work, I'm pretty sure they have simple answers that could help me improve my code.

Thanks for your help

[MartyG-RealSense]

I will recap your questions below to make them easy to find:

---

First thing is I'm not really happy with the stop condition, which I find a bit awkward because it seems like the status is set to RS2_PLAYBACK_STATUS_PLAYING only after the first successful frameset retrieval. Do you have a better recommendation on that?

Second thing is poll_for_frames happens to periodically fail and return false even if the stream has not ended, and it's a strange behavior for a query mode not supposed to depend on frame readiness, don't you think?

---

[blackccpie]

@dorodnic : I know my questions are a bit lost in this already closed issue, so would you like me to create a new issue on this topic? thx

[msacco22]

Hi, are there any examples of how to use the set_real_time and how to then fetch frames one frameset at a time? I'd like to then align the depth and color data from the frameset.

[MartyG-RealSense]

@msacco22 Searching through frames is discussed in the link below:

#3121

[blackccpie]

@msacco22 : you can also refer to my sample code here, where you have access to frames index