Timestamps

[clydemcqueen]

Nice package!

I'm confused about gstreamer timestamps. I ran this test:

gst-launch-1.0 --gst-plugin-path=install/gst_bridge/lib/gst_bridge -v filesrc location=save.mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoconvert ! rosimagesink

... and the resulting ROS timestamps were fairly close to node.now(). From ffprobe I know that save.mp4 has a creation_time field with the original timestamp from several days ago, so I expected the ROS message to have the older timestamps. Is there some setting that I can change? Or I suppose this could be a limitation of qtdemux or some other element in the pipeline. Any pointers would be appreciated.

Thanks in advance,
/Clyde

[BrettRD]

Hi Clyde, thanks!

Gstreamer presents buffers to the sink node with a PTS presentation timestamp, this relates to when the buffer should be rendered on the output devices, and is indexed against the moment the stream starts playback. As far as I'm aware, gst sink elements have no way of knowing any other time.

When the stream goes to playback, the ros sinks capture node.now() as an offset for the PTS values to create valid ros headers.

A lot of my work is with live data feeds, so I hadn't considered the past-timestamps case. Are you using this with rosbag?

I think the best bet would be to add an element property to override the time-offset.
It looks reasonably straightforward in the rosbasesink branch, and should only affect sink->ros_clock_offset in the base class to be available in all sinks.

Cheers,
Brett.

[clydemcqueen]

Hi Brett, thank you for the quick response!

For reference, here is my use case:

I am building an underwater stereo camera using Raspberry Pi Zeros and Pi cameras. In the live case, each Pi uses gstreamer to send h264 data to the primary computer, which runs 2 gstreamer pipelines. The Pi's are not hardware synchronized, so I will use an approximate sync message filter to get reasonable stereo pairs.

But there are lots of variations on this use case, including running without the primary computer while I'm running tests. Decoding the h264 and saving a ROS bag with uncompressed images on the Pi Zero would be difficult. My thought was to simply record the streams in mp4 files on the Pi Zeros, pull the SD cards and then play back the mp4 files on the host computer. This depends on my ability to get the same timestamps on playback so that I can sync the images.

If I understand you correctly, I could use ffprobe or another tool to get the creation_time for each file, then override the time-offset. This would work well. Let me know if you'd like some help!

I'm still curious if there's a way for gstreamer to send time-offset information along a pipeline. This seems like a common use case.

Thanks again!
/Clyde

[BrettRD]

I get the feeling most people using this package are trying to store compressed video for later playback alongside rosbag data.

As far as I know, the buffers passed to sinks only carry timestamps representing when they should be decoded, and when they should be presented (rendered).
I haven't explored the options in "pipeline events" though, there might be some more tools there.

Have a poke around the filesrc and qtdemux to see if they can report any useful timestamps, it would be good to make this a standard feature without additional dependencies.

I'm up to my elbows in the rosbasesink branch with a much-needed refactor, so a pull on this would be painful to manage right now. I'll have a go at adding the clock property alongside the other changes

[clydemcqueen]

I'm up to my elbows in the rosbasesink branch with a much-needed refactor, so a pull on this would be painful to manage right now. I'll have a go at adding the clock property alongside the other change

Ack. I'm happy to help test when it's ready.

I get the feeling most people using this package are trying to store compressed video for later playback alongside rosbag data.

There isn't a well supported solution yet. I took a crack at recording h264 packets as ROS messages (https://github.com/clydemcqueen/h264_image_transport) but a lot depends on how the h264 data is generated. Still, it could be a useful idea.

[BrettRD]

I haven't implemented the clock override, but I've made space for it in the new base classes.
It would be great to have someone give this a brief whirl before I mainline it. #14

Otherwise, I found a bunch of context:
rosbag doesn't yet have a clock-server: ros2/rosbag2#99
rosbag devs are still discussing how that should work: ros2/rosbag2#675
and it's worth thinking about providing or consuming a clock source /clock: http://design.ros2.org/articles/clock_and_time.html#ros-time

[BrettRD]

@clydemcqueen, can I get your opinion on #15?
The nanosecond format is not a very pleasant UI, it's as practical as I could think of.

[clydemcqueen]

#15 LGTM. I will try it next week.

Thanks for the pointers to rosbag2. Interesting discussion. The high frequency /clock workaround is clever. I would love to see time extrapolation in Gazebo / Ignition as well.

It would be nice to have a small ROS message for streaming video and audio so that streams can be bagged with other messages. Do you think GStreamer Data Protocol could be put into a ROS message, and then played back? Something like:

# Generate /image_gdp, which is small enough to be routinely bagged
gst-launch-1.0 -m videotestsrc ! x264enc ! gdppay ! rosgdpsink topic=image_gdp

# Decode on playback, generate /image_raw
gst-launch-1.0 -m rosgdpsrc topic=image_gpd ! gdpdepay ! h264parse ! avdec_h264 ! videoconvert ! rosimagesink topic=image_raw

I'm curious as to your thoughts on this.

[BrettRD]

I'm a little concerned that gdppay hasn't made it out of plugins-bad yet, but I'm definitely on board for a flexible node type.
new issue: #16