How can I get a shot at a specific time with video recording active?

[aimaicai]

Question

Hi,
I'm using the amazing mediamtx (v. 1.5.1) with a couple of net cameras. I'm recording video, but I would like to get, when I need it, an image at a specific time (or the nearest/next frame). Do someone know if it is possible?
Thanks

[asturl]

I'm not aware of any built in way to do this in MediaMTX, but you can almost certainly use the recorded files with the playback server functionality to get a video clip at a specific time of day and then use that to extract and provide a single frame image to a client.

The questions that really affect potential solutions are:

• Do you need to do this for one request at a time at relatively low frequency, or possibly many different ones in short intervals?
• How long of a wait time for the request is acceptable?
• Does your scenario allow running other webservices (of whatever language/framework you want) that can provide endpoints and handle requests for this desired functionality? Can you call the MediaMTX API?

[aimaicai]

Hi, thanks for the reply. The scenario is my home video storage and playback system: for the videos I rely on the built-in recording functionality. But I would like to get images too, just to have the easy option to zoom in, or to draw a rectangle to highlight an object in the image coming from an event (fot this I must be precise with timings).
Answers:

• very low frequency: usually it is a request from me, or it is request from an object detection event;
• < 1s;
• the little project is in java, I can call mediaMTX api and call any framework.

I was thinking as you suggest me to request a the shortest video to mediamtx and then decode the first frame, but I would like to know if there was a more efficient way.
Do you know if the first frame in the video is the nearest frame to the time I request?
Thank you

[aler9]

Hello, you can do this by enabling the playback server:

playback: yes

The playback server allows you to get a frame corresponding to a specific time. You also want to convert this frame into JPEG. To perform this operation, use FFmpeg to download the frame from the playback server and convert it:

ffmpeg -i 'http://localhost:9996/get?path=my_camera&start=2024-02-21T15%3A11%3A15%2B00%3A00&duration=10' -qscale:v 4 -frames:v 1 my_shot.jpg

Playback server is documented in the README.

[aimaicai]

I apologize, perhaps I didn't express myself well. I meant to ask what the timing of videos in playback is based on. I need it to understand which timestamp I need to assign to my events so that if I assign the T timestamp to my event, when I request mediaMtx to have playback with start=T, the first frame corresponds to the same frame that generated the event . I was thinking of something like pkt_dts_time.
Do you think this is a possible thing to do or could there be time lags?
Thank you very much for your replies.

[aler9]

The timestamp of recordings is the absolute timestamp of incoming frames. Currently, there are only two protocols that can route absolute timestamps, and that are RTSP and HLS. Therefore, in case of RTSP and HLS, timestamp of recordings is the original absolute timestamp, while in case of all other protocols, timestamp of recordings is the one of the server, corresponding to the instant in which frames have been received.

If you want to synchronize events and recordings, my advice is to setup your architecture in this way:

incoming streams --> mediamtx --> recordings
                            |--> RTSP or HLS --> video processor / event generator

Make your video processor read streams from MediaMTX, and activate recordings on that same MediaMTX instance. The video processor can then read the absolute timestamp of each frame.

In order to read the absolute timestamp of a frame with FFmpeg and RTSP, instructions are here:
https://superuser.com/questions/1470807/can-ffmpeg-obtain-the-absolute-timestamps-of-frames-captured-in-a-video-stream

[aimaicai]

Thanks for all your suggestions. I tried from ffmpeg cli and I can get the results. Now I proceed to bring everything into my program, reading the frames from standard output and parserizing the showinfo output to obtain the timestamp.
Congratulations for the excellent work on mediaMtx.

[aimaicai]

I'm almost there: I have a problem in the phase of recovering the frame from the "historical" video, in the sense that before arriving at the correct frame, I find many others (I imagine that there are all those starting from the beginning of the fragment), I wanted I'm wondering if you know a way to quickly locate the correct frame with ffmpeg, or if you can please give me some guidance on how to find it another way.
Thank you

[aler9]

I'm almost there: I have a problem in the phase of recovering the frame from the "historical" video, in the sense that before arriving at the correct frame, I find many other

You can use the new integrated playback server and the new "standard MP4" output format available since v1.7.0 to find the exact frame that corresponds to a given instant, without worrying about previous frames (that are usually needed since most frames can be decoded only after previous ones have been decoded). Enable the playback server and request a video that corresponds to the desired instant:

ffprobe 'http://localhost:9996/get?path=stream&start=2024-04-20T23%3A46%3A25.114077%2B02%3A00&duration=0.1&format=mp4' -show_packets -show_entries packet=dts

Thanks to the "standard MP4" format, FFmpeg is able to obtain the frame that corresponds to the desired instant, and that is the first one with a positive DTS:

[PACKET]
dts=-29992
[/PACKET]
[PACKET]
dts=-26994
[/PACKET]
[PACKET]
dts=-23995
[/PACKET]
[PACKET]
dts=-20994
[/PACKET]
[PACKET]
dts=-17996
[/PACKET]
[PACKET]
dts=-14997
[/PACKET]
[PACKET]
dts=-11996
[/PACKET]
[PACKET]
dts=-8998
[/PACKET]
[PACKET]
dts=-5999
[/PACKET]
[PACKET]
dts=-2998
[/PACKET]
[PACKET]
dts=0
[/PACKET]