Configurable Kill Timeout and Possible Preferred Timeout?

[jlpoolen]

I created a script which launches multiple test rtsp clients against an array of Reolink cameras. I noticed the many of the instances would end, some with the error message that the default time of 10 seconds had been exceeded. So I modified mp4.rs changing 10 to 30 at line 568.

I've noticed a substantial improvement -- whereas some connections would die with seconds, now all four remain connected after several minutes. (Going on 11 minutes now without any dropping, whereas with 10 I had some drop almost instantly.)

A user-configurable timeout on the command line would be very helpful.

Moreover, a more elegant solution might be to have two settings... an absolute timeout where the program ends, and then a preferred timeout which, if the time between packets exceeds, yet remains under the absolute value, the date/time and delay value is stored to be printed out upon termination. This would give the user an indication of what might be a more appropriate setting. Example: currently the kill timeout is 10 seconds. Change the code by having a preferred timeout at 10 seconds and a kill timeout at 30. Then when someone like me with delays runs, I'll get a good run and I can kill all the processes after an interval, e.g. 5 minutes, and then evaluate what the seconds were between 10 and 30 to fine-tune a future kill timeout.

I'll try to do this by some hacking, but I'm finding my Perl-like assumptions about code management are really being challenged in rust.

[jlpoolen]

An alternative might be to spit out in the log instances where the actual delay, i.e. 11-29, exceeds the preferred delay, i.e. 10, but remains under the kill delay value, i.e. 30.

[scottlamb]

I think for the debugging/data-gathering you're trying to do that you'll prefer my timedump utility. It basically keeps the RTSP streams open no matter what and dumps all the time data I could think of for each frame (when packets were actually received, what the RTP timestamps said, nominal durations of the frame where applicable) for into a SQLite3 database (plus the periodic RTCP sender reports that most cameras send relating RTP timestamps to wall time). I left it in my "playground" repository because IMHO it's a little too involved to make a good example of how to use the library.

https://github.com/scottlamb/moonfire-playground/tree/master/rtsp

[scottlamb]

btw, based on my own investigations of timestamps so far, I've found the RTP timestamps on several cameras aren't trustworthy at all. They jump forward or back when the time is adjusted on the camera (even though they're supposed to be generated from a monotonic clock source). They have a lot of jitter, even on the fixed-duration audio frames. I'm not sure what players do when they see a sequence of audio frame timestamps with frequent small overlaps and gaps.

I'm considering having Moonfire NVR support a policy like this: if there's an audio stream, assume the audio frames (which are frequent and have a fixed duration) are back-to-back with no overlap or gaps. Also assume that the frames are totally in order—if you get audio frame A1, video frame V1, audio frame A2, video frame V2, etc., then assume they happened in order A1, V1, A2, V2. Use the RTP timestamps only for (a) recovering after RTP packet loss, and (b) fine positioning of video frames within the bounds of adjacent audio frames. When audio is encoded as AAC, there's typically a "frame length" of 1,024 samples, so there's somewhere from 128 ms (@ 8 kHz sampling rate) to ~2.13 ms (@ 48 kHz sampling rate) between audio frames. In the latter case, the fine positioning is almost irrelevant; I don't think anyone cares about a video frame being 2 ms off.

It's harder to make sense of strange video timestamps on cameras that don't support audio. One option is to assume video frames happen at a fixed frame rate. Some cameras tell us what the frame rate is in the VUI options. Others don't; it could be specified by the user. This policy would mean the RTP timestamps are only used when recovering from packet loss.

That'd be in addition to Moonfire's existing separation of "media time" and "wall time" to compensate for clock frequency mismatches.

Retina doesn't have logic for this. I'm not sure if I'd put it in the Retina layer or just leave it to the application.