Audio begins to cut out out after 7 minutes on an RTSP live stream

[sbadger]

After about 7 minutes the audio begins to cut in and out.
I have tried multiple options including -p, -w, -z and -r all with the same results. I have used another vlc to look at the same stream on the same Raspberry Pi and it worked fine so it isn't the stream.

An example wav file and the log file are available in my dropbox here
https://www.dropbox.com/sh/ej5cwo0eav76kah/AAC2_kOWvB-WVF-Qr7mKkt5va?dl=0

[sbadger]

I noticed that their is a small increase in both memory and cpu utilization when the problem occurs but it is small like around 2%.

[sbadger]

I will give someone $50 to fix this!!

[jehutting]

@popcornmix A couple of questions

You state in #445

--live uses audio resampling to manage buffer size, which will have the effect of forcing the media time to sync with the server's version of time.

1. Is this done in omxplayer.cpp code line 1670+ ?

        m_latency = m_latency*0.99f + latency*0.01f;
        float speed = 1.0f;
        if (m_latency < 0.5f*m_threshold)
          speed = 0.990f;
        else if (m_latency < 0.9f*m_threshold)
          speed = 0.999f;
        else if (m_latency > 2.0f*m_threshold)
          speed = 1.010f;
        else if (m_latency > 1.1f*m_threshold)
          speed = 1.001f;

        m_av_clock->OMXSetSpeed(S(speed));
        m_av_clock->OMXSetSpeed(S(speed), true, true);
        CLog::Log(LOGDEBUG, "Live: %.2f (%.2f) S:%.3f T:%.2f\n", m_latency, latency, speed, m_threshold);

2. Is this the reason of the "C"-option in jmeiners issue #305 ?

3. What's the purpose of the first m_av_clock->OMXSetSpeed(S(speed)); ?

@sbadger Is the wav file a recording of what you hear? Are the "silences" missing audio parts (so really cut-outs?).

By the way, omxplayer plays sbadger wav file not completely, due to failing EOS transmittion. Increased the timeout *2 of GetInputBuffer() in the COMXAudio::SubmitEOS() to fix it.

[sbadger]

Yes, the wav file is what I am hearing on the system recorded though my phone. Those audio drops are exactly the problem I am trying to address.

That is odd on the .wav playback I didn't get that error, but then again I didn't it back using omxplayer either. The file is only about 28 seconds long

[jehutting]

Is it a public live stream which I can use to reproduce this issue?
What's your gpu_mem settings in /boot/config.txt, your RPI model & distribution, OMXPlayer version? Don't know if this matters here :-)

[sbadger]

It is not public accessible stream but a quick google showed several sources of public accessible streams. I have tried several different options with gpu_mem which did increase the timeouts by a minute but that was all. Currently it is the stock Rasberian Jesse install that has the latest updates.

Pi:~ $ omxplayer -v
omxplayer - Commandline multimedia player for the Raspberry Pi
Build date: Sat, 02 Jan 2016 13:46:33 +0000
Version : f544084 [master]
Repository: https://github.com/popcornmix/omxplayer.git

[jehutting]

Tried some public rtsp streams, but they all run into the dreadful issue ffmpeg ending (premature) EOF the stream.

[sbadger]

I will see if I can find one with audio that will work for at least 10 minutes. I believe you can get VLC or ffmpeg to create a RTSP stream from some local media

[sbadger]

I found that audio only RTSP streams seem to work fine.

[sbadger]

@jehutting , If i upload the 20 minute of a pcap form the stream can you play it back using tcprelay or whatever to test the stream with the player?

[jehutting]

OK, if you give me a guidance in how to set up my Linux Mint 17 PC (/use tcprelay) and my RPI (B/B+/2B/3) in order to run the test. I already donloaded and installed a tcprelay on my PC. Have to figure out how it works :-)

[jehutting]

@sbadger Installed nginx on my PC, streaming with ffmpeg a rtmp stream. However I can't connect to it outside my PC. VLC running on my PC shows the stream (not great but it works). I'm stuck...

[sbadger]

My network capture was a bad idea, I am working on getting a live stream from a camera that is publicly available so you will have something to test against. A developer friend of mine suggested it might be a "ladder queue" issue where the queue is growing to large and the system can't process it properly. i will post a link to the camera once it is installed.

[sbadger]

@jehutting I finally have a publicly available stream you can use. It is just some fan noise for now but I will add a radio or something in the next couple of days. you can find it here
rtsp://70.186.38.229/axis-media/media.amp?resolution=1280x720

[jehutting]

@sbadger Already listening 1.5 hour and you just at 13:00 turned to another station. The previous station was better :-) Still listening...

[sbadger]

@jehutting LOL, it was but I didn't think I sitting very well and moved it which lost the station.
Are you seeing any audio drops? I am seeing them on the Pi here.

[jehutting]

@sbadger Until I went to bed, it played without noticing any audio drops.

I started listening at RPI-time 18:22:23
At
22:57:55 T:670895474 INFO: CDVDPlayerVideo::Decode dts:16530552378 pts:16530552378 cur:16530552378, size:2019
the last video, and at
22:57:56 T:671964103 INFO: CDVDPlayerAudio::Decode dts:16531713774 pts:16531713774 size:241
I got the last audio.
So it timed out and stopped

22:58:06 T:682037106   ERROR: COMXPlayer::interrupt_cb - Timed out
22:58:06 T:682039342   ERROR: Previous line repeats 1 times.
22:58:06 T:682039342    INFO: COMXVideo::SubmitEOS
22:58:06 T:682040789    INFO: COMXAudio::SubmitEOS
22:58:06 T:682051831   DEBUG: Normal M:16539316724 (A:16531713774 V:16530552378) P:0 A:-7.60 V:-8.76/T:0.20 (1,1,0,0) A:0% V:0% (-16539.26,20.41)
22:58:06 T:682072674   DEBUG: Normal M:16539338060 (A:16531713774 V:16530552378) P:0 A:-7.62 V:-8.79/T:0.20 (1,1,0,0) A:0% V:0% (-16539.28,20.41)
22:58:06 T:682093520   DEBUG: Normal M:16539358904 (A:16531713774 V:16530552378) P:0 A:-7.65 V:-8.81/T:0.20 (1,1,0,0) A:0% V:0% (-16539.30,20.41)
22:58:06 T:682093964    INFO: COMXVideo::IsEOS
22:58:06 T:682095002    INFO: COMXAudio::IsEOS
22:58:06 T:682095500   DEBUG: OMXClock::OMXStop

At your current time 18:05 I see the video randomly incorrect (from a randomly horizontal line (/position) it repeats this last 'correct horizontal line' till the bottom (forming vertical stripes). Audio is playing OK.

I go to bed again...ZZZZZzzzzzzzz

[jehutting]

@sbadger It is now 22:38 your (streaming) time. It is still running...no audio drops.

I got my own streaming running, it was just a matter of making the port open for public (with iptables). BUT I got not such a nice streaming as yours; a lots of video artifacts. Curious about how you set it up.

@anyone: Am I the only one looking/listening to this stream?

[sbadger]

@jehutting What firmware version are you running? My PI here has the same problem with this stream as my internal cameras.
This stream is off of and Axis camera that I opened up through the firewal

[jehutting]

@sbadger A RPI model B with Linux rpitv 4.1.15+ #831 Tue Jan 19 18:36:09 GMT 2016 armv6l GNU/Linux and a RPI 2B with Jessie-Lite Linux raspberrypi 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux. Omxplayer on both based on current omxplayer repository.
What's yours?

I see the light is turned back on...

From this afternoon on I have a lot of 'connection timed out' and EOF's causing running just for a couple of minutes, whereas before it ran smoothly for hours.

Still don't like the radio station which is currently turned on...

[sbadger]

@jehutting It is a PI2 with Linux CPi 4.1.17-v7+ #838 SMP Tue Feb 9 13:15:09 GMT 2016 armv7l GNU/Linux running wheezy.
Can you run /opt/vc/bin/vcgencmd version to see what firmware version you are on?
Here is the output form mine:
CPi:~ $ /opt/vc/bin/vcgencmd version Feb 1 2016 17:51:17 Copyright (c) 2012 Broadcom version b3dc56931507f355d503ea69397778643f7a3dc3 (clean) (release)

I will see if can get another station

[jehutting]

@sbadger My RPI2 runs on Jessie-Lite

$ vcgencmd version
Mar 15 2016 14:47:28 
Copyright (c) 2012 Broadcom
version 1bf9a9a77026af9128a339c82d72e331d3532ee4 (clean) (release)

My RPI B runs on wheezy

 $ vcgencmd version
Jan 15 2016 17:21:55 
Copyright (c) 2012 Broadcom
version 50b1ff57f80db9c96b78757d2d2cfc226ed71f93 (clean) (release)

I will update/upgrade this one to see if I get Feb 1, 2016 too.
Edit: did it on both; firmware stays the same.

Since I used omxplayer option --avdict rtsp_transport:tcp it plays again already for an hour.
Try this option too.

Thanks for changing the station :-) and greetings back (waving with my hand too).

[jehutting]

@sbadger Any improvement with omxplayer option --avdict rtsp_transport:tcp?

[jehutting]

Hmm... I just saw your omxplayer version is of 02 jan 2016.
This doesn't have 26 jan commit for issue # 411, and 14 mar commit with the avdict option added.
Update from http://omxplayer.sconde.net/ you only get the first one. The build is running a little bit behind...

[popcornmix]

Sergio has updated the build on http://omxplayer.sconde.net which now supports lavfpdopts/avdict.

[jehutting]

Thanks @popcornmix & @skgsergio!

[popcornmix]

We've updated the omxplayer in raspberrypi repo, so apt-get upgrade omxplayer should also get the latest version.

[sbadger]

I just updated to omxplayer 6c90c75 and used omxplayer -r -b --avdict rtsp_transport:tcp rtsp://$CAMERA/axis-media/media.amp?resolution=1280x720 --live for the command line and still am seeing the same problems.

[jehutting]

I was hoping that the rtsp_transport setting fixed it. With this option I am able to listen to your stream. Without it omxplayer starts complaining 'Connection Time Out' or End-Of-File. Using your stream to find out what the limit/condition is for ffmpeg to bail out; I have to dig deeper...

[jehutting]

@sbadger I have two RPIs running and the one with the --live is heavily stuttering, while the other one -without the option- is running fine. So try without this --liveoption.
I have to study # 445.

[sbadger]

@jehutting without the --live it is solid for the last 30 minutes.
So was it @skgsergio code that made this possible or just a combination of all of it?

[jehutting]

Finally we are getting somewhere...

I think the rtsp_transport: tcp does the job. Without it, the streaming uses UDP and that protocol doesn't guarantee the deliverance of packets. (Ahem...keep in mind I am not a streaming expert :)

At this time I suspect that the live option is the one causing trouble. I need to look what the -r option does too.

I appreciate it when you could run without these two options.

[jehutting]

@sbadger I had a look at the --liveoption. Still struggling with the code behind it (see my first comment on this issue). I really don't understand it. I viewed the Live logging and as soon as m_latency drops down 1.00 audio slips and at a certain moment it gets 'stable' around

Live: 0.76 (0.80) S:1.000 T:0.70

but audio 'seems to pause/resume' (no drop-outs) if not being kicked-out with EOF/Connection time out.

So don't use --live for sure :)

[its27]

I have tested the latest (as of yesterday) omxplayer (with --live on RTMP and RTSP streams) and found that if there is some packet loss (quickly disconnect network cable) the video recovers but audio disspears. The omxplayer I was using from last year recovers audio and video just fine.

[sbadger]

I have been running for a full week now with no audio issues without the --live option

[its27]

I guess the issue there is that A) Omxplayer does not send the -fflags nobuffer argument to ffmpeg resulting in a large delay (the probed data isnt discarded) B) Playback speed cant be altered to maintain the threshold value. --live worked fine on older builds.

[XionTech]

I've been fighting through this issue recently (funny enough my application is for an rtsp stream as well through an axis communications encoder). So eliminating the --live fixed the issue? I was trying to keep the lowest latency possible so I thought I needed it... but at about 10 minutes my audio starts chunking and never corrects.

[sbadger]

That was the same problem I was experiencing but at the 7 minute mark with Axis cameras. After the update and without the live option I have not had the issue come back. The latency has been <3 seconds for me which is acceptable for my usage.

[burnbrigther]

I read all the way to the bottom of this thread to see if @jehutting got his $50 bounty? LOL

[sbadger]

I reached out to @jehutting to send the money but I never received a response!
@jehutting , the money is your is you just tell me where to send it!!!

[jehutting]

@sbadger Damn it! Despite carefully checking my mailbox for years now, I still managed to miss your mail. (@burnbrigther Just kidding, I did that only for two years! LOL.)
But seriously, if the money is burning a hole in your pocket, just have a drink with your friends or give it to a charity in your neighbourhood. I'm just glad we found a solution that worked for you and hopefully for some others too!

[sbadger]

How about I donate it to the EFF?
Thank you for the fix so long ago. I use this in so many locations around work and I really appreciate you time and effort you put into writing a good app.

[jehutting]

I had more closer things to your community in mind; a food bank, that kind of donation. But do whatever you pleases the most :-).
Again, glad that I was of any help, and it is always good to read what kind of things people do with omxplayer!

[sbadger]

I already have the local food banks covered, But I will add another 50 on for you this month.