Cutted-off audio in recorded streams with Google Chrome on Mac OS X and iOS mobile devices #154

Open
andreslorek opened this Issue Feb 16, 2016 · 14 comments

Projects

None yet

5 participants

@andreslorek

We are having a problem recording streams transmitted from Google Chrome on Mac OS X and with iOS mobile devices (Adobe AIR SDK 16). The final file has a cutted-off audio. The video is fine, and the subscriptors listen the audio correctly during the transmission, but the .flv file has this problem. If we record the audio from a client using Google Chrome on Windows at the same time, this .flv has not cutted-off audio. The stream is recorded server-side using the saveAs method.

We noticed that the audio is cutted-off only if we publish audio and video together. If we only publish audio, the .flv file is not cutted-off.

The problem was tested on Red5 1.0.4, 1.0.6 and 1.0.7-M5. All this versions have the same issue.

Here is attached an example of stream recorded with Chrome and Mac (cuttedoff-chrome.flv) and with an iOS device (cuttedoff-ios.flv).

cuttedoff.zip

We worked on a simple example that you can use on Chrome+Mac and another for iOS devices.
Both examples and the server that recieves connections from them are in this repository: https://github.com/nicop85/myoflademo
The example is built to run with Red5-1.0.6.

@mondain
Member
mondain commented Feb 17, 2016

Have you tried adjusting the delayed write parameters or other tweaks that are covered in the HDFVR blog? https://hdfvr.com/blog/category/red5/

@andreslorek

Yes, when I set fileconsumer.delayed.write to false it seems to work better, but at the red5interest list Octavian advise me against that solution (https://groups.google.com/forum/#!topic/red5interest/KiFbGlD2z1w).

I also set the fileconsumer.queue.size to 1200 and still have the same problem.

I understand that the changes proposed at HDFVR are already on the Red5 build (I'm using Red5-1.0.6). Am I right?

@mondain
Member
mondain commented Feb 18, 2016

Yes, their patches are in the codebase; the delayed write feature is there to assist with bad connections where audio and video are delayed, dont arrive, or come late. It's meant to work with most scenarios.

@andtrev
andtrev commented Feb 19, 2016

I'm seeing the same issue, unable to fix it. Is this specific to Red5, or does Adobe Media Server and Wowza also have this problem?

@andreslorek

That's why I prefered keep enabled the delayed write feature, we have some clients with bad connections.
I didn't try this issue in Adobe Media Server or Wowza.

@yanalavishnu

Where did you use saveAs method? Once change the location to streamBroadcastStart method and try.

@andtrev
andtrev commented Feb 24, 2016

Just checked with Wowza, and everything records fine in Chrome on OSX, so looks like this is specific to Red5.

@andreslorek

We changed the location from saveAs call to streamBroadcastStart method and get the same problem.

@mondain
Member
mondain commented Feb 29, 2016

Have you tried creating a RecordingListener and attaching it to the stream
at streamBroadcastStart? Instead of using saveAs?

On Mon, Feb 29, 2016 at 1:58 PM andreslorek notifications@github.com
wrote:

We changed the location from saveAs call to streamBroadcastStart method
and get the same problem.


Reply to this email directly or view it on GitHub
#154 (comment).

@naicuoctavian

From my experience the sound pops in the cuttedoff-chrome.flv video is not a result of an incorrect fileconsumer.queue.size.We get those pops too when recording even over _localhost _or with high fileconsumer.queue.size values .

However a huge fileconsumer.queue.size. will solve some video recording issues with Chrome as you can read below.

Since we're also investigating the issue today we've done some tests on Firefox (NPAPI Flash 20.0.0.306) and Chrome(PPAPI Flash 20.0.0.306) on a localhost installation of Red5 1.0.6.

Chrome/Pepper Flash results

We've noticed that, even on localhost and the local network, Pepper Flash/Chrome sends the audio data with priority. We've tested Chrome/FP 20 on Windows 10, 7 and OSX .

Over a 45 seconds 720p/HD video there was a gap of almost 40 seconds, meaning that by the time we stopped the recording (second 45) all the audio tags had reached the media server while most of the the video tags (35 seconds worth of video tags) were still buffered on the client.

Here's an example (notice how the audio data that had reached Red was at 16 seconds while the video data was only at 4.9 seconds) :
image 006

Once we stopped the recording Chrome/Pepper Flash stopped sending audio tags (it had already sent all of them) and sent all the remaining video tags.

This is where delayed write and fileconsumer.queue.size has an effect on the video.

If 120 audio tags (the default value in red5.properties) arrive at Red5 but no video tags, Red5 will stop waiting for the video tags and write 25% of the queue (30 tags) to disk/flv .

That's why we recommend a much higher value for fileconsumer.queue.size .

In our tests when the HD video had rached 7 seconds the delayed mechanism already kicked in for the 1st time writing 30 audio tags to disk without their corresponding video tags.

To give you an idea, Red5 has received around 4000 tags (audio+video) from Chrome and 9000 (audio+video) from Firefox for a 1 minute 720p video.

Firefox/NPAPI

From Firefox we received all the audio and video tags in order meaning there was no real need for delayed write.

@naicuoctavian

To get the above logs in the Red5 console/file log just add:

  <logger name="org.red5.server.stream">
    <level value="DEBUG"/>
    <appender-ref ref="FILE"/>
  </logger>

above </configuration> in conf/logback.xml

@naicuoctavian

We've done a bunch more investigations and our findings are the following:

Red5 ships with a fileconsumer.queue.size which is rather low for HD videos. With a standard Red5 1.0.6 installation and any recording demo ran with Chrome we can create .flv files with frozen video. This is an old issue that's been known for ages and it's solved by increasing fileconsumer.queue.size .

With an increased fileconsumer.queue.size Red5 now has the chance to create good proper .flv files but the .flv files are now affected by another issue: interrupted sound, small pops/cutoffs in the audio track (the topic of this thread). This issue is also easily to reproduce after increasing fileconsumer.queue.size by recording a 45 seconds HD video.

From recording tens of such videos with Red5 1.0.6 we realized that:

  1. the gaps in sound are generally after the middle/towards the end of all longer videos
  2. the gaps in sound are accompanied by glitches in the video (blocky/ripped video frames) like this one:

image 013

Looking at the DEBUG logs revealed that these issues (gaps in sound + ripped video) start at the moment in the video that's not yet written to disk by Red5 when you press STOP (ns.close()) in the client.

Let's imagine a typical 45 seconds recording done from Chrome. When you press STOP at the 45 seconds mark, all of the audio frames will have reached Red5 ( because of Chrome's/PPAPI prioritization of audio packets/frames), but maybe only 10 seconds worth of video data will have reached it. This (10 seconds) is the moment when rips in the video and gaps in the sound start to be present in the final .flv.

This is the moment when BOTH RTMPConnectionExecutorand a Red5 Scheduler Worker start creating data slices. Example from video who's audio gaps start at the 18 second mark:

image 015

@naicuoctavian naicuoctavian added a commit to nusofthq/red5-server-common that referenced this issue Mar 10, 2016
@naicuoctavian naicuoctavian Patch for gaps in sound/ripped video in recorded .flv files described…
… at Red5/red5-server#154 (comment) .

ns.close() in the client triggers the RecordingListner.stop() which triggers a second processQueue()  loop.  One might already exist because EventQueueJob triggers a processQueue() every 3 seconds and scheduler.removeScheduledJob(eventQueueJobName) somehow does not remove an existing one.  EventQueueJob  is initialized in RecordingListner.start(). Each processQueue() leads to a pushMessage() in FileCosumer.java which can trigger a write with writeQueuedDataSlice(...) if the packet is of video type. 2 processQueue loops can lead to simultaneous writeQueuedDataSlice() .
9eaa5be
@naicuoctavian

I just submitted a patch/pull request to Red5/red5-server-common.

The patch was tested today with about 2 dozen videos with Chrome and Firefox with Red5 on both localhost and remote. None of the videos exhibited gaps in audio or ripped video frames. The longest one was 8 minutes of HD video.

The technical details are in the pull request.

@mondain
Member
mondain commented Mar 10, 2016

Looks good man, I'll make sure it gets into the next build/release; thanks for all the hard work and debugging you've done on this and for continuing to help out Red5! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment