Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Be explicit on how browsers should handle "priming samples" #1091

Closed
jakearchibald opened this issue Nov 29, 2016 · 11 comments
Closed

Be explicit on how browsers should handle "priming samples" #1091

jakearchibald opened this issue Nov 29, 2016 · 11 comments

Comments

@jakearchibald
Copy link

@jakearchibald jakearchibald commented Nov 29, 2016

https://jakearchibald.github.io/aac-decode-bug/

Chrome stable, Firefox and Edge display a large gap at the start of the decoded AAC. This is added by the encoder as a series of priming samples.

Question is, should these priming samples be present in the decoded audio? The Apple document above says:

a playback system must trim the silent priming samples to preserve correct synchronization

So I guess it depends on who the playback system is. Is it the browser when it decodes, or is it the web audio API user that calls start().

My gut feeling is that Safari is doing the right thing, it certainly feels the most developer-friendly. If it's decided that the gap should be present, we must have an easy way to access this metadata so we know it's there and can skip it.

@padenot
Copy link
Member

@padenot padenot commented Nov 29, 2016

This will probably be handled as part of the grand Media Decoder spec we've planned for v.next.

@rtoy
Copy link
Member

@rtoy rtoy commented Nov 29, 2016

I think this is pretty complicated, at least after raising the issue with the ffmpeg developers a while back. (Chrome uses ffmpeg for decoding audio.) They said at the time that it was fairly difficult because the meta data was optional so there was no way to know for sure if the encoded stream actually included it. If the meta data included the information, then ffmpeg would remove the priming frames.

There are also remainder frames at the end, and the developers said that couldn't be reliably removed either.

Oh, this was for aac and mp3. For ogg, ffmpeg produces the expected output. I think Opus and FLAC decoder produces the expected output.

@jakearchibald
Copy link
Author

@jakearchibald jakearchibald commented Dec 1, 2016

@padenot in the meantime, do you know whether browsers should remove these gaps as part of web audio decoding, or is that still up for debate? A decision here would be useful to point the various browsers at and try and get some consistency.

@padenot
Copy link
Member

@padenot padenot commented Dec 1, 2016

@jakearchibald, we (some people at Mozilla) are currently investigating a number of formats, codec specifications, bad encoders present in the wild, that kind of thing.

The idea here is that we should be able to roundtrip to wav (wav -> compressed format -> wav) and have the same file (more or less, of course taking the fact that compression can be lossy).

It may be that it is not feasible, and that for some formats, UAs will have to use heuristics.

@JohanRonstrom
Copy link

@JohanRonstrom JohanRonstrom commented Dec 9, 2016

We are also having this issue, and are currently trying to find workarounds for our products, for mp3 (LAME tag) and m4a (ITUNSMB tag).

In files where the tags do show up as expected (as seen with ffprobe), getting the start value from webaudio would be a huge first improvement.

You would of course expect decodeAudioData to be able to load the audio as intended and specified by the encoder (trimmed).

@notthetup
Copy link
Contributor

@notthetup notthetup commented Dec 12, 2016

Semi-relevant blog post related to this about MAD (libmad). Not sure if any UA uses libmad, but it seems that it also had some issues with priming samples.

https://thebreakfastpost.com/2016/11/26/mp3-decoding-with-the-mad-library-weve-all-been-doing-it-wrong/

@rtoy
Copy link
Member

@rtoy rtoy commented Dec 12, 2016

Chrome's WebAudio implementation currently uses ffmpeg to handle all of the decoding. Adding heuristics in WebAudio is not the preferred approach since the implementation currently doesn't need to know anything about the contents of the file except for the estimated duration. The removal of these things are best handled by ffmpeg.

@hoch
Copy link
Member

@hoch hoch commented Dec 19, 2016

I agree with @rtoy here. I had to use 'heuristics hack' to address the M4A container priming frames, but it backfired. (A simple OS update will ruin the fix.)

Even if we handle this issue at the level of unified decoder (i.e. FFMPEG) in Chrome, developers still have to fight the cross-browser inconsistency. It is a sad situation, but developers have to sniff the platform/browser or try to decode a short silent file to see the decoded result.

:(

@mdjp
Copy link
Member

@mdjp mdjp commented Oct 25, 2018

This is not a spec level problem, but it may be worth adding a note to the spec to ensure that developers are aware. This would also have to be addressed in any new encoder API.

@rtoy
Copy link
Member

@rtoy rtoy commented Jun 25, 2019

F2F: Do what @mdjp says and add a note.

@rtoy rtoy removed the High Priority V2 label Jun 25, 2019
@mdjp
Copy link
Member

@mdjp mdjp commented Sep 17, 2019

Should be handled by web codec.

@mdjp mdjp closed this Sep 17, 2019
V2 Preparation (DO NOT USE) automation moved this from To do to Done Sep 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.