OPUS HLS support

[haywirez]

Any plans to support OPUS HLS streams? What would it take? OPUS now works everywhere relevant except Safari. To illustrate, using https://video-dev.github.io/hls.js/demo/ in Chrome/FF, I'm not quite sure why this works:

https://f002.backblazeb2.com/file/songsling/hls/mp3/SYqtEZaVbjcfn7cAeTbsk9.m3u8

But this one doesn't:

https://f002.backblazeb2.com/file/songsling/hls/opus/SYqtEZaVbjcfn7cAeTbsk9.m3u8

Expected behavior

Both streams play fine

Actual behavior

Only mp3 stream plays, opus errors

Console output

771.332 | Media element detached
771.333 | Loading https://f002.backblazeb2.com/file/songsling/hls/opus/SYqtEZaVbjcfn7cAeTbsk9.m3u8
771.336 | Loading manifest and attaching video element...
771.339 | Media element attached
771.341 | No of audio tracks found: 0
771.341 | No of quality levels found: 1
771.341 | Manifest successfully loaded

771.365 | Parsing error:no audio/video samples found

[tchakabam]

Hey @haywirez

The reason is that Opus inside HLS is not really a standardized payload.

At Soundcloud we were able to introduce it using custom extensions to the player frameworks we had in place (namely GStreamer and/or FFmpeg but also a MediaSource API based client similar to Hls.js :)).

Extending that support to Hls.js was something I had planned to do since a while but haven't gotten too many requests about it either. What's your use case if I may ask? :)

[haywirez]

@tchakabam my use case is to basically 100% copy what soundcloud is doing! would love more info 😉

[tchakabam]

Ha! :) Well there is not much too it if the OPUS stream has been created as one piece. We initially were doing OTF-transcoding of single MP3 segments, which had us deal with codec-delays and such, we had to use overlapping windows. Tricky and ugly stuff.

But I guess you just want to create on OPUS stream and segment it? Do you use an Ogg container?

It would be cool to create some support for these unofficial HLS payloads. One could possibly use FLAC also if one wants lossless quality.

[haywirez]

@tchakabam universal support including FLAC would be brilliant! What I'm doing is just segmenting an input file straight to Opus segments with ffmpeg (I guess no Ogg container - do I need that for anything ?). Code is roughly looking like this:

ffmpeg(inputFilePath)
        .audioCodec(formatCodecMap.get(format)) // ends up 'libopus'
        .audioBitrate(formatBitrateMap.get(format))
        .outputOptions([
          `-hls_time ${segmentLength}`,
          `-hls_playlist_type vod`,
          `-hls_segment_filename ${path.resolve(tempDir.path) + `/${fileId}`}.%03d.ts`,
          `-hls_base_url ${basePath}`
        ])

I noticed a difference between the total lengths comparing mp3 and opus .m3u8 files, which seems quite disturbing to me:

#EXTINF:2.741500,
https://f002.backblazeb2.com/file/songsling/hls/opus/SYqtEZaVbjcfn7cAeTbsk9.000.ts

vs.

#EXTINF:2.804067,
https://f002.backblazeb2.com/file/songsling/hls/mp3/SYqtEZaVbjcfn7cAeTbsk9.000.ts

Ideally I would like to be able to seamlessly chain the downloaded segments into an AudioBuffer or temp file (not sure yet) for playback. I'm trying to get to the mythical < 1s time-to-play, but then also have a way to manipulate the playback responsively (think DJ-style queing, forward scrub, rewinds...)

[haywirez]

Anything I can check in the codebase to move this forward?

[tchakabam]

@haywirez that's maybe because of mp3 inherent delay, that means the mp3 encoder builds in a bit of silence at the beginning of your first segment in order to initialize itself. also when you are seeking to a point with mp3, you actually need to decode a few samples before that to actually be able to decode the sample that you want to get.

If you'd want to work on this, you'd have to simply make sure that MediaSource.isTypeSupported is true for the codec string in the manifest, or that if you think it's opus (based on file extension), and in that case you can create source-buffers for Opus (see buffer-controller), and then concerning remuxing nothing needs to be done, it's plain pass-through. So in principle, this shouldn't be hard :) Only that our codebase isn't the most inviting in terms of structure and readability, but then again, we're here to answer questions.

[tjenkinson]

@tchakabam are you sure opus is supported natively?

MediaSource.isTypeSupported('audio/ogg; codecs="opus"');

is false in chrome.

MediaSource.isTypeSupported('audio/webm; codecs="opus"');

is supported though

[haywirez]

Seems like mkv and webm are supported 🤔
https://www.chromestatus.com/feature/4891189287321600

[haywirez]

In any case, audio/webm is probably the way to go. Am I right that I now need to find a way to demux the .ts into a .webm container? Or better output the segments as .webm

[tchakabam]

@tjenkinson Yes, it seems they have opted for webm packaging, which is probably a sane choice, so they don't have to maintain an ogg demuxer just for opus in Chromium.

There was a time when opus/ogg was supported, but I think that has been dropped then. Or maybe that was in Firefox.

[tchakabam]

The other obvious options is of course to use the JS opus decoder and WebAudio :)

[tjenkinson]

If you use web audio you can just give it raw opus. Webaudio has other limitations though.

[tchakabam]

Which do you mean? :)

[tjenkinson]

On iOS for example it will be muted with the mute switch, and stop when the browser is no longer the active app. Also I tried to seamlessly switch between segments before with web audio without much success.
It is possible in a convoluted way using multiple nodes and switching gains part way through segments though.

[tchakabam]

Oh yeah the mobile browsers, love them :)

Also I tried to seamlessly switch between segments before

What do you mean exactly by that? Gapless playback across tracks of an album?

[tjenkinson]

I mean hls segments

[tchakabam]

well you need to implement the OPUS decoding as part of the streaming pipeline. basically that would mean use the JS opus decoder (that is why you shouldn't use WebAudio API to decode), and there are a few tricks depending on how your OPUS is encoded to be aware of.

it is definitely not impossible. OPUS bit-streams of the same "profile" can definitely be decoded (same decoder instance) and rendered (not need to re-init the audio device) "seamlessly".

we had done it back then with GStreamr/skippy: https://github.com/soundcloud/skippyHLS, see the OPUS decoder part in src.

[haywirez]

@tjenkinson did you try to implement some kind of a parallel playback position / elapsed time tracking solution for stitching the segments together? I'm tracking this, seems like a problem as the API is not sample accurate:
https://github.com/WebAudio/web-audio-api/issues/296

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[haywirez]

Sorry, still down on the todo list 😢 Still planning to have a proper look & attempt

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jonjomckay]

Does anyone have any pointers on how this could be added in? I'm happy to try and create a PR as this would be very handy in one of my side projects!

[tchakabam]

@jonjomckay Step one should be defining the test content. Afaik it's as simple as generating an opus stream and cutting it on frame boundaries plus writing the m3u8 file for it. That could be done likewise with Gstreamer (hlssink) or ffmpeg (or whatever other way you come up with).

Then, there are several possibilities for playing them back: Local-transcoding to AAC or MP3, native MSE support for OPUS or WebAudio decoding (and then again, decoding using with WASM-based libOpus or native WAAPI decoding support audioContext.decode...).

Depending which one you choose you have to do different things. I would try to support WebAudio based decoding/playback because MSE support of Opus might stay flaky or inexistent, and transcoding to MP3 is using more than three times the CPU time you'd need to "just" decode it in the first place.

So let's go the WebAudio direction: Hls.js BufferController is very coupled to MSE and the whole API expects a HTMLMediaElement. Also, all the scheduling of downloading segments are based on HTMLMediaElement events. That is stuff that you'd have to mock-up/implement for a WebAudio based playback. And then you'd have to overload the BufferController with a WebAudio based impl to get the data and decode/schedule it on the audio-context.

For starters you would make sure these opus segments are passed through to the buffer-controller by making sure they are using the pass-through-remuxer (not attempt any parsing / fmp4 transmuxing on them).

Once you get the opus segments into BufferController you can decode it and/or pass it to any underlying web-audio player.

Finally, you need to create a "fake" HTML5 media-element API implementation that will expose at least currentTime and fire events like seeking and seeked or waiting and basically have it run by your Webaudio based player.

[jonjomckay]

@tchakabam thanks for the in-depth answer! I'll see if I can come up with something

[haywirez]

@jonjomckay if using ffmpeg, you will have to use the more generic segment rather than the hls muxer command to generate valid segments in formats that are different from those in the HLS spec (if I understand correctly, everything else besides aac, mp3 or ac-3). Took me some time to figure this out. Sample command:

ffmpeg -i yourfile.wav -f segment -segment_time 30 -b:a 96k -c:a libopus -segment_list playlist.m3u8 -segment_list_type m3u8 segment_%03d.opus

[DocMarty84]

Following this for interest 😄