WebCodecs 'opus' encoding implementation

[guest271314]

WebCodecs https://github.com/WICG/web-codecs/blob/master/explainer.md implements an 'opus' codec encoder https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/modules/webcodecs/audio_encoder.cc;l=8 that uses libopus. opusdec does not decode the resulting data when written to a single file. opusinfo logs holes in the file. I am not certain how to use opusrtp, if applicable.

I am trying to determine

1. If the 'opus' encoding is valid per the Opus specification;
2. If 1) is true, the specific type of Opus encoding implemented;
3. If 1) and 2) are true what specific Opus tools are necesarry to decode the type of Opus encoding to raw PCM or WAV

Code to reroduce the encoded Opus (I copied and opened the Blob URL in a new tab at Chrome to download the file '938f0cef-3b99-4ee8-bb1e-b0619af0fe96', then renamed to 'test1.opus') and the resulting file (which begins with OpusHead).

If this is not the approriate venue for this question kindly refer to where I should post this inquiry.

[guest271314]

WebCodecsOpusEncodingImplementation.zip

[rillian]

At first glance, your example file is not using Ogg encapsulation, which is the standard format for Opus audio in files and when streaming over http. This is why normal playback tools reject it.

If the WebCodecs is providing raw compressed packets (certainly the most flexible option) you'll need to generate appropriate OpusHead and OpusTags header and feed them to an Ogg encapsulation implementation, followed by the encoded data packets, to produce a valid .opus stream. You could use the code inside the opusrtp.c example you found to do this, just skip the part that captures the live RTP stream and parses out the audio data packets.

[guest271314]

OpusHead is written by the implementation at description https://chromium.googlesource.com/chromium/src/+/bc02c7dea85c6005c1ec15dab26b3d8cc50096ee/media/audio/audio_opus_encoder.cc#255 see https://bugs.chromium.org/p/chromium/issues/detail?id=1177021 and verified at console

if (config !== null) {
      console.log(new TextDecoder().decode(new Uint8Array(config.description));
      // ..
};

when running the code in attached .zip (OpusHead��8��������). I do not see a commit referencing OpusTag.

I do not know C. I did also locate https://github.com/libersys/rtp-ogg-opus-examples today.

This particular inquiry began with attempting to reproduce the claim at https://bugs.chromium.org/p/chromium/issues/detail?id=1161429.

Since audio/x-opus+ogg files can be concatenated and played in sequence using ServiceWorker.FetchEvent.respondWith() or at File each buffer could a discrete Opus encoded in OGG file. In lieu of that being specified and due to not knowing C I am currently experimenting with converting Float32Array from an enencoded AudioBuffer channel data to PCM suitable for input to opusenc using WebTransport as the communication API, something like https://stackoverflow.com/a/35248852

function floatTo16Bit(inputArray, startIndex){
    var output = new Uint16Array(inputArray.length-startIndex);
    for (var i = 0; i < inputArray.length; i++){
        var s = Math.max(-1, Math.min(1, inputArray[i]));
        output[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
    }
    return output;
}

$ opusenc --raw --raw-rate=44100 --raw-chan=1 --max-delay=0 --framesize=2.5 --bitrate=256 --padding=0 ~/Downloads/test.s16le ~/Downloads/output.opus

where instead of reading the file input is STDIN and output a series of discrete Opus files piped from STDOUT to the browser, e.g., https://github.com/guest271314/webtransport/blob/main/webTransportAudioWorkletWebAssemblyMemoryGrow.js.

If the WebCodecs is providing raw compressed packets

That is what I am trying to verify, conclusively.

Thanks for your reply.

[rillian]

Ok, just to make sure I understand your requirements: You're investigating a report that Opus streams from icecast stop playing after an hour or two in Chrome's HTMLMediaElement. To do so, you're trying to construct an equivalent source directly in a webpage, using pure javascript, so there's no dependency on an external service. Is that correct?

The chromium code you linked which generates the OpusHead data looks correct. Note that the buffer is retrieved through OpusAudioEncoder::GetExtraData. Most media encapsulation formats have a slot for codec configuration data like this; you'd need it to construct a .opus, .webm, or .mp4 file. I didn't trace through the callers, but from what you've reported, my hypothesis is that the AudioEncoder is returning this extra data as the first packet, followed by the actual stream data. This is a familiar technique from the internal ffmpeg api, but it's disappointing to see it in a public web api. I think it would be better attached to the config object, etc. Is the type attribute on the first audio frame different? That might be a general way to distinguish it.

So that's one confusion. The other issue is that there's no encapsulation delimiting the Opus data packets. Ideally, the AudioEncoder api would do this for you, based on media type you request. (Note the registered media type is audio/ogg; codecs=opus) As it doesn't, you'd have to implement the Ogg layer itself. It's not a lot of code, but I'm not aware of a pure js port. The rtp-ogg-opus node module you link uses C code to implement this, so you'd still need wasm or a port.

@libersys, do you have any suggestions?

[guest271314]

Ok, just to make sure I understand your requirements: You're investigating a report that Opus streams from icecast stop playing after an hour or two in Chrome's HTMLMediaElement. To do so, you're trying to construct an equivalent source directly in a webpage, using pure javascript, so there's no dependency on an external service. Is that correct?

Yes. I have tested the claim at least two different ways now. Each test spanned roughly 24 hours. I perform a modicum of experimentation and testing, the browser or machine (currently run on only RAM) can and does crash at any time. Thus, I concluded that HTMLAudioElement at Chrome does not arbitrarily stop playing Opus encoded audio when a URL is set at src. Nonetheless, as you observe, I am trying to create a version that when used locally or deployed, essentially uses the API's shipped with the FOSS browser and can be used as server and client, without cessation due to codec or API, while RAM and the caching that browsers perform, etc. might stop the stream if those flags are not properly set; effectively an infinite audio stream, a radio station using the browser as server.

That is how I arrived at testing WebCodecs 'opus' encoding implementation. WICG banned me, W3C banned me, so I do not contribute directly to the specification. My input would have been to not create exotic encoding and decoding implementations; just make each encoded audio chunk a standalone 'audio/ogg;codecs=opus' (which WebCodecs specification mentions as not being supported; on Linux the MIME type lists 'audio/x-opus+ogg') file, which can be concatenated and played at HTMLMediaElement or other media players (that is how this issue plnkr/feedback#544 arose, testing at plnkr to deploy the self-sustaining infinite stream, yet plnkr has issues, too).

but from what you've reported, my hypothesis is that the AudioEncoder is returning this extra data as the first packet, followed by the actual stream data.

That appears to be the case from the commit I located.

This is a familiar technique from the internal ffmpeg api, but it's disappointing to see it in a public web api.

If you find the time perhaps the specification authors and implementers will listen to your expert opinion.

I do not need to use WebCodecs at all. I am testing their wares after minimal contributing to the GitHub repository explainer. I was skeptical then, as I am a skpetic in general, or rather, I vet every claim individuals and institutions make, in every discipline, domain and industry that I am engaged in, without rancor, based on the scientific method.

MediaRecorder vastly outperforms WebCodecs AudioEncoder, how and why is that possible?

I think it would be better attached to the config object, etc. Is the type attribute on the first audio frame different? That might be a general way to distinguish it.

I enqueued that data to the ReadableStreamDefaultController as well

           output: (chunk, config) => {
              if (config) {
                if (!readable.locked) {
                  console.log(config, chunk);
                  controller.enqueue(new Uint8Array(config.description));
                  sw.active.postMessage(readable, [readable]);
                  audio.src = './stream';
                }
              }
              controller.enqueue(new Uint8Array(chunk.data));
            },

It should be less complicated than this to decode the encoded data; that is the entire premise of WebCodecs explainer. In the field that is not the case, even using the complementary AudioDecoder (see https://bugs.chromium.org/p/chromium/issues/detail?id=1157608#c17; https://bugs.chromium.org/p/chromium/issues/detail?id=1180089#c23).

This issue is primarly just to get the expert opinion of the kind of Opus stream implemented, to determine if the implementation is some novel endcoding that decreases rather than increases Web accessibility; even if the type of encoding is understood by you, that still does not remendy the former, thus this issue, so that there can be no denying that it is not a trivial matter to decode the 'opus' WebCodecs implementation on Chrome.

[guest271314]

I filed a Chrome bug for AudioDecoder and "Breakout Box" implementation https://bugs.chromium.org/p/chromium/issues/detail?id=1184070.

The chromium code you linked which generates the OpusHead data looks correct. Note that the buffer is retrieved through OpusAudioEncoder::GetExtraData.

Since the implementation decoder does not output expected result I am interested in determining if the process that you describe re encapsulating the AudioEncoder output in OGG container will produce a different result, decoding using means other than Chrome implementation, or if both encoding and decoding implementations might look correct yet are inherently flawed.

[guest271314]

@rillian So far the only conclusion I am able to reach is that Chrome's implementation of AudioEncoder and AudioDecoder is severely broken.

I am not sure how that API got deployed into the wild as production ready.

Even when storing the decoded chunks first, without attempting to live-stream, the output is decidedly sub-par, there is no quailty in output to speak of.

https://plnkr.co/edit/yY4CZbPNnZMT7Kwd?preview

WebCodecsAudioEncoderAudioDecoderIsBroken.zip

https://plnkr.co/edit/yY4CZbPNnZMT7Kwd?preview

[guest271314]

@rillian FWIW I created a workaround for https://bugs.chromium.org/p/chromium/issues/detail?id=1184070 plnkr https://plnkr.co/plunk/sEupuQ2pYepXQcV6.

[guest271314]

@rillian An algorithm to write WebCodecs configuraion and 'opus' EncodedAudioChunk to a single file, and play back with Media Source Extensions https://github.com/guest271314/WebCodecsOpusRecorder.

From tests so far the file is less than Opus encoded in WebM produced by MediaRecorder on Chromium.