Specify static Map of audio headers that can be prepended to Uint8Array for decodeAudioData()

[guest271314]

Describe the feature
Briefly describe the feature you would like WebAudio to have.

decodeAudioData() does not decode audio data lacking a header.

Is there a prototype?
If you have a prototype (possibly using an AudioWorkletNode), provide links to illustrate this addition. This is the best way to propose a new feature.

No prototype. There is a design pattern.

Describe the feature in more detail
Provide more information how it does and how it works.

The basic algorithm #337 (comment)

I'm a bit late to this one but I can't see how the decoding of partial content could be done automatically with decodeAudioData(). File formats such as OGG Vorbis (and MP3 if I remember correctly) store the information required to decode the audio in the file header. The decoder could store that information somewhere but it would have no idea if N number of calls to decodeAudioData() were sequential and/or partial chunks from the same source file. Also, the structure of the source file needs to be considered, some formats can only be decoded in chunks (frames) of specific sizes.

It's actually possible to "stream" audio now using decodeAudioData() if the programmer understands the format of a particular audio file. I managed to do this with OGG Vorbis (load and decode it in chunks) a while ago by storing the required decoding information and prepending it to the beginning of each chunk prior to decoding. The data loading and splicing can be done in a worker and then transferred (zero-op) to the main thread for decoding.

Basic design pattern

const res = [];
// static Map of codec headers
const opusHeader = AudioContext.headers.get("audio/ogg;codecs=opus");
async function processStream({value, done}) {
  if (done) {
    return this.closed;
  }
  for (let i = 0; i < value.length; i++) {
    res[res.length] = value[i];
  }
  // wait for header and enough audio for 1 second playback
  if (res.length > 43000) {
     const init = await decodeAudioData(new Uint8Array(res.splice(0, res.length)).buffer);
     audioWorklet.port.postMessage({init}, [init.getChannelData(0)]);
  } else {
     const audioDataWithoutHeader = res.splice(0, res.length);
     audioDataWithoutHeader.unshift(opusHeader); // add header
     const next = await decodeAudioData(new Uint8Array(audioDataWithoutHeader).buffer);
     audioWorklet.port.postMessage({next}, [next.getChannelData(0)]);
  }
}
fetch("/path/to/oggContainerOpusCodec")
.then(response => response.getReader())
.then(reader => {
    reader.read.call(reader).then(processStream.bind(reader))
})
.catch(console.error);

This will allow streaming any audio without using MediaSource extensions or HTMLMediaElement.

[padenot]

This is what we're doing in Web Codecs.

[padenot]

As noted a bit everywhere (in closed and open issues on this repo and the v2 repo, and public minutes of calls):

• we're not taking any features, those go in the v2 repo
• the web audio api's scope is to do process and audio output device access, not decoding. decodeAudioData has too many problems, and we've decided against fixing them by piling hacks on top of it, and instead add a proper API to do this, unrelated to the Web Audio API (but that composes well with the Web Audio API)
• implement what is needed here in js/wasm has been done numerous times before, and works well, allowing authors that need to solve this use case to wait until WebCodec is ready.

[rtoy]

Let me also add that we will work closely with the WebCodecs group to make sure that it can provide solutions to the problems people have had with decodeAudioData. Ideally, WebCodecs should be able to do everything that decodeAudioData does and more.