Ability to append to AudioBuffer

[Kukunin]

Describe the feature
Currently, if I have a source of PCM, the most suitable way to feed them into WebAudio pipe is via AudioBufferSourceNode and AudioBuffer.

While it works well if I have the whole buffer upfront, there is no way (or I didn't find) to feed PCM data as soon as I receive it. It's useful for low-latency streaming. Once a buffer is assigned to AudioBufferSourceNode, it can't be changed (https://stackoverflow.com/questions/20134384/web-audio-api-how-to-play-a-stream-of-mp3-chunks/20136399). I suspect, when I'll replace a buffer, it will occur glitches in sound.

In streaming applications there are no full PCM upfront, but they receive portions via communication channel (WebSocket or so), so there should be a way to feed PCM by portions.

Is there a prototype?
It'd be cool if there was a method like AudioBuffer.append to append a new portion of PCM.

Describe the feature in more detail
The current way to feed data dynamically is to use either createScriptProcessor or pass buffers to AudioWorkletProcessor via MessagePort. Both ways require a subtle and well-tuned speed of feeding:

• if you feed slowly you will get a chopped sound
• if you feed fast you increase a latency

[Kukunin]

My current implementation is using PlaybackBuffer, which calculates how many frames were drained since lasts tick, and reads the same amount of bytes to compensate it:

/**
 Class to calculate amount of buffer for constant feeding rate while playing
 On every tick() it calculates how much time was passed, and how much frames
 it should read to compensate.

 A second with 48000Hz sample rate contains 48k samples per second
*/
class PlaybackBuffer {
  constructor(sampleRate, channels, bufferSize) {
    this.fraction = RENDER_QUANTUM * channels;
    if(bufferSize % this.fraction !== 0) {
      throw new Error("Buffer size should be even to the fraction size: " + bufferSize);
    }
    this.bufferSize = bufferSize * channels;
    this.samplesPerMs = sampleRate * channels / 1000;
  }

  start() {
    this.lastTick = performance.now();
    this.remainder = this.bufferSize;
    return this.tick();
  }

  tick() {
    const now = performance.now();
    const samplesToFeed = (now - this.lastTick) * this.samplesPerMs + this.remainder;
    const samplesToRead = Math.round(samplesToFeed / this.fraction) * this.fraction;
    this.remainder = samplesToFeed - samplesToRead;
    this.lastTick = now;

    return samplesToRead;
  }
}

this.playback = new PlaybackBuffer(this.context.sampleRate, channels, this.bufferSize);
this._readAndFeed(this.playback.start());
this.interval = setInterval(() => {
  this._readAndFeed(this.playback.tick());
}, Math.floor(RENDER_QUANTUM / this.context.sampleRate / 1000 ));

function  _readAndFeed(amount) {
  if(!amount) return;

  let buffer = new Float32Array(amount);
  fillBufferInSomeWay(buffer);
  this.node.port.postMessage({message: 'data', data: buffer});
}

It calls .tick() function every 2ms and checks, how many bytes AudioWorkletProcessor has consumed since last tick, and send a new block to the processor via MessagePort.

You can see, how complex and fragile implementation is.

[padenot]

Why having this in a Web Audio API implementation would be any different from doing it in JavaScript ?

The proper solution to this problem is to write an SPSC lock-free ring-buffer, which is perfectly doable in JavaScript and WASM. We've specced AudioWorklet so that it can use SharedArrayBuffer and use atomics.

[Kukunin]

Actually, you might be right. A couple of notes

• thanks for the pointing to SharedArrayBuffer and SPSC lock-free ring-buffer (i know how it's called now)
• SharedArrayBuffer should be faster than my current implementation using postMessage to send every chunk
• the problem with streaming is how to handle buffer underflow and overflow. when we have a network drop for a second, we have a buffer underflow first and then, we queue the all PCM for the previous second increasing the latency. The way how it's solved might be too app-specific to put it in a standard.

Bottom line: the current situation seems ok, and instead of extending the specification, just a good documentation/proof of concept/example might be a better solution.

Thanks @padenot

[padenot]

I'm closing this, but the Audio Working Group is having a look at writing some documentation or other things to facilitate this pattern, we discussed this during a call yesterday.

[Kukunin]

I found another benefit to having the ability to feed dynamic data to AudioBufferSourceNode - is resampling. Assume that I have PCM in a different sample rate, there is the ability to set the source sample rate for AudioBuffer and browser will resample data automatically.

But when I feed data to AudioWorklet using SPSC lock-free ring-buffer, I should resample it by myself using custom code (slow and buggy).

AFAIK, there is no way to feed data to AudioWorkletNode in original sample rate and use a native node to do resampling without custom code, is there?

[padenot]

But when I feed data to AudioWorklet using SPSC lock-free ring-buffer, I should resample it by myself using custom code (slow and buggy).

Use a library that works, compiled to WASM, then.

Another approach in theory would be to create an AudioContext with a specific sample-rate, and feed it to another AudioContext with another sample-rate, the resampling will happen automatically then. Depending on the implementation, you might have varying latency figures, though.

[Kukunin]

Use a library that works, compiled to WASM, then.

that's exactly what I do

Another approach in theory would be to create an AudioContext with a specific sample-rate

OfflineAudioContext seems suitable for that, since it allows to set the sample rate. But it also requries length, which doesn't work for streaming, since there is no finite length

[padenot]

No. Just use an AudioContext with a specific sample-rate. Now, this is outside the scope of the spec, it's best to use stack overflow or something like this for programming questions.

[nathan-eko]

While implementing a ring buffer in JS is not a tremendous undertaking, the fact that we can't actually use one given the single-serving nature of the audio API's buffers makes it utterly pointless to do so as we have to continuously allocate and garbage collect new buffers in a single-threaded environment no matter what. AudioWorklets might solve for something someday when some browser vendor other than Google actually implements them, but right now there's pretty much no good solution to streaming incoming PCM data from a web socket to the audio API that works everywhere. That seems like a pretty massive oversight and there's certainly no shortage of complaints about this.

If you have some kind of incredibly simple solution to this then you should absolutely document it somewhere rather than dismissively throwing "use a ring buffer" out there like that's some kind of answer. It's not, and none of this person's findings have really changed or improved that I can see:

https://blog.mecheye.net/2017/09/i-dont-know-who-the-web-audio-api-is-designed-for/

[padenot]

AudioWorklet is being implemented by other vendors. Streaming PCM in a way that is robust and works everywhere has always been hacky, because the API was designed without this use-case in mind, as you note. There has been robust code available for a long time to do this however, free to reuse, for example in emscripten: https://github.com/emscripten-core/emscripten/blob/incoming/src/library_openal.js or in Shumway https://github.com/mozilla/shumway/blob/16451d8836fa85f4b16eeda8b4bda2fa9e2b22b0/src/flash/media/SoundChannel.ts.

Since then, we've been working on providing a good solution to solve this problem, but things take time and look easier than they are (we started working on this in #113, quite a long time ago).

As far as using ring buffers and such with AudioWorklet, @hoch had written https://developers.google.com/web/updates/2018/06/audio-worklet-design-pattern long before my comments, that has all code needed, licensed under Apache 2, and lots of info about the inner working and reason for this apparent complexity.

[GeorgeTailor]

I second the reddit thread when it comes to AudioWorklet interface.
The problems I have faced are:

• obscure and lacking docs at https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet;
• unable to use neither importScripts nor import for code separation;
• connect doesn't work as per https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletGlobalScope?? Chrome throws an error, so does Firefox.
• lack of scheduling mechanism as in AudioBufferSourceNode, I have to manually track the played samples and continuously output them, compared to audioBufferSourceNode where I could just set the start time and the web audio would handle that complex part for me, which is nice, but this is done on the main thread, when dealing with a lot of such nodes it becomes a problem. Additionally intersecting buffers is a breeze with audioBufferSourceNode, since they are automatically merged on output, which is not the same for AudioWorklet

It would be interesting to see whether those AudioWorklets are actually used.

As a side question, why not simply expose audioContext object to web workers? (simple for end consumer dev, I mean)

[GeorgeTailor]

Seems like #2423 would solve most of the problems, at least in my case

[GeorgeTailor]

@guest271314 your example works only in chrome.
AudioWorklet is an Web Audio API interface, JS modules is a part of language spec on how to separate and load code., I don't understand what your argument tries to prove.
Anyway, this is drifting away from the main topic, I will refrain from continuing this conversation.

[luca992]

If anyone is running into this issue and needs to stream audio binary in chunks and append them to a play buffer: I found a good example thanks to this stackoverflow question:
https://stackoverflow.com/questions/47600421/web-audio-api-proper-way-to-play-data-chunks-from-a-nodejs-server-via-socket

The example:
https://github.com/kmoskwiak/node-tcp-streaming-server/blob/master/client/js/app.js