Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[audioworklet] AudioWorklets integration with WebAssembly #5

Open
juj opened this issue Sep 28, 2017 · 16 comments

Comments

@juj
Copy link

commented Sep 28, 2017

Reading the AudioWorklet reference at https://github.com/WebAudio/web-audio-api/wiki/AudioWorklet-Examples and https://github.com/WebAudio/web-audio-api/wiki/AudioWorklet-IDL-Proposal, it looks like code that would utilize WebAssembly to render audio in an AudioWorklet would look something like the following:

registerAudioWorkletProcessor('WasmInAudioWorklet', class extends AudioWorkletProcessor {
  constructor (options) {
    super(options);

    // Compile and Instantiate Wasm Module here, communicate with SharedArrayBuffer with the remaining program
    this.wasmModule = ...;
    this.wasmInstance = ...;
    this.wasmHeap = new Float32Array(...);
  }

  process(inputs, outputs, parameters) {
    for(var i = 0; i < inputs.length; ++i) { // *
      var input = inputs[i]; // *
      for(var j = 0; j < input.length; ++j) this.wasmHeap[... + j] = input[j]; // *
    } // *
    this.wasmInstance.processAudio();

    for(var i = 0; i < outputs.length; ++i) {  // *
      var output = outputs[i]; // *
      for(var j = 0; j < output.length; ++j) { // *
        output[j] = this.wasmHeap[... + j]; // *
      } // *
    } // *
  }
});

Talking with WebAssembly developers and @lukewagner, he is suggesting that it would be made possible to register the WebAssembly heap at AudioWorklet construction time, so that one would be able to optimize away the copies on the lines marked with stars // *.

Would something like this be feasible?

@hoch hoch changed the title AudioWorklets integration with WebAssembly [audioworklet] AudioWorklets integration with WebAssembly Sep 28, 2017
@hoch

This comment has been minimized.

Copy link
Member

commented Sep 28, 2017

to register the WebAssembly heap at AudioWorklet construction time

Do you mean AudioWorkletNode's object construction? Or the construction of the entire AudioWorkletGlobalScope? AudioWorklet per se is not an instantiable object. It is a singleton owned by window.

Would something like this be feasible?

Does this mean that we have to introduce some AudioWorklet-specific WASM behaviors? It sounds like WorkletGlobalScope or AudioWorkletGlobalScope should embrace such change.

PS: The exmaples on Wiki are not up-to-date, so please consider referring the current spec. I'll update the wiki pages so it won't get looked at any more.

@lukewagner

This comment has been minimized.

Copy link

commented Sep 28, 2017

I'm unfamiliar with the basic spec concepts here, so forgive me for imprecise terminology, but I would think that we would register the memory (to use as input/output) once per independent "audio stream" of which there could be multiple per global.

Also, I was imagining the registration would pass in the memory as typed arrays which means the API would work for both JS (and similarly be an optimization for JS) and wasm, not semantically caring which was used.

One potential problem and a question: judging from the code sample, inputs and outputs aren't simply arrays, but arrays of arrays. If the number of input/output arrays is static, then this seems easy enough to fix, you just register arrays of typed arrays up front. But if the number of arrays is dynamic, something fancier would be required.

@rtoy

This comment has been minimized.

Copy link

commented Sep 28, 2017

In general, the arrays are dynamic. But if you set up channelCount and channelCountMode appropriately, you can make it static. (I think channelCountMode = "explicit", channelCount = however many channels you want. Then the number of arrays is fixed.)

@lukewagner

This comment has been minimized.

Copy link

commented Sep 28, 2017

Ah, good to hear. Given the complications of a good solution for a dynamic count, perhaps this feature could be stipulated to require or set an explicit channel count.

@rtoy

This comment has been minimized.

Copy link

commented Sep 28, 2017

Which feature are we talking about here?

I think it's critically important that an AudioWorkletNode at least support dynamic channels for a single-input/single-output case. Then it is conceptually possible to define every current node with an AudioWorkletNode. And that's a great indication that the AudioWorklet has minimum set of features to be generally useful in constructing new nodes.

@lukewagner

This comment has been minimized.

Copy link

commented Sep 28, 2017

The feature of (in addition to what currently exists) being able to say up front (before process() starts being called) "use these typed arrays for input and output". In theory (I'd be interested to hear from implementers), this could allow less copying.

@hoch

This comment has been minimized.

Copy link
Member

commented Sep 28, 2017

@lukewagner

But if the number of arrays is dynamic, something fancier would be required.

Yes, this exactly is AudioWorklet's case. I believe that the 1-in-1-out node will be the majority of use cases.

The feature of (in addition to what currently exists) being able to say up front (before process() starts being called)

I think what you're asking might be relevant to this issue. Basically this storage will be a static one, so you can cache it on the global scope then it is accessed by all other objects in the scope. Not a pretty picture.

What kind of API are you envisioning? How can we do "I need this memory upfront, but expose it only inside of process() method"?

@lukewagner

This comment has been minimized.

Copy link

commented Sep 28, 2017

One more-specific API idea would be to pass the array-of-typed-arrays for input and output to the AudioWorkletProcessor constructor as arguments. IIUC, that has the right associativity.

Question: does the number of input/output channels only change in direct response to the user calling some JS method/setter or does it ever change implicitly in response to something else? If the former, then perhaps said method for changing the number of channels could also take a new array-of-typed-arrays (matching the new number of channels). Such a method would also allow being called with the same number of channels as before, which would give the user a way to change what what memory was being used at runtime.

@rtoy

This comment has been minimized.

Copy link

commented Sep 28, 2017

The number of input channels can change by itself, sort of. Say you have an oscillator connected to your worklet. It's mono. Now connect() an AudioBufferSource with 5.1 channels to it. The number of input channels changes to 6. This is under your control. But when the AudioBufferSource is done playing, it can disconnect itself from the worklet at some point, and that changes the input count back to 1.

If this really matters to you (and it seems it does), you can arrange things so that the number of input channels is fixed (using channelCount = desired count, channelCountMode = "explicit"). It's then up to you to figure out if that's really appropriate for you application.

@lukewagner

This comment has been minimized.

Copy link

commented Sep 28, 2017

Ah, I see, thanks. So maybe if you specify your buffers up-front, then that automatically sets channelCount = number-of-given-channels and sets chanelCountMode = "explicit"?

@hoch

This comment has been minimized.

Copy link
Member

commented Sep 28, 2017

One more-specific API idea would be to pass the array-of-typed-arrays for input and output to the AudioWorkletProcessor constructor as arguments. IIUC, that has the right associativity.

Currently AudioWorkletProcessor is not user-instantiable object. It is implicitly created when user creates AudioWorkletNode in the main global scope. This is a cross-thread operation, so any transaction between two instances will be serialized.

@lukewagner

This comment has been minimized.

Copy link

commented Sep 28, 2017

Ah, ok, I was reading Jukka's code in the first comment and thinking these arrays would be passed as arguments to the super() call.

@padenot

This comment has been minimized.

Copy link
Member

commented Oct 5, 2017

@padenot pinging myself so I answer here tomorrow or something.

@joeberkovitz

This comment has been minimized.

Copy link

commented Oct 12, 2017

From the WG discussion today:

  • We feel that Wasm is usable with the current AudioWorklet API, noting that more copying does take place than is strictly necessary -- but while the copying is annoying/inelegant, it's unlikely to be a big performance issue.
  • We don't want to further complicate the API for the non-Wasm use case, i.e. we'd like to layer any further options on top of the current proposal rather than fundamentally changing it. (Just as an example, we could allow internal methods to be called to configure special buffer management inside the AudioWorkletProcessor constructor.)
  • We recently enhanced AudioWorklet to dynamically adjust the number of channels based on the input to an AudioWorkletNode, and also allow static input/output channel configuration on a per-node basis.

Our conclusion is that we'd like to defer the issue of optimizing copying to the next version of Web Audio, so we come up with the best of the multiple possible approaches to this problem. In the mean time we'll learn a lot about where the meaningful gaps exist in AudioWorklet in its current form.

@joeberkovitz

This comment has been minimized.

Copy link

commented Nov 8, 2017

Just noting that we talked directly with the WebAssembly folks about this today at TPAC and resolved to keep working on it, taking a good look in the near future at a "bespoke tailored" connection between AudioWorklet and WASM.

@mdjp mdjp transferred this issue from WebAudio/web-audio-api Sep 16, 2019
@hoch hoch added this to In discussion in V2 Sep 16, 2019
@karlt

This comment has been minimized.

Copy link

commented Sep 16, 2019

An extension point to consider when designing new APIs is an alternative registerProcessorV2() method on AudioWorkletGlobalScope. From a wasm perspective, registering a function would be more natural than a class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
V2
In discussion
7 participants
You can’t perform that action at this time.