investigate AudioWorklet

[mreinstein]

Chrome is about to land AudioWorklet and finally deprecate ScriptProcessorNode.

It would be awesome if Recorderjs could be used a normal WebAudio node!

https://www.chromestatus.com/features/4588498229133312

[chris-rudmin]

I would love to support audio worklet. I didn't know any browser had a working implementation yet. :)

[chris-rudmin]

@mreinstein I just took a look at the AudioWorklet feature and it seems it's main use case is for developers to create their own audio node for inclusion in an audio graph which is latency sensitive. Since this doesn't output PCM audio to a buffer, and we don't have any latency requirements, I don't think it actually fits this use case.

Thoughts?

[mreinstein]

Since this doesn't output PCM audio to a buffer, and we don't have any latency
requirements, I don't think it actually fits this use case.

I'm under the impression that AudioWorklet is a complete replacement for ScriptProcessorNode. It's supposed to eliminate the stuttering, glitching, and UI slowdowns that happen when ScriptProcessorNodes are used because worklets are doing CPU intensive stuff on a separate thread.

[ZachMoreno]

Thanks @mreinstein for bringing up this opportunity & @chris-rudmin for looking into it back in December. AudioWorklet is GA in Chrome 66 & I was able to output PCM data from an AudioWorklet in this example.

Our startup, SquadCast.fm, is dependent on the recorder & encoder. ~10% of the files recorded on our platform are negatively affected by the known issues with ScriptProcessorNode outlined in this talk by the API owner of Web Audio on Chrome, @hoch.

If you see this opportunity as being in scope, I'm happy to contribute.

[mreinstein]

All of the nodes in a webaudio graph essentially operate on pcm data (even the fancy new audioworklet.) opus-recorder takes pcm as input and produces opus encoded binary data as output. Maybe it's possible to use audio worklet as a node that just receives audio data and does not send it anywhere, because an audio worklet is intended to produce pcm as output, and this module doesnt do that.

[hoch]

Sorry for jumping in here -

Maybe it's possible to use audio worklet as a node that just receives audio data and does not send it anywhere, because an audio worklet is intended to produce pcm as output, and this module doesnt do that.

AudioWorklet can be configured as a "sink" node, which suitable for use cases like analysis and recording. It also comes with MessagePort, so you can capture/encode the audio stream and transfer it to the main scope. If the encoding is too expensive, the load can be distributed to Worker as well.

[chris-rudmin]

@ZachMoreno The existing code for the encoder should port itself pretty easily to an AudioWorklet. I would probably start a new repo for such a project as a proof of concept, and when there is more mainstream support for AudioWorklets, we could release a new version of opus-recorder.

[ZachMoreno]

Please, allow me to clarify. I'm not suggesting that we refactor the encoder from a Worker to an AudioWorklet. Rather, I'm suggesting that we refactor the recorder to process the PCM data off of the Main Thread. It's my understanding that AudioWorklet empowers us to move that processing to the Audio Render Thread. This way the data originates on the Audio Render Thread, is sent to the encoder's Worker Thread, & is only accessible from the Main Thread ondataavailable.

It should also be possible to utelize AudioWorkletNode over ScriptProcessorNode as a progressive enhancement for browsers that offer support, no?

[mreinstein]

I'm suggesting that we refactor the recorder to process the PCM data off of the Main Thread

My suspicion right now is that won't have a huge impact on performance. here is the script processor currently: https://github.com/chris-rudmin/opus-recorder/blob/master/src/recorder.js#L64-L76

It basically just reads data from the stream and posts it to a worker running in another thread. There doesn't seem to be much cpu load being introduced in here, it's mostly i/o

but then again, I haven't tested this, just gut feeling! If you're feeling ambitious and want to move that to an audio worklet, give it a try. would be interesting to get some real numbers on the perf impact.

[ZachMoreno]

It's helping me to think about this visually

[mreinstein]

ooh a UML diagram. How luxurious! :)

Yes this looks like an accurate representation of the data flow and the sequence. My previous point still stands; my 5$ gut reaction is that current vs proposed won't have a big performance impact because all the heavy lifting happens in a worker thread already; that's where the bulk of the cpu time is likely being spent.

Again, I don't want to discourage us from trying this. I just don't expect to see big performance gains.

[hoch]

I generally agree with @mreinstein from a practical view point, but I have seen many problematic cases. If the main thread suffers from some intensive processing (which is quite common), postMessage inside of onaudioprocess will suffer. Also frequent postMessage calls will generate garbages and GC is never a good news for the smooth audio operation.

@ZachMoreno I also recommend to use the internal buffering inside of Worklet instead of sending message for every 3ms. Here's a ring buffer example for AudioWorklet. (It's WIP so use at your own risk...)

[ZachMoreno]

@mreinstein Thanks kindly for reopening!
#TheFinerThings

We agree on net equal perf. I'm seeking reliability on capture. scriptProcessorNode.onaudioprocess is unreliable because it lives on the Main Thread & double buffers with the Audio Render Thread. Posting to the Worker Thread thereafter is cool as long as the buffer it's sending is accurate. Capturing on the Audio Render Thread & posting to the Worker Thread should eliminate the race conditions that result in glitches, drop-outs, & stuttering.

[mreinstein]

Okay, so the tl;dr is not much a net perf improvement but there stands to be some gain in terms of reduced latency and it's related side effects. Cool! Sounds worthwhile then.

[hoch]

I believe buffering is sort of prerequisite for the stability, the latency might not be a big win here. However, the worklet version will survive when the main thread is hammered.

FireFox is also implementing it at the moment, so hopefully at least 2 browsers will support it soon.

[mreinstein]

something that might be nice to consider; chrome and ff actually support recording opus natively. That would enable bypassing all of this userland encoding entirely (on supported platforms)

This works for me: https://jsbin.com/hedujihuqo/edit?js,console

[chris-rudmin]

@mreinstein I had created the opus-recorder interface after the MediaRecorder interface with the thought that one day that opus-recorder could be a polyfill or at least make switching between them not too costly.

[chris-rudmin]

@ZachMoreno I would love it if you would like to make a pull request to add AudioWorklet support. To support such a feature I would definitely like it to fallback to ScriptProcessorNode when worklets are not available.