Audio Worklets #16449

juj · 2022-03-08T10:01:03Z

Add support for Wasm Audio Worklets, based on Wasm Workers.

Wasm Audio Worklets enables code to create Wasm-based Web Audio processing nodes, that run on a dedicated thread of their own. (all such nodes run sequentially on a single main audio thread, there won't be a single thread for each node)

Fix Chrome not continuing the AudioWorklet processing if a number 1 is returned from the callback - must return 'true' specifically. Add new tone generator sample. Adjust comment. Use MessagePort.onmessage instead of add/removeEventListener(), since onmessage .start()s the MessagePort automatically. Fix name noise-generator to tone-generator Improve assertions.

…IS_AUDIO_WORKLET.

# Conflicts: # emcc.py # src/settings.js

…depended on.

juj · 2022-03-16T11:44:04Z

This PR has now been merged on top of latest main, and is good for final review and landing.

src/audio_worklet.js

kripken · 2022-03-25T16:17:24Z

What do you think about adding a doc page with an overview of the API and some simple examples? I think that would be useful in general for users later but also help in review.

How does this relate to WebAudio? I see webaudio.h also has methods related to that and not to audio worklets. I'm not very familiar with either WebAudio or AudioWorklets however so maybe that's something obvious, apologies if so.

Also please add a changelog entry.

kripken · 2022-03-25T16:21:34Z

cc @hoch , this might interest you (no worries if you don't have time, of course).

tklajnscek · 2022-04-06T08:38:30Z

Hey @juj!

Finally got around to testing this out along with the general wasm workers support. Thanks for all the work on this!

The first realization I had was that with pure wasm workers none of the standard library code will work multithreaded (e.g. malloc will not be thread-safe etc) and we have quite a bit of code (from 3rd parties as well) that would need to change so for the foreseeable future pthreads is what we're stuck with.

Looking through the tests I realized we can run pthreads and wasm workers side by side so I set out to test running just the audio worklet bit as a wasm worker using this the code from this PR, but with pthreads enabled the JS doesn't load in an audio worklet context.

First it asserts, but removing that it tries running PThread.init and fails etc.

So I went and took this PR's branch, rebased it on latest main for ease of testing and made a few changes to skip over any remains of PThread code when not running in a PThread context.

Here's the commit - I can't do an automatically mergable PR to this branch because it's on a rebased branch:
tklajnscek@1824e25

This makes the simple example work.

I'll run some more complex setup and update here.

tklajnscek · 2022-04-11T08:44:16Z

Here's quick status update...

I was able to make it work with a combination of pthreads for normal threads and wasm workers for the audio worklet code.

I used the synchronization primitives from wasm_worker.h to setup the communication between the audio worklet and the pthreads and it worked fine.

In this specific use case I had to get a middleware product ready for production so I actually trimmed down the worklet code so much that in the end the pthread part uses wasm_worker sync primitives in C++ code compiled to wasm like you'd expect, but the audio worklet part is pure javascript that uses Atomics.notify etc directly and it works fine and avoids a dependency on this PR in our deployment :)

I will schedule some time in the following weeks to look at our full blown audio playback engine in the context of this PR and to see what it would take to make the audio worklet code not depend on pthreads at all, but have some other priorities I need to deal with first (like always) :)

juj · 2022-04-12T18:06:51Z

What do you think about adding a doc page with an overview of the API and some simple examples? I think that would be useful in general for users later but also help in review.

Sure, I'll add a doc page. The added tests serve as simple examples, so pointing users to them to get started.

How does this relate to WebAudio? I see webaudio.h also has methods related to that and not to audio worklets. I'm not very familiar with either WebAudio or AudioWorklets however so maybe that's something obvious, apologies if so.

Audio Worklets are a subpart of Web Audio API. So in order to enable integration with Audio Worklets, we need to have at least a minimal integration/understanding of Web Audio AudioContext object with the PR. That is why I added a shallow shim that allows creating an AudioContext from C side, and then a C&JS mechanism to manage the Wasm AudioContext<->int opaque handle binding.

Also please add a changelog entry.

Will do, I'll address review once I get through my next round of bug PRs.

kripken · 2023-01-26T19:13:59Z

src/library.js

@@ -2358,7 +2358,7 @@ mergeInto(LibraryManager.library, {
                               "  };\n" +
                               "} else " +
 #endif
-#if USE_PTHREADS
+#if USE_PTHREADS && !AUDIO_WORKLET


Does this mean that just enabling AUDIO_WORKLET affects code not running in the worklet?

When building with AUDIO_WORKLET enabled, we cannot unconditionally use performance.timeOrigin + performance.now(), since those don't exist in AudioWorklets.

In that case, we must use one of the forms below that feature check for presence of performance.now(), and only use that if available.

Line 2376 below will still implement the same behavior for non-Audio Worklet threads, so they get performance.timeOrigin + performance.now().

juj · 2023-01-26T19:34:41Z

CI is giving an error

em++: error: LLVM version for clang executable "C:/Users/circleci/emsdk/upstream/bin\clang.exe" appears incorrect (seeing "17.0", expected "16") [-Wversion-check] [-Werror]

upstream LLVM bumped recently?

sbc100 · 2023-01-26T19:35:53Z

CI is giving an error

em++: error: LLVM version for clang executable "C:/Users/circleci/emsdk/upstream/bin\clang.exe" appears incorrect (seeing "17.0", expected "16") [-Wversion-check] [-Werror]

upstream LLVM bumped recently?

Yup, updating to latest main should resolve those.

kripken

Overall looks good aside from a few notes.

I am not really a user of these APIs so I don't have thoughts on the API usability aspect, but from all the feedback from people this seems useful so I have no concerns there.

src/library_webaudio.js

src/shell.js

kripken · 2023-01-31T21:02:24Z

src/shell.js

@@ -90,6 +92,10 @@ var quit_ = (status, toThrow) => {
 // Determine the runtime environment we are in. You can customize this by
 // setting the ENVIRONMENT setting at compile time (see settings.js).

+#if AUDIO_WORKLET
+var ENVIRONMENT_IS_AUDIO_WORKLET = typeof AudioWorkletGlobalScope !== 'undefined';
+#endif


Maybe it makes sense to make this new env status be more like ENVIRONMENT_IS_PTHREAD? That is, parallel to the main environments, and not one of a disjoint set of the main states. Here is that code:

emscripten/src/shell.js

Lines 120 to 126 in 008017b

// Three configurations we can be running in:

// 1) We could be the application main() thread running in the main JS UI thread. (ENVIRONMENT_IS_WORKER == false and ENVIRONMENT_IS_PTHREAD == false)

// 2) We could be the application main() thread proxied to worker. (with Emscripten -sPROXY_TO_WORKER) (ENVIRONMENT_IS_WORKER == true, ENVIRONMENT_IS_PTHREAD == false)

// 3) We could be an application pthread running in a worker. (ENVIRONMENT_IS_WORKER == true and ENVIRONMENT_IS_PTHREAD == true)

// ENVIRONMENT_IS_PTHREAD=true will have been preset in worker.js. Make it false in the main runtime thread.

var ENVIRONMENT_IS_PTHREAD = Module['ENVIRONMENT_IS_PTHREAD'] || false;

If this is meant to be a new main environment then arguably it should go into the docs in var ENVIRONMENT in settings.js, in particular. But I think it's more like a pthread, a kind of internal state?

ENVIRONMENT_IS_AUDIO_WORKLET is internal, but it can be useful for JS lib developers to access, to get an outlined short hand to be able to avoid having to type typeof AudioWorkletGlobalScope !== 'undefined' out in full in their own detection code.

I am not sure if it would make sense to have that be part of ENVIRONMENT, since it would break the Single Source of Truth rule. I.e. what would it mean to build with a mismatched

-sAUDIO_WORKLET -sENVIRONMENT=web
or
-sENVIRONMENT=web,audioworklet
command line?

With Web Workers I can see it being different that someone might want to build with pthreads disabled, but with -sENVIRONMENT=worker if they plan to run the whole code in a worker. But a similar "I'll do everything manually" deployment cannot happen (is not supported) with Wasm Audio Worklets.

I think we might be saying the same thing. I agree that ENVIRONMENT_IS_AUDIO_WORKLET is useful for JS devs.

What I'm unsure about is line 117, below - we use ENVIRONMENT_IS_AUDIO_WORKLET there in the same way e.g. ENVIRONMENT_IS_NODE. Note that ENVIRONMENT_IS_PTHREAD is not present on line 117. I am asking whether we can treat ENVIRONMENT_IS_AUDIO_WORKLET more like ENVIRONMENT_IS_PTHREAD.

I guess it doesn't matter since audio worklets and SHELL have no overlap, but I think it would be simpler to have ENVIRONMENT_IS_AUDIO_WORKLET be like ENVIRONMENT_IS_PTHREAD.

Hmm, line 117 reads:

var ENVIRONMENT_IS_SHELL = !ENVIRONMENT_IS_WEB && !ENVIRONMENT_IS_NODE && !ENVIRONMENT_IS_WORKER && !ENVIRONMENT_IS_AUDIO_WORKLET;

Note that ENVIRONMENT_IS_PTHREAD is not present on line 117.

I think the reason that ENVIRONMENT_IS_PTHREAD is not present on that line is that ENVIRONMENT_IS_PTHREAD implies ENVIRONMENT_IS_WORKER, so testing !ENVIRONMENT_IS_PTHREAD there would be redundant.

I think it would be incorrect to omit && !ENVIRONMENT_IS_AUDIO_WORKLET on that check. I am thinking maybe someone would like to do a build that would work simultaneously on node.js and with audio worklets enabled, but they might not just ever initialize any audio worklets when they run against node.js. In such case, if && !ENVIRONMENT_IS_AUDIO_WORKLET was omitted there, then the Audio Worklet thread would incorrectly get ENVIRONMENT_IS_SHELL be true and likely execute code that will later fail.

If I now locally test just omitting && !ENVIRONMENT_IS_AUDIO_WORKLET from that check, I get test suite errors with

RuntimeError: Aborted(Assertion failed: shell environment detected but not enabled at build time. Add 'shell' to `-sENVIRONMENT` to enable.)

so we would then have to enforce that people don't simultaneously target shell and audio worklets at the same time? I think that might be convenient to have, say, I might have a game JS file that would know to run in headless mode in node.js (e.g. for unit tests or a server), but then if the same js file is run in a browser, it would know to pick up webgl rendering and audio worklet enabled audio.

I see now, thanks. Makes sense!

…TS_GLOBALTHIS internal setting to help simplify this.

… runs.

…ore#18402.

juj · 2023-02-02T19:40:44Z

I noticed when merging with main that there were some regressions with compatibility against main. Updated with fixes to those, and fixed the test suites to work again. Would be be closer towards a merge? :)

kripken · 2023-02-02T20:41:43Z

emcc.py

@@ -2352,6 +2352,19 @@ def phase_linker_setup(options, state, newargs):
      settings.WASM_WORKER_FILE = unsuffixed(os.path.basename(target)) + '.ww.js'
    settings.JS_LIBRARIES.append((0, shared.path_from_root('src', 'library_wasm_worker.js')))

+  settings.SUPPORTS_GLOBALTHIS = settings.MIN_CHROME_VERSION >= 71 and settings.MIN_EDGE_VERSION >= 79 and settings.MIN_FIREFOX_VERSION >= 65 and settings.MIN_IE_VERSION == settings.TARGET_NOT_SUPPORTED and settings.MIN_SAFARI_VERSION >= 120100 # and settings.MIN_NODE_VERSION >= 120000


Rather than baking this into emcc.py, see

https://github.com/emscripten-core/emscripten/blob/main/tools/feature_matrix.py

The numbers can be in there and this place can call caniuse().

kripken · 2023-02-02T20:42:47Z

Lgtm aside from the caniuse issue and the discussion around the comparison to ENVIRONMENT_IS_PTHREAD (but that discussion doesn't matter much).

juj · 2023-02-03T11:34:17Z

Lgtm aside from the caniuse issue and the discussion around the comparison to ENVIRONMENT_IS_PTHREAD (but that discussion doesn't matter much).

Updated code to utilize caniuse. On the ENVIRONMENT_IS_PTHREAD / ENVIRONMENT_IS_AUDIO_WORKLET part, I think the current code form is required.

… IN_TEST_HARNESS increased in size (added 'typeof ENVIRONMENT_IS_AUDIO_WORKLET&&ENVIRONMENT_IS_AUDIO_WORKLET)' code)

kripken

lgtm!

Great to have this functionality, I know many have been waiting for it.

Jonathhhan · 2023-02-16T01:50:26Z

Thank you very much for implementing this. :)

This was referenced Mar 8, 2022

Wasm workers #12833

Merged

Basic support for running Emscripten code as AudioWorklets #12502

Open

juj added 2 commits March 8, 2022 12:52

Add src/audio_worklet.js to eslint ignore

d2d9176

Optimize code size

10b7322

Jonathhhan mentioned this pull request Mar 11, 2022

OF 0.12.0 Release Checklist openframeworks/openFrameworks#6692

Closed

77 tasks

juj added 15 commits March 11, 2022 12:20

Add emscripten_current_thread_is_audio_worklet(), remove ENVIRONMENT_…

de13987

…IS_AUDIO_WORKLET.

Fix to work with Closure

194896b

Simplify MINIMAL_RUNTIME shell module preamble generation.

cfca4a5

Fix Closure and simple AudioWorklet creation

75a744d

Fix Module import for AudioWorklets, and move towards globalThis

1105580

Disable -sAUDIO_WORKLET + -sTEXTDECODER=2 combo

59475e5

Fix shell case

2c0dacb

Mark AUDIO_WORKLET and SINGLE_FILE not mutually compatible

397820c

Default runtime support

9c12ec3

Merge remote-tracking branch 'origin/main' into audio_worklets

0a5cc67

# Conflicts: # emcc.py # src/settings.js

Revert unnecessary code

629f7b9

Fix merge

c666d74

AudioWorklets functioning in default runtime.

674469d

Remove #if WASM_WORKERS checks since Wasm Workers is unconditionally …

7b56e06

…depended on.

Add more interactive tests

26bd5b6

Test Audio Worklets with Closure

666d657

juj enabled auto-merge (squash) March 16, 2022 14:26

msabwat reviewed Mar 23, 2022

View reviewed changes

src/audio_worklet.js Show resolved Hide resolved

kripken reviewed Jan 26, 2023

View reviewed changes

juj added 2 commits January 26, 2023 23:38

Merge remote-tracking branch 'origin/main' into audio_worklets

1555f63

Merge remote-tracking branch 'origin/main' into audio_worklets

421946c

kripken reviewed Jan 31, 2023

View reviewed changes

juj added 10 commits February 1, 2023 16:43

Merge remote-tracking branch 'origin/main' into audio_worklets

65895dc

Fix MINIMAL_RUNTIME test run

5bbfa02

Fix tests

d5d5050

Fix test

072a41f

Remove duplicated part

e7e8b24

Clean up shell_minimal.js handling of Module import, introduce SUPPOR…

ae19a6e

…TS_GLOBALTHIS internal setting to help simplify this.

Fix browser test harness to take into account errors in audio worklet…

3a46c1a

… runs.

Fix Audio Worklet futex waits to work after change in PR emscripten-c…

aac666c

…ore#18402.

Fix typo

81b1fd8

Fix Audio Worklets stack exporting after upstream change.

db20434

kripken reviewed Feb 2, 2023

View reviewed changes

Move check for globalThis into feature_matrix.py

3b31c2b

juj added 3 commits February 3, 2023 13:34

Flake

c871919

Merge remote-tracking branch 'origin/main' into audio_worklets

b494b1e

Allow more slack in test_small_js_flags because the scaffold code for…

c16c298

… IN_TEST_HARNESS increased in size (added 'typeof ENVIRONMENT_IS_AUDIO_WORKLET&&ENVIRONMENT_IS_AUDIO_WORKLET)' code)

kripken approved these changes Feb 3, 2023

View reviewed changes

juj added 2 commits February 3, 2023 22:16

Merge remote-tracking branch 'origin/main' into audio_worklets

605319e

Merge remote-tracking branch 'origin/main' into audio_worklets

1d2df94

juj force-pushed the audio_worklets branch from ca057cd to 1d2df94 Compare February 5, 2023 08:22

juj merged commit 5402fc9 into emscripten-core:main Feb 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Audio Worklets #16449

Audio Worklets #16449

juj commented Mar 8, 2022

juj commented Mar 16, 2022

kripken commented Mar 25, 2022

kripken commented Mar 25, 2022

tklajnscek commented Apr 6, 2022

tklajnscek commented Apr 11, 2022

juj commented Apr 12, 2022

kripken Jan 26, 2023

juj Jan 26, 2023

juj commented Jan 26, 2023

sbc100 commented Jan 26, 2023

kripken left a comment

kripken Jan 31, 2023

juj Feb 1, 2023

kripken Feb 2, 2023 •

edited

juj Feb 3, 2023 •

edited

kripken Feb 3, 2023

juj commented Feb 2, 2023

kripken Feb 2, 2023

juj Feb 3, 2023

kripken commented Feb 2, 2023

juj commented Feb 3, 2023

kripken left a comment

Jonathhhan commented Feb 16, 2023

	// Three configurations we can be running in:
	// 1) We could be the application main() thread running in the main JS UI thread. (ENVIRONMENT_IS_WORKER == false and ENVIRONMENT_IS_PTHREAD == false)
	// 2) We could be the application main() thread proxied to worker. (with Emscripten -sPROXY_TO_WORKER) (ENVIRONMENT_IS_WORKER == true, ENVIRONMENT_IS_PTHREAD == false)
	// 3) We could be an application pthread running in a worker. (ENVIRONMENT_IS_WORKER == true and ENVIRONMENT_IS_PTHREAD == true)

	// ENVIRONMENT_IS_PTHREAD=true will have been preset in worker.js. Make it false in the main runtime thread.
	var ENVIRONMENT_IS_PTHREAD = Module['ENVIRONMENT_IS_PTHREAD'] \|\| false;

Audio Worklets #16449

Audio Worklets #16449

Conversation

juj commented Mar 8, 2022

juj commented Mar 16, 2022

kripken commented Mar 25, 2022

kripken commented Mar 25, 2022

tklajnscek commented Apr 6, 2022

tklajnscek commented Apr 11, 2022

juj commented Apr 12, 2022

kripken Jan 26, 2023

Choose a reason for hiding this comment

juj Jan 26, 2023

Choose a reason for hiding this comment

juj commented Jan 26, 2023

sbc100 commented Jan 26, 2023

kripken left a comment

Choose a reason for hiding this comment

kripken Jan 31, 2023

Choose a reason for hiding this comment

juj Feb 1, 2023

Choose a reason for hiding this comment

kripken Feb 2, 2023 • edited

Choose a reason for hiding this comment

juj Feb 3, 2023 • edited

Choose a reason for hiding this comment

kripken Feb 3, 2023

Choose a reason for hiding this comment

juj commented Feb 2, 2023

kripken Feb 2, 2023

Choose a reason for hiding this comment

juj Feb 3, 2023

Choose a reason for hiding this comment

kripken commented Feb 2, 2023

juj commented Feb 3, 2023

kripken left a comment

Choose a reason for hiding this comment

Jonathhhan commented Feb 16, 2023

kripken Feb 2, 2023 •

edited

juj Feb 3, 2023 •

edited