[worklets] relationship to JavaScript agent clusters

[annevk]

For the worklets that support StructuredSerialize and StructuredDeserialize, see also WebAudio/web-audio-api#1194 for the details of that, we'll need to define how worklets relate to JavaScript agent clusters. Otherwise it's unclear whether SharedArrayBuffer always fails or always succeeds.

Maybe as part of whatwg/html#2518 or a follow-up.

cc @bakulf @domenic @lars-t-hansen

[annevk]

I'll probably fix this as part of whatwg/html#2521. The tentative conclusion is that each worklet is its own Agent Cluster so it cannot share memory with anyone but itself.

[annevk]

Note that the tentative conclusion changed due to audio worklets. They should now be part of their owner's agent cluster (while being their own agent) and #385 needs to be fixed to get it properly defined.

Tentative [[CanBlock]] value is false.

[annevk]

If a worklet has two globals, such as paint worklets, should those be in the same agent? I guess it technically doesn't matter, but it seems a little cleaner if they are. Is there some terminology available that gets you all of the globals?

[domenic]

I'd also like to use this issue (or maybe a separate one?) as a reminder to the worklets teams to write web platform tests for the [[CanBlock]] of worklets, whatever it ends up being. We have tests for the other agents in web-platform-tests/wpt#5569 which you can emulate.

[annevk]

Per WebAudio/web-audio-api#1194 (comment) it should be false, though it would be good for @padenot and @hoch to confirm. It'll primarily affect audio worklets as nobody else gets postMessage().

[padenot]

Yes, it should be false for AudioWorklet.

[hoch]

Confirming @padenot's comment; false for AudioWorklet V1. For the next version of Web Audio API, this is still an open question.

On a related note, I believe folks working AnimationWorklet project are also considering some sorts of communication channel between threads or global scopes. @bfgeek might be able to share his opinion on this.

[padenot]

For the next version of Web Audio API, this is still an open question.

I don't quite understand how we would allow blocking calls from an AudioWorklet, what do you have in mind ?

[hoch]

So I used a broad term: "the next version of Web Audio API". Just to keep the door open for SAB in the future.

[annevk]

To be clear, if we you base postMessage() on the HTML Standard's infrastructure, as you should, SharedArrayBuffer objects are part of the deal. I guess technically you can throw some kind of "not implemented" exception, but technically, it should just work from a standards point of view.

[domenic]

Right, I think @hoch probably meant keeping the door open for Atomics.wait/[[CanBlock]] = true in the future.

[padenot]

There is no way we can ever allow Atomic.wait in an AudioWorklet (this is fundamentally against the programming model used in real-time programming), so I'm still not sure what @hoch meant.

[hoch]

For AudioWorklet case, I agree that it does not make sense much. Are you saying that Web Audio API (the spec in its entirety) should not ever support SAB or Atomic in the future? If you think so, we might have to file an issue and let AudioWG investigate the possibility.

I know some people are seriously interested in using a worker + SAB + audio callback. Should we just tell them that it won't happen in Web Audio API ever?

[domenic]

The thing to understand is that SAB and Atomics.wait are separable. @padenot is saying we should never support Atomics.wait. But that is a totally separate question from supporting SAB. And as @annevk points out, it would require extra spec work to disallow SAB in AudioWorklet.

[padenot]

My position is that we should allow SAB in AudioWorklet (for the reasons explained in WebAudio/web-audio-api#1194 (comment)), but disallow Atomics.wait, for the reasons explained here.

[hoch]

I completely misunderstood the issue here and I am still not sure of this question is about SAB or Atomics. @domenic told me it's about both and I don't think AudioWG had a discussion on these new terms at all.

So I will defer my confirmation until we resolve this in our WG.

[domenic]

I might have confused things a bit, as we meandered through this thread. Let me try to re-summarize.

The agent/agent cluster formalism defines two separable things:

• Who you share memory with
• Whether you can use the Atomics.wait() API.

Atomics.wait() is governed by a boolean, [[CanBlock]]. It seems like everyone is on board with that boolean being false (so you cannot use Atomics.wait()). Thus when people say "false for AudioWorklet v1", since false is a boolean, it is easy to interpret this as being about Atomics.wait(), and unrelated to sharing memory. I'm not sure what was intended though.

The question of who you can share memory with is a separate question that also needs to be resolved. It could be the case that we say for "v1" you cannot share memory with anyone. But it is a pretty easy change to allow sharing memory with the parent window, and it is entirely on the HTML side. It is basically the same issue as #385 / #224.

If you say that worklets and their main thread cannot share memory ("are not in the same agent cluster"), then postMessaging SABs between them will cause messageerror events (instead of the usual message event).

Another thing to note is that even if you don't allow sharing memory between the worklet and main thread, SharedArrayBuffer still "works" inside worklets. You just can't share it with anyone, so it's kind of pointless. SharedArrayBuffer is a JavaScript global like Promise or Map or Math and can't be turned off.

So it seems like we are still not settled on whether worklets can share memory with the main thread. @padenot thinks they should be able to. @hoch would like to discuss in the WG. I think that is where we are now.

[annevk]

In particular, agents define whether you can use Atomics.wait(), agent clusters define who you can share memory with. I had forgotten we could indeed make each worklet its own agent cluster, thereby disabling SharedArrayBuffer objects.

[lars-t-hansen]

Quoth @domenic:

So it seems like we are still not settled on whether worklets can share memory with the main thread. @padenot thinks they should be able to. @hoch would like to discuss in the WG. I think that is where we are now.

I had a discussion with @padenot about this last night and my take on it - so far - is that worklets cannot be in the same agent cluster as the main thread because their lifetimes are insufficiently coordinated. Specifically, my understanding is that the worklet thread is simply destroyed if the document in the main thread, and its worker threads, are pushed into the window history. When that document and its workers are subsequently re-activated, they may be left hanging forever waiting for something (an update to shared memory) that will never come from the worklet, because the worklet no longer exists.

On the other hand: if there is some kind of reliable and predictable infrastructure around that (eg, the main document can listen for some event that said it was deactivated and reactivated, and the liveness of the worklet can be inferred reliably from that event) then my concern in the previous paragraph may not matter. All I care about is that it should be possible to write reliable threaded code, and for that to happen, programs must know whether threads are shut down by the embedding. (But I don't know enough about the details to the web stack and worklets to make a call on that.)

[annevk]

The document and its worker threads may be destroyed too when you navigate (and are in most implementations, I think only Firefox currently allows them to be revived).

[lars-t-hansen]

That seems like a potentially massive source of incompatibility...

[padenot]

On the other hand: if there is some kind of reliable and predictable infrastructure around that (eg, the main document can listen for some event that said it was deactivated and reactivated, and the liveness of the worklet can be inferred reliably from that event) then my concern in the previous paragraph may not matter. All I care about is that it should be possible to write reliable threaded code, and for that to happen, programs must know whether threads are shut down by the embedding. (But I don't know enough about the details to the web stack and worklets to make a call on that.)

This is the case, at least for an AudioWorklet, which belongs to an AudioContext.

An AudioContext has a state, and a state transition callback. If the state is "running", then the audio callbacks are firing, and the worklet code runs. If the AudioContext state is "suspended" or "closed", then the AudioContext's audio thread is not running, so the worklet is not running.

A triplet of methods (suspend, resume, close) allows authors to control this state. Those methods return Promises that resolve when the state transition has been reflected to the main thread, and a "statechange" event is fired to the AudioContext.

When loading up a document (either from history, or going to it from the first time), an AudioContext is initially "suspended", and statechange is fired when it goes to "running", so it's possible to coordinate the main thread and the worklet.

When navigating away from a document, the AudioContext is suspended. When closing the tab, it's closed.

Proper synchronization is easy to do, with this mechanism, so we are in the situation you describe in your second paragraph.

[lars-t-hansen]

OK, given that I don't see any obvious technical problems with allowing the worklet agent to be part of the agent cluster and allowing it to share memory with the other agents in the cluster.

(I'm indifferent to the value of [[CanBlock]] for worklets, but it's certainly safe to start out by setting it to false. The atomic ops will remain available so safe nonblocking algorithms are possible. I'm not yet convinced they will be good, safe nonblocking algorithms since they'll have to busy-wait, but perhaps there's some other mechanism that allows them to suspend and restart.)

[hoch]

After reading through comments again, I believe the conclusion here is to add the worklet agent to be part of the agent cluster. Perhaps someone should work on a PR?

[annevk]

#224 is related as well (and referenced from the HTML Standard, which would need to be updated).

[hoch]

@annevk @padenot @domenic

Okay. If I were to make the necessary spec change, would this be good enough?

• Add Worklet agent to the agent cluster and remove the red note here:
  https://html.spec.whatwg.org/multipage/webappapis.html#integration-with-the-javascript-agent-cluster-formalism

Per the last comment in #224, the Worklet spec already has the bit about the owner document. Please advise if I am missing anything.

[domenic]

What you'd need to do is:

• Define what other agents worklet agents can share memory with, in precise spec language.
• Remove the red box
• (Optional but a good idea) add a worklet bullet point or two to the example box below
• Write tests showing what you can and can't share memory with

[hoch]

Thanks for the pointer, @domenic!

But I am not sure about where the first bullet point is being done in the HTML spec. (except for the list of relationships in the example box) Can you provide an example of such definition? I believe the rest seems straightforward.

[domenic]

A similar-origin window agent, dedicated worker agent, shared worker agent, or service worker agent, agent, can share memory with any dedicated worker agent whose single realm's global object's owner set contains an item whose relevant Realm belongs to agent.

(In imprecise English, this is saying that windows and workers can share memory with dedicated workers that they own.)

[hoch]

I see - I was looking for something more extensive. For Audio Worklet, an AudioWorkletGlobalScope is associated with a BaseAudioContext, which only can be owned by a window agent. (Workers can't instantiate BAC.) That leaves us:

(a) A similar-origin window agent can share memory with any Worklet agent (b) whose single realm's global object's owner set contains an item whose relevant Realm belongs to agent

TBH I am having trouble understanding the part (b), but I believe it roughly matches the concept of "WorkletGlobalScope" (an event loop on a thread).

Also if this is an "imprecise" version of it, how do we make it more "precise"?

[domenic]

That looks precise, and yeah, the (b) part is annoyingly circuitous in the existing text; I'm not sure how to make it better though.

Do WorkletGlobalScopes have owner sets, which contain Document objects? If so then that definition seems perfect. But I kind of thought some of the open issues here were about defining the ownership relationship in the first place.

[hoch]

Do WorkletGlobalScopes have owner sets, which contain Document objects?

I believe so.

Thanks for your advice and I will try to draft a PR for it.

[domenic]

I see they have owner documents. (I guess worklets can only be owned by one document, whereas because of the shared- and service-worker cases, workers can be owned by multiple.) So that should simplify us to

A similar-origin window agent agent can share memory with any Worklet agent whose single realm's global object's owner document's relevant Realm belongs to agent.

Because it is Friday I drew you a picture:

If you squint closely enough you should be able to find:

• agent, the similar-origin Window agent
• The Worklet agent
• Its single realm
• That realm's global object
• That realm's global object's owner document
• That realm's global object's owner document's relevant realm
• The way in which the realm from the previous bullet point belongs to agent

^_^

[hoch]

Glad we're looking at the same spot in the spec. This is a great overview for me; TIL agent, realm and owner document. :)