Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix #1933: Use FrozenArray for AudioWorkletProcessor process() #2250

Merged
merged 21 commits into from
Nov 19, 2020
Merged

Fix #1933: Use FrozenArray for AudioWorkletProcessor process() #2250

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
607eb07
Fix #1933: Use FrozeArray for AudioWorkletProcessor process()
rtoy Sep 22, 2020
de45916
Address review comments from karlt.
rtoy Sep 25, 2020
0366317
Replace process() method with callback sections
rtoy Sep 25, 2020
097a0ed
Update expected errors
rtoy Sep 25, 2020
f14cc12
Address some review comments
rtoy Oct 1, 2020
e90a8ab
Describe what happens to inputs/outputs if things change
rtoy Oct 1, 2020
cc2bef4
Address some review comments
rtoy Oct 2, 2020
b2fdc7d
Update README.md
svgeesus Sep 29, 2020
c0af014
Fix typo in "messsage" (#2255)
rtoy Oct 2, 2020
2b6dc31
chore: Fix markdown for bold (#2258)
Korilakkuma Oct 5, 2020
443e3d5
Update implementation-report.html
svgeesus Oct 8, 2020
2c4f0c2
Update implementation-report.html
svgeesus Oct 8, 2020
0af14fe
Update implementation-report.html
svgeesus Oct 8, 2020
df661ef
Update implementation-report.html
svgeesus Oct 8, 2020
c7cac46
Fix typo "atleast" (#2259)
rtoy Oct 9, 2020
f019a91
Address #2185: Fix duplicated IDs for decode callbacks (#2253)
rtoy Oct 12, 2020
3ed4110
Fix #2262: Clone bikeshed and install it (#2263)
rtoy Oct 13, 2020
c6030e2
Fix #2257: Make channel ordering table visible in dark mode (#2264)
rtoy Oct 14, 2020
b2c9791
Merge branch 'main' into 1933-awn-frozen-arrays
rtoy Nov 2, 2020
6135447
Add ECMAScript algorithm to freeze parameters.
rtoy Nov 3, 2020
ae984bd
Some cleanups.
rtoy Nov 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 0 additions & 2 deletions expected-errs.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,4 @@ LINE: Can't find the 'destinationNode' argument of method 'AudioNode/disconnect(
LINE: Can't find the 'destinationNode' argument of method 'AudioNode/disconnect(destinationParam, output)' in the argumentdef block.
LINE: Can't find the 'destinationNode' argument of method 'AudioNode/disconnect(destinationParam, output)' in the argumentdef block.
LINE: Can't find the 'input' argument of method 'AudioNode/disconnect(destinationParam, output)' in the argumentdef block.
LINE: Can't find method '['AudioWorkletProcessor/process(inputs, outputs, parameters))']'.
LINE: No 'idl' refs found for 'process(inputs, outputs, parameters))'.
✔ Successfully generated, but fatal errors were suppressed
183 changes: 103 additions & 80 deletions index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@ Markup Shorthands: markdown on, dfn on, css off
spec: ECMAScript; url: https://tc39.github.io/ecma262/#sec-data-blocks; type: dfn; text: data block;
url: https://www.w3.org/TR/mediacapture-streams/#dom-mediadevices-getusermedia; type: method; for: MediaDevices; text: getUserMedia()
</pre>

<!-- We want {{object}} to go to the WebIDL object -->
<pre class=link-defaults>
spec:webidl; type:interface; text:object
</pre>
<link rel="preload" href="https://www.w3.org/scripts/MathJax/2.7.5/MathJax.js?config=TeX-AMS-MML_HTMLorMML" as="script">
<link rel="preload" href="https://www.w3.org/scripts/MathJax/2.7.5/jax/output/HTML-CSS/jax.js?rev=2.7.5" as="script">
<link rel="preload" href="https://www.w3.org/scripts/MathJax/2.7.5/jax/output/HTML-CSS/fonts/STIX/fontdata.js?rev=2.7.5" as="script">
Expand Down Expand Up @@ -10372,13 +10375,18 @@ and the definition of the class manifests the actual audio processing.
Note that the an {{AudioWorkletProcessor}} construction can only happen as a
result of an {{AudioWorkletNode}} contruction.

<pre class="idl">
<xmp class="idl">
[Exposed=AudioWorklet]
interface AudioWorkletProcessor {
constructor ();
readonly attribute MessagePort port;
};
</pre>

callback AudioWorkletProcessCallback =
boolean (FrozenArray<FrozenArray<Float32Array>> inputs,
FrozenArray<FrozenArray<Float32Array>> outputs,
object parameters);
</xmp>

{{AudioWorkletProcessor}} has two internal slots:

Expand All @@ -10389,7 +10397,7 @@ interface AudioWorkletProcessor {

: <dfn>[[callable process]]</dfn>
::
A boolean flag representing whether {{AudioWorkletProcessor/process()}} is
A boolean flag representing whether [=process()=] is
a valid function that can be invoked.
</dl>

Expand Down Expand Up @@ -10470,86 +10478,57 @@ Constructors</h5>
resources to be [[html#ports-and-garbage-collection|collected]].
</dl>

<h5 id="AudioWorkletProcessor-methods">
Methods</h5>
<h5 id="callback-audioworketprocess-callback">
Callback {{AudioWorkletProcessCallback}}</h5>

Users can define a custom audio processor by extending
{{AudioWorkletProcessor}}. The subclass MUST define a method
named {{process()}} that implements the audio processing
{{AudioWorkletProcessor}}. The subclass MUST define an {{AudioWorkletProcessCallback}}
named <code><dfn>process()</dfn></code> that implements the audio processing
algorithm and may have a static property named
<code><dfn>parameterDescriptors</dfn></code> which is an iterable
of {{AudioParamDescriptor}}s.

<dl dfn-type=method dfn-for="AudioWorkletProcessor">
: <dfn>process(inputs, outputs, parameters)</dfn>
::
Implements the audio processing algorithm for the
{{AudioWorkletProcessor}}.
The [=process()=] callback function is handled as specified when <a href="#rendering-a-graph">rendering a graph</a>.

The {{process()}} method is called
synchronously by the audio <a>rendering thread</a> at
every <a>render quantum</a>, if {{AudioWorkletNode}} is
<div class="note">
The return value of this callback controls the lifetime
of the {{AudioWorkletProcessor}}'s associated
{{AudioWorkletNode}}.

This lifetime policy can support a variety of approaches
found in built-in nodes, including the following:

* Nodes that transform their inputs, and are active only
while connected inputs and/or script references exist. Such
nodes SHOULD return <code>false</code> from
[=process()=] which allows the presence or absence of
connected inputs to determine whether the {{AudioWorkletNode}} is
[=actively processing=].

<div class="note">
The return value of this method controls the lifetime
of the {{AudioWorkletProcessor}}'s associated
{{AudioWorkletNode}}.

This lifetime policy can support a variety of approaches
found in built-in nodes, including the following:

* Nodes that transform their inputs, and are active only
while connected inputs and/or script references exist. Such
nodes SHOULD return <code>false</code> from
{{process()}} which allows the presence or absence of
connected inputs to determine whether the {{AudioWorkletNode}} is
[=actively processing=].

* Nodes that transform their inputs, but which remain active
for a <a>tail-time</a> after their inputs are disconnected. In
this case, {{process()}} SHOULD return
`true` for some period of time after
<code>inputs</code> is found to contain zero channels. The
current time may be obtained from the global scope's
{{AudioWorkletGlobalScope/currentTime}} to
measure the start and end of this tail-time interval, or the
interval could be calculated dynamically depending on the
processor's internal state.

* Nodes that act as sources of output, typically with a
lifetime. Such nodes SHOULD return `true` from
{{process()}} until the point at which they are no
longer producing an output.

Note that the preceding definition implies that when no
return value is provided from an implementation of
{{process()}}, the effect is identical to returning
<code>false</code> (since the effective return value is the falsy
value {{undefined}}). This is a reasonable behavior for
any {{AudioWorkletProcessor}} that is active only when it has
active inputs.
</div>

<pre class=argumentdef for="AudioWorkletProcessor/process(inputs, outputs, parameters))">
inputs:
The input audio buffer from the incoming connections provided by the user agent. It has type <code>sequence&lt;sequence&lt;Float32Array>></code>.<code class="nobreak">inputs[n][m]</code> is a {{Float32Array}} of audio samples for the \(m\)th channel of the \(n\)th input. While the number of inputs is fixed at construction, the number of channels can be changed dynamically based on [=computedNumberOfChannels=].

If there are no [=actively processing=] {{AudioNode}}s connected to the \(n\)th input of the {{AudioWorkletNode}} for the current render quantum, then the content of <code>inputs[n]</code> is an empty array, indicating that zero channels of input are available. This is the only circumstance under which the number of elements of <code>inputs[n]</code> can be zero.

outputs:
The output audio buffer that is to be consumed by the user agent. It has type <code>sequence&lt;sequence&lt;Float32Array>></code>.<code class="nobreak">outputs[n][m]</code> is a {{Float32Array}} object containing the audio samples for \(m\)th channel of \(n\)th output. Each of the {{Float32Array}}s are zero-filled. The number of channels in the output will match [=computedNumberOfChannels=] only when the node has a single output.

parameters:
An [=ordered map=] of <var>name</var> → <var>parameterValues</var>. <code>parameters["<var>name</var>"]</code> returns <var>parameterValues</var>, which is a {{Float32Array}} with the automation values of the <var>name</var> {{AudioParam}}.

For each array, the array contains the [=computedValue=] of the parameter for all frames in the [=render quantum=]. However, if no automation is scheduled during this render quantum, the array MAY have length 1 with the array element being the constant value of the {{AudioParam}} for the [=render quantum=].
</pre>

<div>
<em>Return type:</em> {{boolean}}
</div>
</dl>
* Nodes that transform their inputs, but which remain active
for a <a>tail-time</a> after their inputs are disconnected. In
this case, [=process()=] SHOULD return
`true` for some period of time after
<code>inputs</code> is found to contain zero channels. The
current time may be obtained from the global scope's
{{AudioWorkletGlobalScope/currentTime}} to
measure the start and end of this tail-time interval, or the
interval could be calculated dynamically depending on the
processor's internal state.

* Nodes that act as sources of output, typically with a
lifetime. Such nodes SHOULD return `true` from
[=process()=] until the point at which they are no
longer producing an output.

Note that the preceding definition implies that when no
return value is provided from an implementation of
[=process()=], the effect is identical to returning
<code>false</code> (since the effective return value is the falsy
value {{undefined}}). This is a reasonable behavior for
any {{AudioWorkletProcessor}} that is active only when it has
active inputs.
</div>

The example below shows how {{AudioParam}} can be defined and used in an
{{AudioWorkletProcessor}}.
Expand Down Expand Up @@ -10584,6 +10563,51 @@ class MyProcessor extends AudioWorkletProcessor {
}
</xmp>

<h6 id="audioworkletprocess-callback-parameters" callback lt="AudioWorkletProcessCallback()">
Callback {{AudioWorkletProcessCallback}} Parameters
</h6>
The following describes the parameters to the {{AudioWorkletProcessCallback}} function.

In general, the {{AudioWorkletProcessCallback/inputs!!argument}} and
{{AudioWorkletProcessCallback/outputs!!argument}} arrays will be reused
between calls so that no memory allocation is done. However, if the
topology changes, because, say, the number of channels in the input or the
output changes, new arrays are reallocated. New arrays are also
reallocated if any part of the
{{AudioWorkletProcessCallback/inputs!!argument}} or
{{AudioWorkletProcessCallback/outputs!!argument}} arrays are
transferred.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't include parameters object. We can (and Chrome does) freeze the object and reallocate if the topology changes.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/freeze
https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/modules/webaudio/audio_worklet_processor.cc;l=460

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, the comments in #1933 says we'd have to do much more work than just saying "Freeze" because there's no WebIDL concept of freeze other than FrozenArrays.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. It's less ideal, but we can directly refer ECMAScript algorithm:
https://tc39.es/ecma262/#sec-object.freeze

Or start from WebIDL's FrozenArray's algorithm:
https://heycam.github.io/webidl/#es-frozen-array


<dl dfn-type=argument dfn-for="AudioWorkletProcessCallback">
: {{AudioWorkletProcessCallback/inputs!!argument}}, of type <code>{{FrozenArray}}&lt;{{FrozenArray}}&lt;{{Float32Array}}&gt;&gt;</code>
:: The input audio buffer from the incoming connections provided by the user agent. <code class="nobreak">inputs[n][m]</code> is a {{Float32Array}} of audio samples for the \(m\)th channel of the \(n\)th input. While the number of inputs is fixed at construction, the number of channels can be changed dynamically based on [=computedNumberOfChannels=].

If there are no [=actively processing=] {{AudioNode}}s connected to the \(n\)th input of the {{AudioWorkletNode}} for the current render quantum, then the content of <code>inputs[n]</code> is an empty array, indicating that zero channels of input are available. This is the only circumstance under which the number of elements of <code>inputs[n]</code> can be zero.

: {{AudioWorkletProcessCallback/outputs!!argument}}, of type <code>{{FrozenArray}}&lt;{{FrozenArray}}&lt;{{Float32Array}}>></code>
:: The output audio buffer that is to be consumed by the user agent. <code class="nobreak">outputs[n][m]</code> is a {{Float32Array}} object containing the audio samples for \(m\)th channel of \(n\)th output. Each of the {{Float32Array}}s are zero-filled. The number of channels in the output will match [=computedNumberOfChannels=] only when the node has a single output.

: {{AudioWorkletProcessCallback/parameters!!argument}}, of type {{object}}
:: An [=ordered map=] of <var>name</var> → <var>parameterValues</var>. <code>parameters["<var>name</var>"]</code> returns <var>parameterValues</var>, which is a {{FrozenArray}}&lt;{{Float32Array}}> with the automation values of the <var>name</var> {{AudioParam}}.

For each array, the array contains the [=computedValue=] of the parameter for all frames in the [=render quantum=]. However, if no automation is scheduled during this render quantum, the array MAY have length 1 with the array element being the constant value of the {{AudioParam}} for the [=render quantum=].

This object is frozen according the the following steps
<div algorithm="freeze parameter object">
1. Let |parameter| be the [=ordered map=] of the name and parameter values.
1. <a href="http://www.ecma-international.org/ecma-262/#sec-setintegritylevel">SetIntegrityLevel</a>(|parameter|, frozen)
</div>

This frozen [=ordered map=] computed in the algorithm is passed to the
{{AudioWorkletProcessCallback/parameters!!argument}}
argument.

Note: This means the object cannot be modified and
hence the same object can be used for successive calls
unless length of an array changes.

</dl>

<h5 dictionary lt="AudioParamDescriptor">
{{AudioParamDescriptor}}</h5>

Expand Down Expand Up @@ -11233,16 +11257,15 @@ operation such as resolution of {{Promise}}s in the {{AudioWorkletGlobalScope}}.
argument of [=input buffer=], output buffer and
[=input AudioParam buffer=]. A buffer containing copies of the
elements of the {{Float32Array}}s passed via the
<a href="#dom-audioworkletprocessor-process-inputs-outputs-parameters-outputs">
outputs</a> parameter to <var>processFunction</var> is
{{AudioWorkletProcessCallback/outputs!!argument}} parameter to <var>processFunction</var> is
<a href="#available-for-reading">made available for reading</a>.

1. At the conclusion of <var>processFunction</var>,
<a href="https://tc39.github.io/ecma262/#sec-toboolean"><code>ToBoolean</code></a>
is applied to the return value and the result is
assigned to the associated {{AudioWorkletProcessor}}'s
<a>active source</a> flag. This in turn affects whether
subsequent invocations of {{process()}} occur, and has
subsequent invocations of [=process()=] occur, and has
an impact on the lifetime of the node.

1. Else if {{[[callable process]]}} is `false`,
Expand All @@ -11263,7 +11286,7 @@ operation such as resolution of {{Promise}}s in the {{AudioWorkletGlobalScope}}.
5. If this {{AudioNode}} is a <a>destination node</a>,
[=Recording the input|record the input=] of this {{AudioNode}}.

6. Else, <a>process</a> the <a>input buffer</a>, and
6. Else, [=processing-input-buffer|process=] the <a>input buffer</a>, and
[=Making a buffer available for reading|make available for reading=] the
resulting buffer.

Expand Down Expand Up @@ -11301,7 +11324,7 @@ usage.
running the algorithm for this {{AudioNode}} to produce 128
sample-frames.

<dfn lt="process">Processing an input buffer</dfn> means
<dfn lt="processing-input-buffer">Processing an input buffer</dfn> means
running the algorithm for an {{AudioNode}}, using an <a>input
buffer</a> and the value(s) of the {{AudioParam}}(s) of this
{{AudioNode}} as the input for this algorithm.
Expand Down