forked from w3c/webrtc-encoded-transform
-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.bs
403 lines (328 loc) · 20.5 KB
/
index.bs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
<pre class='metadata'>
Title: WebRTC Encoded Transform
Shortname: webrtc-encoded-transform
Level: None
Status: ED
Group: webrtc
Repository: w3c/webrtc-encoded-transform
URL: https://w3c.github.io/webrtc-encoded-transform/
Editor: Harald Alvestrand, Google https://google.com, hta@google.com
Editor: Guido Urdaneta, Google https://google.com, guidou@google.com
Editor: Youenn Fablet, Apple https://www.apple.com, youenn@apple.com
Abstract: This API defines an API surface for manipulating the bits on
Abstract: {{MediaStreamTrack}}s being sent via an {{RTCPeerConnection}}.
Markup Shorthands: css no, markdown yes
</pre>
<pre class=link-defaults>
spec:webidl; type:dfn; text:resolve
</pre>
<pre class=biblio>
{
"WEB-CODECS": {
"href":
"https://github.com/WICG/web-codecs/blob/master/explainer.md",
"title": "Web Codecs explainer"
}
}
</pre>
<pre class=link-defaults>
spec:streams; type:interface; text:ReadableStream
</pre>
# Introduction # {#introduction}
The [[WEBRTC-NV-USE-CASES]] document describes several functions that
can only be achieved by access to media (requirements N20-N22),
including, but not limited to:
* Funny Hats
* Machine Learning
* Virtual Reality Gaming
These use cases further require that processing can be done in worker
threads (requirement N23-N24).
Furthermore, the "trusted JavaScript cloud conferencing" use case
requires such processing to be done on encoded media, not just the raw
media.
This specification gives an interface inspired by [[WEB-CODECS]] to
provide access to such functionality while retaining the setup flow of
RTCPeerConnection.
This iteration of the specification provides access to encoded media,
which is the output of the encoder part of a codec and the input to the
decoder part of a codec.
# Terminology # {#terminology}
# Specification # {#specification}
The Streams definition doesn't use WebIDL much, but the WebRTC spec does.
This specification shows the IDL extensions for WebRTC.
It uses an additional API on {{RTCRtpSender}} and {{RTCRtpReceiver}} to
insert the processing into the pipeline.
<pre class="idl">
// New dictionary
dictionary RTCInsertableStreams {
ReadableStream readable;
WritableStream writable;
};
typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
// New methods for RTCRtpSender and RTCRtpReceiver
partial interface RTCRtpSender {
attribute RTCRtpTransform? transform;
};
partial interface RTCRtpReceiver {
attribute RTCRtpTransform? transform;
};
</pre>
## Extension operation ## {#operation}
At the time when a codec is initialized as part of the encoder, and the
corresponding flag is set in the {{RTCPeerConnection}}'s {{RTCConfiguration}}
argument, ensure that the codec is disabled and produces no output.
### Stream creation ### {#stream-creation}
At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the following steps:
2. Initialize [=this=].`[[transform]]` to null.
3. Initialize [=this=].`[[readable]]` to a new {{ReadableStream}}.
4. <a dfn for="ReadableStream">Set up</a> [=this=].`[[readable]]`. [=this=].`[[readable]]` is provided frames using the [=readEncodedData=] algorithm given |this| as parameter.
5. Set [=this=].`[[readable]]`.`[[owner]]` to |this|.
6. Initialize [=this=].`[[writable]]` to a new {{WritableStream}}.
7. <a dfn for="WritableStream">Set up</a> [=this=].`[[writable]]` with its [=WritableStream/set up/writeAlgorithm=] set to [=writeEncodedData=] given |this| as parameter and its [=WritableStream/set up/sizeAlgorithm=] to an algorithm that returns <code>0</code>.
<p class="note">Chunk size is set to 0 to explictly disable streams backpressure on the write side.</p>
8. Set [=this=].`[[writable]]`.`[[owner]]` to |this|.
9. Initialize [=this=].`[[pipeToController]]` to null.
1. Initialize [=this=].`[[lastReceivedFrameCounter]]` to <code>0</code>.
1. Initialize [=this=].`[[lastEnqueuedFrameCounter]]` to <code>0</code>.
1. [=Queue a task=] to run the following steps:
1. If [=this=].`[[pipeToController]]` is not null, abort these steps.
2. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}.
<!-- FIXME: Use pipeTo algorithm when available. -->
3. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal.
### Stream processing ### {#stream-processing}
The <dfn>readEncodedData</dfn> algorithm is given a |rtcObject| as parameter. It is defined by running the following steps:
1. Wait for a frame to be produced by |rtcObject|'s encoder if it is a {{RTCRtpSender}} or |rtcObject|'s packetizer if it is a {{RTCRtpReceiver}}.
1. Increment |rtcObject|.`[[lastEnqueuedFrameCounter]]` by <code>1</code>.
1. Let |frame| be the newly produced frame.
1. Set |frame|.`[[owner]]` to |rtcObject|.
1. Set |frame|.`[[counter]]` to |rtcObject|.`[[lastEnqueuedFrameCounter]]`.
1. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[readable]]`.
The <dfn>writeEncodedData</dfn> algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps:
1. If |frame|.`[[owner]]` is not equal to |rtcObject|, abort these steps and return [=a promise resolved with=] undefined. A processor cannot create frames, or move frames between streams.
1. If |frame|.`[[counter]]` is equal or smaller than |rtcObject|.`[[lastReceivedFrameCounter]]`, abort these steps and return [=a promise resolved with=] undefined. A processor cannot reorder frames, although it may delay them or drop them.
1. Set |rtcObject|.`[[lastReceivedFrameCounter]]` to |frame|`[[counter]]`.
4. Enqueue the frame for processing as if it came directly from the encoded data source, by running one of the following steps:
* If |rtcObject| is a {{RTCRtpSender}}, enqueue it to |rtcObject|'s packetizer, to be processed [=in parallel=].
* If |rtcObject| is a {{RTCRtpReceiver}}, enqueue it to |rtcObject|'s decoder, to be processed [=in parallel=].
5. Return [=a promise resolved with=] undefined.
On sender side, as part of [=readEncodedData=], frames produced by |rtcObject|'s encoder MUST be enqueued in |rtcObject|.`[[readable]]` in the encoder's output order.
As [=writeEncodedData=] ensures that the transform cannot reorder frames, the encoder's output order is also the order followed by packetizers to generate RTP packets and assign RTP packet sequence numbers.
On receiver side, as part of [=readEncodedData=], frames produced by |rtcObject|'s packetizer MUST be enqueued in |rtcObject|.`[[readable]]` in the same encoder's output order.
To ensure the order is respected, the depacketizer will typically use RTP packet sequence numbers to reorder RTP packets as needed before enqueuing frames in |rtcObject|.`[[readable]]`.
As [=writeEncodedData=] ensures that the transform cannot reorder frames, this will be the order expected by |rtcObject|'s decoder.
## Extension attribute ## {#attribute}
A RTCRtpTransform has two private slots called `[[readable]]` and `[[writable]]`.
The <dfn attribute for="RTCRtpSender,RTCRtpReceiver">transform</dfn> getter steps are:
1. Return [=this=].`[[transform]]`.
The `transform` setter steps are:
2. Let |transform| be the argument to the setter.
3. Let |checkedTransform| set to |transform| if it is not null or to an [=identity transform stream=] otherwise.
3. Let |reader| be the result of [=ReadableStream/getting a reader=] for |checkedTransform|.`[[readable]]`.
4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`.
5. Initialize |newPipeToController| to a new {{AbortController}}.
6. If [=this=].`[[pipeToController]]` is not null, run the following steps:
1. [=AbortSignal/Add=] the [=chain transform algorithm=] to [=this=].`[[pipeToController]]`.signal.
2. [=AbortSignal/signal abort=] [=this=].`[[pipeToController]]`.signal.
7. Else, run the [=chain transform algorithm=] steps.
8. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
9. Set [=this=].`[[transform]]` to |transform|.
The <dfn>chain transform algorithm</dfn> steps are defined as:
1. If |newPipeToController|'s [=AbortSignal/aborted flag=] is true, abort these steps.
2. [=ReadableStreamDefaultReader/Release=] |reader|.
3. [=WritableStreamDefaultWriter/Release=] |writer|.
4. Assert that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`.
<!-- FIXME: Use pipeTo algorithm when available. -->
5. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|.signal.
6. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|.signal.
This algorithm is defined so that transforms can be updated dynamically.
There is no guarantee on which frame will happen the switch from the previous transform to the new transform.
If a web application sets the transform synchronously at creation of the {{RTCRtpSender}} (for instance when calling addTrack), the transform will receive the first frame generated by the {{RTCRtpSender}}'s encoder.
Similarly, if a web application sets the transform synchronously at creation of the {{RTCRtpReceiver}} (for instance when calling addTrack, or at track event handler), the transform will receive the first full frame generated by the {{RTCRtpReceiver}}'s packetizer.
# SFrameTransform # {#sframe}
<p class="note">
The API presented in this section represents a preliminary proposal
based on protocol proposals that have not yet been adopted by an IETF WG.
As a result, both the API and underlying protocol are likely to change
significantly going forward.
</p>
<xmp class="idl">
enum SFrameTransformRole {
"encrypt",
"decrypt"
};
dictionary SFrameTransformOptions {
SFrameTransformRole role = "encrypt";
};
typedef [EnforceRange] unsigned long long SmallCryptoKeyID;
typedef (SmallCryptoKeyID or bigint) CryptoKeyID;
[Exposed=(Window,DedicatedWorker)]
interface SFrameTransform {
constructor(optional SFrameTransformOptions options = {});
Promise<undefined> setEncryptionKey(CryptoKey key, optional CryptoKeyID keyID);
attribute EventHandler onerror;
};
SFrameTransform includes GenericTransformStream;
enum SFrameTransformErrorEventType {
"authentication",
"keyID",
"syntax"
};
[Exposed=(Window,DedicatedWorker)]
interface SFrameTransformErrorEvent : Event {
constructor(DOMString type, SFrameTransformErrorEventInit eventInitDict);
readonly attribute SFrameTransformErrorEventType errorType;
readonly attribute CryptoKeyID? keyID;
readonly attribute any frame;
};
dictionary SFrameTransformErrorEventInit : EventInit {
required SFrameTransformErrorEventType errorType;
required any frame;
CryptoKeyID? keyID;
};
</xmp>
The <dfn constructor for="SFrameTransform" lt="SFrameTransform(options)"><code>new SFrameTransform(<var>options</var>)</code></dfn> constructor steps are:
1. Let |transformAlgorithm| be an algorithm which takes a |frame| as input and runs the <a href="#sframe-transform-algorithm">SFrame transform algorithm</a> with |this| and |frame|.
2. Set |this|.`[[transform]]` to a new {{TransformStream}}.
3. <a dfn for="ReadableStream">Set up</a> [=this=].`[[transform]]` with [=TransformStream/set up/transformAlgorithm=] set to |transformAlgorithm|.
4. Let |options| be the method's first argument.
5. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"].
6. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`.
7. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.
## Algorithm ## {#sframe-transform-algorithm}
The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps:
1. Let |role| be |sframe|.`[[role]]`.
1. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
1. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
1. Let |data| be undefined.
1. If |frame| is a {{BufferSource}}, set |data| to |frame|.
1. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}}
1. If |frame| is a {{RTCEncodedVideoFrame}}, set |data| to |frame|.{{RTCEncodedVideoFrame/data}}
1. If |data| is undefined, abort these steps.
1. Let |buffer| be the result of running the SFrame algorithm with |data| and |role| as parameters. This algorithm is defined by the <a href="https://datatracker.ietf.org/doc/draft-omara-sframe/">SFrame specification</a> and returns an {{ArrayBuffer}}.
1. If the SFrame algorithm exits abruptly with an error, [=queue a task=] to run the following sub steps:
1. If the processing fails on decryption side due to |data| not following the SFrame format, [=fire an event=] named {{SFrameTransform/onerror|error}} at |sframe|,
using the {{SFrameTransformErrorEvent}} interface with its {{SFrameTransformErrorEvent/errorType}} attribute set to {{SFrameTransformErrorEventType/syntax}}
and its {{SFrameTransformErrorEvent/frame}} attribute set to |frame|.
1. If the processing fails on decryption side due to the key identifier parsed in |data| being unknown, [=fire an event=] named {{SFrameTransform/onerror|error}} at |sframe|,
using the {{SFrameTransformErrorEvent}} interface with its {{SFrameTransformErrorEvent/errorType}} attribute set to {{SFrameTransformErrorEventType/keyID}},
its {{SFrameTransformErrorEvent/frame}} attribute set to |frame| and its {{SFrameTransformErrorEvent/keyID}} attribute set to the keyID value parsed in the SFrame header.
1. If the processing fails on decryption side due to validation of the authentication tag, [=fire an event=] named {{SFrameTransform/onerror|error}} at |sframe|,
using the {{SFrameTransformErrorEvent}} interface with its {{SFrameTransformErrorEvent/errorType}} attribute set to {{SFrameTransformErrorEventType/authentication}}
and its {{SFrameTransformErrorEvent/frame}} attribute set to |frame|.
1. Abort these steps.
1. If |frame| is a {{BufferSource}}, set |frame| to |buffer|.
1. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|.
1. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|.
1. [=ReadableStream/Enqueue=] |frame| in |sframe|.`[[transform]]`.
## Methods ## {#sframe-transform-methods}
The <dfn method for="SFrameTransform">setEncryptionKey(|key|, |keyID|)</dfn> method steps are:
1. Let |promise| be [=a new promise=].
2. If |keyID| is a {{bigint}} which cannot be represented as a integer between 0 and 2<sup>64</sup>-1 inclusive, [=reject=] |promise| with a {{RangeError}} exception.
3. Otherwise, [=in parallel=], run the following steps:
1. Set |key| with its optional |keyID| as key material to use for the SFrame transform algorithm, as defined by the <a href="https://datatracker.ietf.org/doc/draft-omara-sframe/">SFrame specification</a>.
2. If setting the key material fails, [=reject=] |promise| with an {{InvalidModificationError}} exception and abort these steps.
3. [=Resolve=] |promise| with undefined.
4. Return |promise|.
# RTCRtpScriptTransform # {#scriptTransform}
<pre class="idl">
// New enum for video frame types. Will eventually re-use the equivalent defined
// by WebCodecs.
enum RTCEncodedVideoFrameType {
"empty",
"key",
"delta",
};
dictionary RTCEncodedVideoFrameMetadata {
long long frameId;
sequence<long long> dependencies;
unsigned short width;
unsigned short height;
long spatialIndex;
long temporalIndex;
long synchronizationSource;
sequence<long> contributingSources;
};
// New interfaces to define encoded video and audio frames. Will eventually
// re-use or extend the equivalent defined in WebCodecs.
[Exposed=(Window,DedicatedWorker)]
interface RTCEncodedVideoFrame {
readonly attribute RTCEncodedVideoFrameType type;
readonly attribute unsigned long long timestamp;
attribute ArrayBuffer data;
RTCEncodedVideoFrameMetadata getMetadata();
};
dictionary RTCEncodedAudioFrameMetadata {
long synchronizationSource;
sequence<long> contributingSources;
};
[Exposed=(Window,DedicatedWorker)]
interface RTCEncodedAudioFrame {
readonly attribute unsigned long long timestamp;
attribute ArrayBuffer data;
RTCEncodedAudioFrameMetadata getMetadata();
};
// New interfaces to expose JavaScript-based transforms.
[Exposed=DedicatedWorker]
interface RTCTransformEvent : Event {
readonly attribute RTCRtpScriptTransformer transformer;
};
partial interface DedicatedWorkerGlobalScope {
attribute EventHandler onrtctransform;
};
[Exposed=DedicatedWorker]
interface RTCRtpScriptTransformer {
readonly attribute ReadableStream readable;
readonly attribute WritableStream writable;
readonly attribute any options;
};
[Exposed=Window]
interface RTCRtpScriptTransform {
constructor(Worker worker, optional any options, optional sequence<object> transfer);
};
</pre>
## Operations ## {#RTCRtpScriptTransform-operations}
The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|)</code></dfn> constructor steps are:
1. Set |t1| to an [=identity transform stream=].
2. Set |t2| to an [=identity transform stream=].
3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
5. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
1. Let |transformerOptions| be the result of [$StructuredDeserialize$](|serializedOptions|, the current Realm).
2. Let |readable| be the result of [$StructuredDeserialize$](|serializedReadable|, the current Realm).
3. Let |writable| be the result of [$StructuredDeserialize$](|serializedWritable|, the current Realm).
4. Let |transformer| be a new {{RTCRtpScriptTransformer}}.
5. Set |transformer|.`[[options]]` to |transformerOptions|.
6. Set |transformer|.`[[readable]]` to |readable|.
7. Set |transformer|.`[[writable]]` to |writable|.
8. Let |event| be the result of [=creating an event=] with {{RTCTransformEvent}}.
9. Set |event|.type attribute to "rtctransform".
10. Set |event|.transformer to |transformer|.
11. Dispatch |event| on |worker|’s global scope.
// FIXME: Describe error handling (worker closing flag true at RTCRtpScriptTransform creation time. And worker being terminated while transform is processing data).
## Attributes ## {#RTCRtpScriptTransformer-attributes}
A RTCRtpScriptTransformer has three private slots called `[[options]]`, `[[readable]]` and `[[writable]]`.
The <dfn attribute for="RTCRtpScriptTransformer">options</dfn> getter steps are:
1. Return [=this=].`[[options]]`.
The <dfn attribute for="RTCRtpScriptTransformer">readable</dfn> getter steps are:
1. Return [=this=].`[[readable]]`.
The <dfn attribute for="RTCRtpScriptTransformer">writable</dfn> getter steps are:
1. Return [=this=].`[[writable]]`.
# Privacy and security considerations # {#privacy}
This API gives Javascript access to the content of media streams. This
is also available from other sources, such as Canvas and WebAudio.
However, streams that are isolated (as specified in
[[WEBRTC-IDENTITY]]) or tainted with another origin, cannot be
accessed using this API, since that would break the isolation rule.
The API will allow access to some aspects of timing information that are
otherwise unavailable, which allows some fingerprinting surface.
The API will give access to encoded media, which means that the JS application
will have full control over what's delivered to internal components like
the packetizer or the decoder. This may require additional care with
auditing how data is handled inside these components.
For instance, packetizers may expect to see data only from trusted encoders,
and may not be audited for reception of data from untrusted sources.
# Examples # {#examples}
See the [explainer document](https://github.com/w3c/webrtc-encoded-transform/blob/master/explainer.md#code-examples).