This is the repository for stream-mediastreamtrack, an experimental API for generating Streams out of a MediaStreamTrack.
You're welcome to contribute! Let's make the Web rock our socks off!
Streams are designed to provide real time streams of data with powerful semantics (e.g. built-in backpressure and queuing) to allow users to build higher-level abstractions.
MediaStreamTracks are opaque handles to Real-Time media being transported in the browser. This media is produced or consumed via Sources and Sinks offered by the platform (see the following diagram); however, if a given functionality is not readily, users's options vary.
The capabilities for user-defined audio processing are quite developed thanks to the WebAudio bridge and its ScriptProcessorNode that essentially enables what this Spec is looking for.
For video, however, unsupported source/sink functionality forces users to resort to contortions such as reflexion on intermediate HTML elements (e.g. <canvas>, see the Workarounds Section) or offline processing (e.g. using MediaRecorder). These approaches, however, lose the timing information, introduce friction in the interoperability between elements and need unnecessary processing steps.
This situation is made only more evident with the arrival of powerful programmable environments such as WebAssembly where users will naturally expect to be able to manipulate Real-Time media.
This API reconciles these two existing ways to access Media on the browser, enhancing platform ergonomics and orthogonality while only attempting to define the minimum amount of new data types (VideoFrame).
Use cases that depend explicitly on timing are enabled, e.g.:
- Measuring the amount of video frames produced; calculating the source frame rate.
- Calculating inter-frame measures, e.g. motion flow, presence/absence, stabilization,
- Adding subtitles to video/audio.
- Manipulating depth data streams.
Whereas use cases that do not depend explicitly on timing are not enabled but are enhanced:
- Producing per-frame analysis and transformations and the endless array of digital image processing algorithms, e.g. edge enhancement or chrome keying.
- Adjusting the presentation timestamp of the media to speed up or slow down video, hence creating a timelapse or slow-motion effect.
Making a MediaStreamTrack both a ReadableStream and a WritableStream (i.e. a TransformStream) would allow for even more sophisticated use cases where users could 'plug' custom elements in MediaStreamTrack-based pipelines, e.g. WebAssembly operations between WebCam capture and Recording, etc.
The most usual hack to access video data is to cast a given MediaStreamTrack onto a <video> element and this in turn onto a <canvas> that is subsequently read back -- <video> elements provide no drawn event so it's up to the user to blit from <video> to <canvas> on a timely basis (see e.g. this article). Moreover, usually reading from <canvas> implies a costly read back from GPU and potential pixel conversions needed.
Chrome Pepper API introduced and supports both MediaStreamVideoTrack and MediaStreamAudioTrack addressing a similar situation as the one described.
As mentioned before, WebAudio ScriptProcessorNode (or its successor) enables similar use cases for audio.
http://rawgit.com/yellowdoge/streams-mediastreamtrack/master/index.html#examples
To compile, run:
curl https://api.csswg.org/bikeshed/ -F file=@index.bs -F force=1 > index.html
if the produced file has a strange size (i.e. zero), then something went terribly wrong; run instead
curl https://api.csswg.org/bikeshed/ -F file=@index.bs -F output=err
and try to figure out why bikeshed did not like the .bs :'(