Why is start 5.3 in the specification?

[guest271314]

start(optional unsigned long timeslice)

3. If at any point, a track is added to or removed from the stream's track set, the UA MUST immediately stop gathering data, discard any data that it has gathered, and queue a task, using the DOM manipulation task source, that runs the following steps:

   1. Set state to inactive.
   2. Fire an error event named InvalidModificationError at target.
   3. Fire a blob event named dataavailable at target with blob.
   4. Fire an event named stop at target.

Why is the language of start() 5.3 (still) in the specification?

• Why was recording multiple tracks potentially of the same kind either not originally pursued to be specified, or attempted and not successful (technically)?

• Have we not now advanced to the point where multiple tracks can be recorded?

[alvestrand]

Recording multiple tracks was intended to be possible from day one. Many formats handle multiple tracks.

When it was pointed out that a lot of container formats couldn't handle increasing the number of channels mid-recording, we were left with two choices:

• Make the behavior dependent on container format (unpalatable)
• Make the behavior consistent, but not very useful (ie stop).
  The WG chose the latter.

I am not aware of any change in the landscape of container formats that seems to indicate that varying the number of tracks is a generally available option. If you know of such changes, please provide references.

As for the "replace track" option - I don't think anyone thought about that possibility at the time.

[guest271314]

@alvestrand

Recording multiple tracks was intended to be possible from day one. Many formats handle multiple tracks.

That intention should not be abandoned. Or state clearly to abandon all hope of multiple tracks

https://searchfox.org/mozilla-central/source/dom/media/MediaRecorder.cpp#778

      // When MediaRecorder supports multiple tracks, we should set up a single
      // MediaInputPort from the input stream, and let main thread check
      // track principals async later.

When it was pointed out that a lot of container formats couldn't handle increasing the number of channels mid-recording, we were left with two choices:

• Make the behavior dependent on container format (unpalatable)
• Make the behavior consistent, but not very useful (ie stop).
  The WG chose the latter.

From a novice front-end perspective (within the context of the use case of not stopping MediaRecorder when src of <video> changes), with no experience coding in C or C++, reading the source code appears to allow for changing the source of the principal media resource using array indexes (MediaStream.selectTrack()). Does the functionality need to be more complex than that?

I am not aware of any change in the landscape of container formats that seems to indicate that varying the number of tracks is a generally available option. If you know of such changes, please provide references.

Again, from a novice front-end perspective with little experiece reading (https://gist.github.com/guest271314/f942fb9febd9c197bd3824973794ba3e ; https://gist.github.com/guest271314/17d62bf74a97d3f111aa25605a9cd1ca) and no experience writing media containers, given that the decoder/encoder/writer must write in sequential, linear order, it should not make any difference what the input media is. Each media container (and codec) that MediaRecorder supports must write the data; e.g., <timecode>0.026</timecode> to output from 0 to N in sequential order, no matter whether the media track changes or the input decoded <timecode>0.026</timecode>, if applicable, is not sequential with reference to the data being recorded and written?

[guest271314]

@alvestrand For clarity, the use case is not necessarily to record and write multiple individual tracks, but to write a single video track, which is already possible for audio using .connect(createMediaStreamSource(MediaStream([audioTrack]))) or .connect(createMediaStreamTrackSource(audioTrack)). The same functionality should be possible for video tracks, without having to use requestAnimationFrame, Web Animation API. ReadableStream, etc.

Re

I am not aware of any change in the landscape of container formats that seems to indicate that varying the number of tracks is a generally available option. If you know of such changes, please provide references.

Is the concern variable video width, height, and audio channel and playback rate, etc.?

What exactly do you mean by "varying the number of tracks"?

From a cursory point of view one of the WebM VOD patterns http://wiki.webmproject.org/adaptive-streaming/webm-vod-baseline-format should be able to achieve the requirement?

Have been considering composing a very basic media container format (not necessarily with compression as the model, but creation in modern FOSS browsers using only the API's shipped with FOSS browsers), where for every 1 second N (e.g., 25; 30; 60) images and 1 or more audio are in 1 "packet"; the images can have any width or height; any inidividual images "packet" playback rate can be variable in relation to the adjacent "packet" or "media chunk"; each audio can have any playback rate; the individual media segments can be re-arranged in any given order with currentTime being reflected in output relevant to the adjacent media chunks, if any, while a note about the original source segment slice currentTime can also being included in the data; either can be rendered while not breaking the arbitrary playback sequence, in essence having the possiblity of multiple currentTime and or duration; all created using Web Audio API, Media Recorder API (as a pipe to create media chunks and files), Media Stream API, capable of being "streamed", as a file, i.e., video/x-browser-created-media (.browsercreatedmedia) over HTTP; from the browser, to the browser, using various means; a simple array of arrays [audio, video] with the ability to add text, etc.

Ultimately trying to do something like

$ mkvmerge -w -o int_all.webm int.webm + int1.webm

for static files and/or live "media streams".

Not a novel concept.

Would be unnecessary if the media encoder, media decoder, webm writer and web media player internal code was exposed as API's, for users to decode, encode and write their own media, without the boundaries of specifications or implementation of the day.

FWIW, the WebRTC and Media Capture and Streams API's are ingenious:+1:

[guest271314]

@alvestrand

I am not aware of any change in the landscape of container formats that seems to indicate that varying the number of tracks is a generally available option. If you know of such changes, please provide references.

If reading the below links correctly apparently Matroska supports multiple tracks in the same file

See Select Video Stream from multi-video MKV https://forum.kodi.tv/showthread.php?tid=152756&pid=2567118#pid2567118 at https://forum.kodi.tv/showthread.php?tid=152756&pid=2567118#pid2567118 (implementation FernetMenta/xbmc@d05b884)

This allows a single video file to have multiple video tracks where supported by the file format e.g. mkv

followed from Multi-Angle / Multi-Video Track MKV Support and Recommendations https://emby.media/community/index.php?/topic/46826-multi-angle-multi-video-track-mkv-support-and-recommendations/

[dontcallmedom]

The original question ("why") has been answered, closing this