Clarification needed on ended HTMLMediaElement with MediaStream becoming active

[Pehrsons]

Spec says:

Note that once ended equals true the HTMLMediaElement will not play media even if new MediaStreamTrack's are added to the MediaStream (causing it to return to the active state) unless autoplay is true or the web application restarts the element, e.g., by calling play()

This explains what a user can see and hear, but not how the HTMLMediaElement acts in other aspects.
For instance,

• Should a media element where a live track appears progress readyState and raise events as it does so?
  • If yes, how should the media element handle the stream going inactive again before playback has started?
• Or should all events and state changes be ignored until playback starts through play() or autoplay?
  • This would be different from initial playback of the MediaStream (where readyState advances before play()), as it would, for example, always raise "waiting" on play().

This is related to #453 but is more specifically regarding the state in between ending playback and starting it again.

[jan-ivar]

@Pehrsons Does the way #637 plugs us into the ended playback definition give any insight or clarity on how to proceed here?

[Pehrsons]

Maybe, because mediacapture-main no longer says "set its ended attribute", but leaves this to the media element spec instead. That means the media element spec also dictates when to unset the ended attribute.

It says this about having ended playback (which is when ended returns true):

A media element is said to have ended playback when:
-    The element's readyState attribute is HAVE_METADATA or greater, and

-    Either:
        The current playback position is the end of the media resource, and
        The direction of playback is forwards, and
        The media element does not have a loop attribute specified. 

     Or:
        The current playback position is the earliest possible position, and
        The direction of playback is backwards. 

If we've ended playback and the stream becomes active/audible (video/audio), that might mean that the end of the media resource is again +inf?

Looking at "media resource" the spec says:

A media resource can have multiple audio and video tracks. For the purposes of a media element, the video data of the media resource is only that of the currently selected track (if any) as given by the element's videoTracks attribute when the event loop last reached step 1, and the audio data of the media resource is the result of mixing all the currently enabled tracks (if any) given by the element's audioTracks attribute when the event loop last reached step 1.

So the end of the media resource can only be extended if when the MediaStream becomes active, those new AudioTrack and VideoTrack instances become enabled or selected, respectively.

There are AudioTracks and VideoTracks becoming enabled/selected without the application interacting, in the resource selection algorithm with mode local, specifically in the media processing steps list that are run whenever new data becomes available:

Whenever new data for the current media resource becomes available, queue a task to run the first appropriate steps from the media data processing steps list below.

Note that the entire resource selection algorithm is aborted when the entire media resource is aborted:

When the current media resource is permanently exhausted (e.g. all the bytes of a Blob have been processed), if there were no decoding errors, then the user agent must move on to the final step below. This might never happen, e.g. if the current media resource is a MediaStream.

The final step:

Final step: If the user agent ever reaches this step (which can only happen if the entire resource gets loaded and kept available): abort the overall resource selection algorithm.

I guess the remaining question is whether a MediaStream becoming inactive/inaudible means that it is exhausted? I guess the This might never happen, e.g. if the current media resource is a MediaStream. is the answer, and means that a MediaStream never becomes permanently exhausted. Then this means that an ended media element where its (MediaStream) current media resource becomes active, will stop being ended, advance the readyState, select tracks anew, etc.

Can someone sign off on this making sense? If so I think we can close this.

Though it would probably be useful with a note in mediacapture-main noting this, because it took some digging to get through.

[alvestrand]

@guidou did you have an opinion here?

[guidou]

I agree with the notion that a MediaStream never becomes permanently exhausted and therefore an ended element stops being ended when the MediaStream becomes active.