Show/Hide Subtitle/CC Text Track Events

[cjpillsbury]

Q: What should events to show/hide subtitle/cc text tracks look like?

Assumptions:

• Media Chrome Elements will receive attributes in the form proposed in Text Track Data #59
• Elements will very likely need to parse these back into JS objects resembling a (partial) TextTrack
• It's common to have a single control element for showing/hiding both Subtitles & CC
• While uncommon, there are cases where users/players would want multiple Subtitles/CCs showing simultaneously.
• It's common to have a UI that "disables"/hides any and all currently showing Subtitles & CC

Proposal

Create 2 generic events: one for showing and one for hiding 1+ subtitles/cc text tracks (or both)

Show Text Tracks

• type: "mediashowtexttracksrequest"
• detail (required): [{ kind: TEXT_TRACK_KIND, label?: OPTIONAL_TRACK_LABEL, language: TRACK_LANGUAGE }, ...] || { kind: TEXT_TRACK_KIND, label?: OPTIONAL_TRACK_LABEL, language: TRACK_LANGUAGE }

Result: changes any matching TextTrack to mode="showing" (NOTE: In this proposal, this would not change the mode of any currently "showing" TextTrack)

Examples:

// Example 1
const track = { kind: "captions", language: "en_US", label: "English (US)" };
const evt1 = new CustomEvent("mediashowtexttracksrequest", { composed: true, bubbles: true, detail: track });
this.dispatchEvent(evt1);

// Example 2 (multi-select)
const track1 = { kind: "captions", language: "en_US" }; // Assumes this caption track has no `label`
const track2 = { kind: "subtitles", label: "English (with descriptions)", language: "en_US" };
const evt2 = new CustomEvent("mediashowtexttracksrequest", { composed: true, bubbles: true, detail: [track1, track2] });
this.dispatchEvent(evt2);

Hide Text Tracks

• type: "mediahidetexttracksrequest"
• detail (optional): [{ kind: TEXT_TRACK_KIND, label?: OPTIONAL_TRACK_LABEL, language: TRACK_LANGUAGE }, ...] || { kind: TEXT_TRACK_KIND, label?: OPTIONAL_TRACK_LABEL, language: TRACK_LANGUAGE } || undefined

Result: if detail is present, changes any matching TextTrack to mode="disabled". If detail is not present, changes any TextTrack that currently has mode="showing" to mode="disabled".

Examples:

// Example 1
const track = { kind: "captions", language: "en_US", label: "English (US)" };
const evt1 = new CustomEvent("mediahidetexttracksrequest", { composed: true, bubbles: true, detail: track });
this.dispatchEvent(evt1);

// Example 2 (multi-select)
const track1 = { kind: "captions", language: "en_US" }; // Assumes this caption track has no `label`
const track2 = { kind: "subtitles", label: "English (with descriptions)", language: "en_US" };
const evt2 = new CustomEvent("mediahidetexttracksrequest", { composed: true, bubbles: true, detail: [track1, track2] });
this.dispatchEvent(evt2);

// Example 3 (disable/hide all currently showing)
const evt3 = new CustomEvent("mediahidetexttracksrequest", { composed: true, bubbles: true });
this.dispatchEvent(evt1);

[heff]

Good writeup, thanks!

I'm seeing some devils in the details. In general, I'm trying to think through the implications of multiple UI elements interacting with a track, a track with different potential modes, and also the difference between the track kinds. (While also trying not to overthink this)

As a made up example, what if an external transcript element wants a captions track to be always active (at least hidden, showing ok, not disabled) while the user turns captions on and off (should a request to turn off captions disable them or make them hidden)?
Off the top of my head I think we could put it on the transcript element to re-activate the track, but it makes me want a better real example of using a 'hidden' track.
But I don't know if we can use the term "hide" for disabling a track since a "hidden" track is a real thing that might have a real use case.

How should show/hide apply to chapters, descriptions, and metadata tracks, which don't really have a difference today between hidden and shown AFAIK?

It feels a little inconsistent to be working with independent lists of track kinds at the attr level but then general tracks in events.

How do you feel about these tweaks?

Kind-specific events

mediashowcaptionsrequest
mediashowsubtitlesrequest
mediashowchaptersrequest
mediashowdescriptionsrequest
mediashowmetadatarequest

I like your example of having a hide request event with no detail hide all tracks, and I think it's actually more useful when it's kind-specific.

Four kinds

• show - If visibility is available, visibility is requested. UI elements can interpret what "show" vs. just "enabled" means depending on the use case. For example do chapter markers show up progress bar or not.
• hide - Keep active but request no visibility.
• enable - Request active but not visible. Does not undo visibility.
• disable - Track becomes inactive

I chose enable/disable from language used on MDN. Activate/deactivate was another option.

The potential downside is that tracks would continue to be 'active' after being just 'hidden' and when technically not needed anymore. The question is, is that actually a problem? If so we could encourage people to use show and disable for the primary track menus.

Allow a string as the detail for a request event

const evt1 = new CustomEvent("mediashowcaptionsrequest", {
  composed: true, 
  bubbles: true,
  detail: "en-US:English" // URL encoded label
});

Using an object would still be an option. detail: { language: "en-US", label: "English" }

The combination of lang and label plus kind-specific events/attrs creates a unique ID for a track. So I can see myself just using the track lang:label string as an object key or ID somewhere, and then it being easy just to pass that to the event rather than needing to create an object. It also creates some consistency with the attributes.

Related to this polymorphism, what's a use case where you'd want enable multiple tracks at the same time (in an array)? Does that need go away with kind-specific events?

[cjpillsbury]

Great follow up. I think your general concerns and approach make sense here. A couple of high level considerations:

• Most of the use cases where we're discriminating between how we want to "toggle" between 3 possible states/modes ("disabled", "hidden", "showing") would require something keeping track of previous states (and possibly defaults by kind if we don't have a "previous state"), at least to know what we should toggle to when a caption/subtitle track is "showing". I don't think that's a reason against your proposal, but it probably means we should have a related conversation about what we expect in terms of state management (in e.g. media-controller, or in our out of box media chrome elements for captions/subtitles, or both), and how we expect our "out of box" elements to behave.
• Probably the most common media chrome element use case (and what will be our "out of box" captions/subtitles selection element) will be a component that allows showing exactly one subtitle or caption track. If our events are distinguished by kind, this adds complexity/burden to element implementors. Again, I don't think that's a deal breaker for your proposal, but it's worth considering. We may also solve this by adding a more generic set of e.g. mediashowtexttrackrequest and similar, for which the kind-specific versions are just shorthand.
• The primary reason I thought supporting an array detail for whatever events we settle on is to make "toggle" use cases easier for implementors. For example, "toggling" between show-ing/disable-ing whatever subtitles/captions are "selected". The distinction between "selection" vs. "mode" doesn't exist in the browser world, but is common in players. While a media chrome elements could e.g. dispatch an event forEach track, this feels a bit clumsy, especially since the state changes will have downstream effects. Allowing for a "batching" use case feels like it would be handy (regardless of what particular events we settle on) and wouldn't be especially more complex to implement than the single track case.

[heff]

how we want to "toggle" between 3 possible states

I don't think we should ever just be 'toggling' these states, as in the UI element doesn't care what the current state is and just wants to switch it. But I think what you're saying is we just need to know the state for the tracks, and I see I missed that in my proposal. What I had in mind was something along the lines of:

<media-ui-el media-captions-showing="en-US:English es">

Does that at least address the issue you're describing? Then probably also:

<media-ui-el media-captions-enabled="en-US:English es fr">

If our events are distinguished by kind, this adds complexity/burden to element implementors.

You're right, some extra burden, but even with a mediashowtexttrackrequest you still need to supply the kind, so whether you're doing that in the event name or the detail object, doesn't seem like big difference, unless I'm missing something. If we think this is burden we need to solve I think the answer looks more like creating a unified concept of captions and subtitles, that's usable throughout. Similar to how we handle it in Mux Video, where subtitles is one list, inclusive of captions, which have a closed_captions:true param (i.e. "subtitles for the deaf and hard of hearing (SDH)"). I'd be concerned with straying too far from the underlying browser definition and making 'subtitles' include all here, but I see the upside of a combination considering I usually see them combined in the UI, so it's worth considering.

...especially since the state changes will have downstream effects. Allowing for a "batching" use case feels like it would be handy...

That makes sense to me. I see a correlation between a 'captions off' menu item and consistently using an array, even if only caption lang is selected.

[cjpillsbury]

I don't think we should ever just be 'toggling' these states, as in the UI element doesn't care what the current state is and just wants to switch it...

Let me give a concrete example: I have a select or "flyaway menu" or similar. It has a list of 3 subtitle TextTracks:

1. current mode: "hidden" (say, because it's being used for your transcript scenario)
2. current mode: "disabled"
3. current mode: "showing" (say, because it was flagged as default)

Let's say a user:

1. selects 1
2. selects 2
3. selects 3

What would you expect to happen to the mode of each in each step?

[heff]

Got it.

selects 1

Mode is not "showing", menu-el should know that, menu-el should send a mediashowsubtitlesrequest event.
mode becomes "showing"

selects 2

Mode is not "showing", menu-el should know that, menu-el should send a mediashowsubtitlesrequest event.
mode becomes "showing"

selects 3

Mode is "showing", menu-el should know that, menu-el should - 2 options:

We tell menu-el author to send a mediahidesubtitlesrequest event.
If we aren't concerned about the performance of captions continuing to be 'active' after no longer needed by the user, this is the more ecosystem-friendly option.

We tell menu-el author to send a mediadisablesubtitlesrequest event.
If we are concerned with unused track performance, then this is the more performance-friendly option.
At the same time, we would need to tell other UI-el authors that if they need a track.

To be clear, I'm very not interested in trying to manage internal state of UI element track requests in media-controller. I'm also not confident we'll actually run into any real issues on this front either way we go.

Without more info on performance my assumption is it's not an issue and we should start with the the first option as guidance for el authors, mediahidesubtitlesrequest.

[dylanjha]

Catching up here, in the example <media-ui-el media-captions-showing="en-US:English es">, this element is a fictional, possibly future element of an external transcript element, correct?

Allowing for a "batching" use case feels like it would be handy (regardless of what particular events we settle on) and wouldn't be especially more complex to implement than the single track case.

Agree with this 👍🏼

We tell menu-el author to send a mediahidesubtitlesrequest event.
If we aren't concerned about the performance of captions continuing to be 'active' after no longer needed by the user, this is the more ecosystem-friendly option.

My S1 is that this feels better because it avoids the downstream side effects that come with disabling a text track.

I don't have a strong opinion on Kind-specific events (mediashowcaptionsrequest / mediashowsubtitlesrequest) vs. the general event (mediashowtexttrackrequest). My S1 is what Christian pointed out:

the most common media chrome element use case (and what will be our "out of box" captions/subtitles selection element) will be a component that allows showing exactly one subtitle or caption track. If our events are distinguished by kind, this adds complexity/burden to element implementors.

On the side of the UI-element author who creates a single menu containing both captions and subtitles tracks, psuedo-code would look like:

function showTextTrack (track) {
    let eventName
   if (track.kind === 'captions') {
     eventName = 'mediashowcaptionsrequest'
   }
   if (track.kind === 'subtitles') {
     eventName = 'mediashowsubtitlesrequest'
   }
  const evt1 = new CustomEvent(eventName, {
    composed: true, 
    bubbles: true,
    detail: encodeURIComponent(`${track.language}:${track.label}:`)
  });
}

The little bit of extra overhead comes with that conditional to figure out the right event name, not a deal breaker, and Christian mentioned another way we can work around that.

[cjpillsbury]

We tell menu-el author to send a mediahidesubtitlesrequest event.
If we aren't concerned about the performance of captions continuing to be 'active' after no longer needed by the user, this is the more ecosystem-friendly option.

My S1 is that this feels better because it avoids the downstream side effects that come with disabling a text track.

So after discussion we decided to go with the mediadisablesubtitlesrequest/mediadisablecaptionsrequest for the "default behavior", for a few reasons:

• Per @heff 's callout, there will be some performance hit (though likely negligible from a "big picture" perspective
• network/data usage - for any cases where a dev isn't simply adding a <track/> element for their TextTrack (e.g. text tracks defined via media groups for captions/subtitles in an HLS primary playlist), setting the tracks as "hidden" would entail fetching the data (and parsing, etc.) for each subtitle/cc. This can include tracks in e.g. fmp4/ts containers, adding even more network/data usage
• default initial track states - by default a subtitle/cc track would actually be "disabled" and not "hidden", so this would be the most likely "non-showing" state for baseline text track usage
• surprise events - for users who are monitoring events from text tracks, they may be surprised to get e.g. cuechange events from tracks they weren't intentionally having as "hidden". If we used "hidden", the number of tracks that would dispatch cuechange events would change over time simply based on a user interacting with the subtitles/cc select/flyaway menu

On the side of the UI-element author who creates a single menu containing both captions and subtitles tracks, psuedo-code would look like:

function showTextTrack (track) {
    let eventName
   if (track.kind === 'captions') {
     eventName = 'mediashowcaptionsrequest'
   }
   if (track.kind === 'subtitles') {
     eventName = 'mediashowsubtitlesrequest'
   }
  const evt1 = new CustomEvent(eventName, {
    composed: true, 
    bubbles: true,
    detail: encodeURIComponent(`${track.language}:${track.label}:`)
  });
}

The little bit of extra overhead comes with that conditional to figure out the right event name, not a deal breaker, and Christian mentioned another way we can work around that.

Yup. Also, for our "built in" CC/subtitle track selector, any time we choose to "show" one track, we also need to make sure we "hide" (actually disable, per ☝️) all currently showing subtitle/cc tracks. This isn't especially more complex than your example, but it does mean we may be dispatching up to 3 events to make a transition. Not a show stopper, but worth calling out I think.