Add TextTrackCue ctor & cue + cuebackground attributes #9771

marcoscaceres · 2023-09-21T19:22:21Z

Native UA rendering of out-of-band subtitles and captions is currently only possible with WebVTT. If a site can't or won't make its subtitles available in-band, and can't or won't publish its subtitles and captions in WebVTT, the site must render its subtitles and captions itself.

This pull request aim to:

Allow video publishers to continue storing captions in legacy or custom formats.
Make sure users can control the presence and style of captions with system accessibility settings.
Allow captions to be served in custom formats, but still be rendered by the browser engine.

This pull requests makes the following changes to HTML:

Exposes a constructor for TextTrackCue which takes startTime, endTime, and cueNode arguments.
Defines authoring and processing requirements for cue fragments
Adds cue and cuebackground global attributes.
Add infrastructure for selecting ::cue and ::cue-background pseudo elements, which can then be styles by the CSS or the system.

This pull request is based on:
https://github.com/WebKit/explainers/tree/main/texttracks

At least two implementers are interested (and none opposed):
- WebKit
- …
Tests are written and can be reviewed and commented upon at:
- Included with WebKit PR
- Sent to WPT Tests for TextTrackCue ctor and cue + cuebackground global attrs web-platform-tests/wpt#42243
Implementation bugs are filed:
- Chromium: https://bugs.chromium.org/p/chromium/issues/detail?id=1486858
- Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1855236
- WebKit: https://bugs.webkit.org/show_bug.cgi?id=262053
- Deno (only for timers, structured clone, base64 utils, channel messaging, module resolution, web workers, and web storage): …
- Node.js (only for timers, structured clone, base64 utils, channel messaging, and module resolution): …
MDN issue is filed: …

(See WHATWG Working Mode: Changes for more details.)

/dom.html ( diff )
/index.html ( diff )
/indices.html ( diff )
/infrastructure.html ( diff )
/media.html ( diff )

annevk

I realize this is still a draft, but I thought I'd give some minor editorial feedback that might help with other changes.

source

marcoscaceres · 2023-09-27T14:15:10Z

@zcorpan, Anne suggested I ping you for guidance and review... the proposal doesn't currently talk much about the rendering side of things, so could use some guidance there. I'll take a look at how some other elements define things, but any pointers would be great.

nigelmegitt · 2023-09-27T14:20:53Z

A thought about rendering from the requirements perspective: typically a TextTrackCue would begin rendering at its begin time, but the target behaviour should be to complete rendering as close as possible to the begin time. If this proposal can be worded to give the UA some flexibility to pre-load the HTML fragment and begin rendering ahead of the begin time so that this goal can be achieved, that would be another accessibility benefit of this proposal.

Same applies for clearing the HTML fragment from the DOM and having the presentation update as closely as possible to the TextTrackCue end time, obviously.

nigelmegitt · 2023-09-27T16:17:16Z

source

+  <p>When the <code data-x="attr-cuebackground">cuebackground</code> attribute is present, the element serves as cue background
+  for a <span>text track cue</span>.</p>
+
+  <p>The <code data-x="attr-cue">cue</code> and <code data-x="attr-cuebackground">cuebackground</code> attributes must each be present once.</p>


Does this mean at least once per fragment? Otherwise it is overly restrictive, since a cue fragment may reasonably contain multiple descendant elements that each should be styled as a 'cue' or 'cue background'.

Suggested change

The <code data-x="attr-cue">cue</code> and <code data-x="attr-cuebackground">cuebackground</code> attributes must each be present once.

The <code data-x="attr-cue">cue</code> and <code data-x="attr-cuebackground">cuebackground</code> attributes must each be present on at least one element in the <var>cueFragment</var>.

Right, this was intended more user agent requirement (each must appear once... but that's enforce the processing algorithm). I'll rephrase it so it's more clear as per your intent (and yes, can definitely have multiple).

nigelmegitt · 2023-09-27T16:24:01Z

source

+  <p>When the <code data-x="attr-cue">cue</code> attribute is present, the element serves as a <span>text track
+  cue</span>.</p>


Isn't this circular, since the element only serves as a text track cue if it is attached to a TextTrackCue object? Adding a cue attribute to some DOM element doesn't inherently make it a text track cue without constructing a TextTrackCue.

Is there an additional concept needed here, which is to identify somehow the payload text element of a text track cue, i.e. that which should be styled using the ::cue pseudo-element?

Maybe something conceptually like:

Suggested change

When the <code data-x="attr-cue">cue</code> attribute is present, the element serves as a text track

cue.

When the <code data-x="attr-cue">cue</code> attribute is present, the element can be styled as the text content of a text track

cue using the ::cue text track cue

pseudo-element.

In the explainer we had an abstract concept a "cue fragment", so might bring that in... something like:

A cue fragment is a DocumentFragment containing cue text, represented in a limited subset of HTML and styled with CSS with ::cue and ::cue-background.

https://bugs.webkit.org/show_bug.cgi?id=262053 rdar://116002871 Reviewed by Eric Carlson. Implements spec changes suggested at: whatwg/html#9771 Specifically: * supports `b` and `i` elements, to match WebVTT * drops support for allowing `img` in the cue fragment * implements more robust error checking to prevent malformed ::cue and :cuebackground hierarchies * As such, ::tagPseudoObjects() can now throw * Switch from InvalidStateError to InvalidNodeTypeError for when required attributes are missing * implements cuebackground and cue global attributes * LayoutTests/imported/w3c/web-platform-tests/custom-elements/reactions/HTMLElement-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/custom-elements/reactions/HTMLElement.html: * LayoutTests/imported/w3c/web-platform-tests/html/dom/elements/global-attributes/cue-expected.txt: Added. * LayoutTests/imported/w3c/web-platform-tests/html/dom/elements/global-attributes/cue.html: Added. * LayoutTests/imported/w3c/web-platform-tests/html/dom/elements/global-attributes/cuebackground-expected.txt: Added. * LayoutTests/imported/w3c/web-platform-tests/html/dom/elements/global-attributes/cuebackground.html: Added. * LayoutTests/imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/historical-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/historical.html: * LayoutTests/imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/interfaces/TextTrackCue/constructor-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/interfaces/TextTrackCue/constructor.html: * LayoutTests/inspector/model/remote-object/dom-expected.txt: * LayoutTests/media/track/texttrackcue/texttrackcue-constructor-expected.txt: * LayoutTests/media/track/texttrackcue/texttrackcue-constructor.html: * Source/WebCore/html/HTMLAttributeNames.in: * Source/WebCore/html/HTMLElement.idl: * Source/WebCore/html/track/TextTrackCue.cpp: (WebCore::isLegalNode): (WebCore::tagPseudoObjects): (WebCore::removePseudoAttributes): (WebCore::TextTrackCue::create): (WebCore::TextTrackCue::setPauseOnExit): (WebCore::cueAttributName): Deleted. (WebCore::cueBackgroundAttributName): Deleted. Canonical link: https://commits.webkit.org/268644@main

evanbliu · 2023-10-03T18:13:34Z

Chrome is generally supportive of this effort, but won't have the bandwidth to implement the changes any time soon.

nigelmegitt · 2023-10-05T10:23:40Z

I think this goes a little to far in preventing the creation of a TextTrackCue with no cueFragment, which is important for those use cases where the purpose of using a cue object is to fire onEnter and onExit handlers. As I read it in this proposal, doing so would throw an InvalidNodeTypeError DOMException.

I'm not sure the best document change to allow this, whether to explicitly allow the cueFragment to be null, or to support two constructors, one requiring cueFragment, the other with it absent.

marcoscaceres · 2023-10-16T00:57:45Z

I think this goes a little to far in preventing the creation of a TextTrackCue with no cueFragment, which is important for those use cases where the purpose of using a cue object is to fire onEnter and onExit handlers. As I read it in this proposal, doing so would throw an InvalidNodeTypeError DOMException.

Right, but you could provide just an empty, for example . That's not amazing, so yeah... tend to I agree.

I'm not sure the best document change to allow this, whether to explicitly allow the cueFragment to be null, or to support two constructors, one requiring cueFragment, the other with it absent.

Can you help me understand that use case a little bit more?... I'm also leaning towards just allowing the DocumentFragment to be optional (would be equivalent to calling new VTTCue(1,2, "") ).

nigelmegitt · 2023-10-16T09:30:08Z

Can you help me understand that use case a little bit more?... I'm also leaning towards just allowing the DocumentFragment to be optional (would be equivalent to calling new VTTCue(1,2, "") ).

There are loads of use cases for this. Generically, anyone wanting to subclass TextTrackCue to add either metadata or time-based playback behaviour changes that don't involve adding HTML fragments to the shadow DOM attached to the <video>. E.g. adding HTML fragments to some other element on the page, like an indicator of the score during a sports video, or reactions, or "here's an interesting fact about this thing that's happening right now", etc etc.

Assuming that the subclass has to instantiate the base class, the base class constructor needs to have the HTML fragment as an optional parameter. It's a small thing, especially in Ecmascript, but I'd lean towards allowing omission of the HTML fragment rather than specifying an empty string for it, just for cleanliness.

past · 2023-10-20T18:07:58Z

Tagging @zcorpan and @alastor0325 per the discussion in #9836.

zcorpan · 2023-10-24T10:37:42Z

The rendering of WebVTT is currently specified as a custom non-CSS renderer of a non-DOM tree. I believe browsers have mostly implemented the rendering using CSS and DOM. I think they also don't follow the rendering requirements exactly, e.g. Safari has some padding on the background box and rounded corners, whereas the spec requires "uglier" background box without padding.

Maybe this could be an opportunity to make the spec more aligned with implementations?

marcoscaceres · 2023-11-19T23:57:56Z

Maybe this could be an opportunity to make the spec more aligned with implementations?

We discussed internally with WebKit folks, and yes: we should actually fix the interop issues with WebVTT across user agents in addition to this.

We still think this proposal might still be worth pursuing, but fixing WebVTT would probably give a higher return on investment.

We might let this proposal sit for a while, while we focus on increasing interop of WebVTT.

marcoscaceres force-pushed the TextTrackCue branch from cb842a6 to e46b5a0 Compare September 22, 2023 17:02

annevk added addition/proposal New features or enhancements topic: media labels Sep 25, 2023

annevk reviewed Sep 25, 2023

View reviewed changes

source Show resolved Hide resolved

source Outdated Show resolved Hide resolved

source Outdated Show resolved Hide resolved

source Outdated Show resolved Hide resolved

marcoscaceres force-pushed the TextTrackCue branch 6 times, most recently from 47da44c to 5362a5e Compare September 25, 2023 17:14

marcoscaceres mentioned this pull request Sep 25, 2023

Implement updated TextTrackCue constructor proposal WebKit/WebKit#18176

Merged

marcoscaceres marked this pull request as ready for review September 25, 2023 17:50

marcoscaceres requested a review from zcorpan September 27, 2023 14:14

marcoscaceres mentioned this pull request Sep 27, 2023

TextTrackCue enhancements for programmatic subtitle and caption presentation mozilla/standards-positions#891

Open

marcoscaceres force-pushed the TextTrackCue branch from 5362a5e to 0ab61ed Compare September 27, 2023 15:05

nigelmegitt reviewed Sep 27, 2023

View reviewed changes

marcoscaceres mentioned this pull request Sep 29, 2023

Tests for TextTrackCue ctor and cue + cuebackground global attrs web-platform-tests/wpt#42243

Open

marcoscaceres force-pushed the TextTrackCue branch from 0ab61ed to 0087d58 Compare October 5, 2023 08:29

annevk mentioned this pull request Oct 5, 2023

Upcoming HTML standard issue triage meeting on 10/5/2023 #9706

Closed

Add TextTrackCue ctor & cue + cuebackground attrs

69001aa

marcoscaceres force-pushed the TextTrackCue branch 2 times, most recently from 4489cb1 to 69001aa Compare October 5, 2023 13:54

estelle mentioned this pull request Oct 5, 2023

IMSC: is it still relevant or should it be removed? mdn/content#29506

Closed

past mentioned this pull request Oct 20, 2023

Upcoming HTML standard issue triage meeting on 10/19/2023 #9836

Closed

add example, fix more xrefs

9c2aa7c

past mentioned this pull request Nov 2, 2023

Upcoming HTML standard issue triage meeting on 11/2/2023 #9874

Closed

-  <p>When the <code data-x="attr-cue">cue</code> attribute is present, the element serves as a <span>text track
-  cue</span>.</p>
+  <p>When the <code data-x="attr-cue">cue</code> attribute is present, the element can be styled as the text content of a <span>text track
+  cue</span> using the <span>::cue</span> <span>text track cue
+     pseudo-element</span>.</p>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TextTrackCue ctor & cue + cuebackground attributes #9771

Add TextTrackCue ctor & cue + cuebackground attributes #9771

marcoscaceres commented Sep 21, 2023 •

edited by pr-preview bot

Loading

annevk left a comment

marcoscaceres commented Sep 27, 2023

nigelmegitt commented Sep 27, 2023

nigelmegitt Sep 27, 2023

marcoscaceres Sep 28, 2023 •

edited

Loading

nigelmegitt Sep 27, 2023

marcoscaceres Sep 28, 2023

evanbliu commented Oct 3, 2023

nigelmegitt commented Oct 5, 2023

marcoscaceres commented Oct 16, 2023

nigelmegitt commented Oct 16, 2023

past commented Oct 20, 2023

zcorpan commented Oct 24, 2023

marcoscaceres commented Nov 19, 2023

	<p>The <code data-x="attr-cue">cue</code> and <code data-x="attr-cuebackground">cuebackground</code> attributes must each be present once.</p>
	<p>The <code data-x="attr-cue">cue</code> and <code data-x="attr-cuebackground">cuebackground</code> attributes must each be present on at least one element in the <var>cueFragment</var>.</p>

		<p>When the <code data-x="attr-cue">cue</code> attribute is present, the element serves as a <span>text track
		cue</span>.</p>

Add TextTrackCue ctor & cue + cuebackground attributes #9771

Are you sure you want to change the base?

Add TextTrackCue ctor & cue + cuebackground attributes #9771

Conversation

marcoscaceres commented Sep 21, 2023 • edited by pr-preview bot Loading

annevk left a comment

Choose a reason for hiding this comment

marcoscaceres commented Sep 27, 2023

nigelmegitt commented Sep 27, 2023

nigelmegitt Sep 27, 2023

Choose a reason for hiding this comment

marcoscaceres Sep 28, 2023 • edited Loading

Choose a reason for hiding this comment

nigelmegitt Sep 27, 2023

Choose a reason for hiding this comment

marcoscaceres Sep 28, 2023

Choose a reason for hiding this comment

evanbliu commented Oct 3, 2023

nigelmegitt commented Oct 5, 2023

marcoscaceres commented Oct 16, 2023

nigelmegitt commented Oct 16, 2023

past commented Oct 20, 2023

zcorpan commented Oct 24, 2023

marcoscaceres commented Nov 19, 2023

marcoscaceres commented Sep 21, 2023 •

edited by pr-preview bot

Loading

marcoscaceres Sep 28, 2023 •

edited

Loading