Live captioning - JS API review

[silviapfeiffer]

To address REQ1 of #318 , we are after an extension of the TextTrack JS API.

The principle idea is that we create a new empty cue with a NULL end time, render it, then add text to its content, change its position, change text color and other attributes until the cue is complete at which stage we know the end time and close it off. To render rollup captions, we can further create cue regions and successively add new cues to them.

The functionality of the 608 live captioning commands will inform what needs we have, so here is a first go at a mapping table that we created at FOMS. Thanks to Evol from JWplayer for his help!

Ref	608 control commands	TextTrackCue API calls
1	start caption text / resume caption text / resume direct captioning	new VTTCue(now(), NULL, '') - make sure to set the defaults as required by 608 - then: textTrack.addCue(cue)
2	add a character	cue.text += char
3	next row down toggle (includes end all style)	cue.text += '\n' (may need to end </c>,</i>,</b>,</u>)
4	row indicator (one of 15 rows)	cue.line = row (whichever row calculated)
5	underline toggle	cue.text += "<u>" or cue.text += "</u>" (need to keep track of toggle state)
6	style change (one of 7 text colors and italics)	cue.text += "<c.white>" (need to have the color style classes pre-defined) and cue.text += "<i>"
7	8 ident positions	cue.position = offset (whichever offset calculated from ident pos)
8	8 background colors	cue.text += "<c.bg_white>" (need to have the background color style classes pre-defined)
9	backspace	cue.text = cue.text.substr(0, cue.text.length - 1)
10	delete till end of row	cue.text = cue.text.substr(0, cursor_pos) (need to keep track of the 608 cursor position)
11	rollup caption with 2, 3 or 4 rows	new VTTRegion() then region.lines = x
12	flash on (srlsy?)	cue.text += "<c.blink>"
13	erase displayed memory (clear screen)	cue.text = ''
14	carriage return (scroll lines up)	cue.endTime = now(); cue.region = region; new VTTCue();
15	end of caption	cue.endTime = now()
16	clear screen (erase display memory)	cue.text = ''
17	tab offset 1/2/3 (add whitespace)	cue.text += " " * num_space (calculate numspace from tab offset as per https://dvcs.w3.org/hg/text-tracks/raw-file/default/608toVTT/608toVTT.html )

Analysing the table and browser implementations, it seems we already have most of the required functionality to do live captioning in the browser, bar the following:

• Chrome has a bug that when you change any object properties of an already rendered TextTrackCue, the rendering does not get updated.
• It's not possible to create cues with an open end time. The work-around for this right now is to use an end time that is really far into the future and then to update it when the end time is known.

Note that the now() function is a replacement for any start timestamp information that is available to the browser for the cue and may possibly be video.currentTime.

[DanielBaulig]

Chrome has a bug that when you change any object properties of an already rendered TextTrackCue, the rendering does not get updated.

Is there a bug reported against Blink/Chromium? Is there a bug id?

[silviapfeiffer]

Is there a bug reported against Blink/Chromium? Is there a bug id?

@DanielBaulig I don't know - might be worth checking and registering

[johnpallett]

Line 2 would suggest allowing addition of string (not just character)