AX: Consider VTT-based audio descriptions with text-to-speech. #3486

eric-carlson · 2022-08-19T20:52:13Z

`969ac99`

AX: Consider VTT-based audio descriptions with text-to-speech.
https://bugs.webkit.org/show_bug.cgi?id=243600
rdar://98206665

Reviewed by Jer Noble.

* LayoutTests/media/track/captions-webvtt/captions-descriptions.vtt: Added.
* LayoutTests/media/track/track-description-cue-expected.txt: Added.
* LayoutTests/media/track/track-description-cue.html: Added.
* LayoutTests/TestExpectations: Feature is Cocoa-specific so far, skip test globally.
* LayoutTests/platform/ios/TestExpectations: Enable test on iOS.
* LayoutTests/platform/mac/TestExpectations: Enable test on macOS.

* Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml: Add 'AudioDescriptionsEnabled'
setting.

* Source/WebCore/Modules/speech/SpeechSynthesis.cpp:
(WebCore::SpeechSynthesis::handleSpeakingCompleted): Call utterance to fire event.
(WebCore::SpeechSynthesis::boundaryEventOccurred): Ditto.
(WebCore::SpeechSynthesis::didStartSpeaking): Ditto.
(WebCore::SpeechSynthesis::didPauseSpeaking): Ditto.
(WebCore::SpeechSynthesis::didResumeSpeaking): Ditto.
(WebCore::SpeechSynthesis::fireEvent const): Deleted.
(WebCore::SpeechSynthesis::fireErrorEvent const): Deleted.
* Source/WebCore/Modules/speech/SpeechSynthesis.h:
(WebCore::SpeechSynthesis::userGestureRequiredForSpeechStart const):
(WebCore::SpeechSynthesis::removeBehaviorRestriction): Make public.

* Source/WebCore/Modules/speech/SpeechSynthesisUtterance.cpp:
(WebCore::SpeechSynthesisUtterance::create): Add version that takes a completion handler.
(WebCore::SpeechSynthesisUtterance::SpeechSynthesisUtterance): Add completion handler
parameter.
(WebCore::SpeechSynthesisUtterance::eventOccurred): New. Call completion handler or
dispatch event.
(WebCore::SpeechSynthesisUtterance::errorEventOccurred): Ditto.
* Source/WebCore/Modules/speech/SpeechSynthesisUtterance.h:
* Source/WebCore/Modules/speech/SpeechSynthesisUtterance.idl: JSGenerateToJSObject.

* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::updateActiveTextTrackCues): Return early if a seek
is pending. Call new executeCueEnterOrLeaveAction method instead of dispatching
events directly.
(WebCore::HTMLMediaElement::setSpeechSynthesisState): Maintain synthesis state.
(WebCore::HTMLMediaElement::speakCueText): Speak a cue.
(WebCore::HTMLMediaElement::pauseSpeakingCueText):
(WebCore::HTMLMediaElement::resumeSpeakingCueText):
(WebCore::HTMLMediaElement::cancelSpeakingCueText):
(WebCore::HTMLMediaElement::shouldSpeakCueTextForTime):
(WebCore::HTMLMediaElement::executeCueEnterOrLeaveAction): Trigger cue speech if
the track contains descriptions and we are entering a cue range, schedule an enter
or exit event.
(WebCore::HTMLMediaElement::seekWithTolerance): INFO_LOG -> ALWAYS_LOG
(WebCore::HTMLMediaElement::seekTask): Cancel speaking if necessary.
(WebCore::HTMLMediaElement::finishSeek): Update logging. If there isn't a pending
seek, queue a task to update text track cues.
(WebCore::HTMLMediaElement::configureTextTrackGroup): When processing descriptions
and the user wants text descriptions, set `fallbackTrack` to the first track seen
in case none of the tracks matches the audio track language.
(WebCore::HTMLMediaElement::playPlayer): Call resumeSpeakingCueText.
(WebCore::HTMLMediaElement::pausePlayer): Call pauseSpeakingCueText.
(WebCore::HTMLMediaElement::effectiveVolume const): Use the speech volume multiplier
when calculating the effective volume.
(WebCore::m_categoryAtMostRecentPlayback): Deleted.
* Source/WebCore/html/HTMLMediaElement.h:
(WebCore::HTMLMediaElement::cueBeingSpoken const):

* Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.cpp:
(WebCore::MediaControlTextTrackContainerElement::updateDisplay): Skip spoken tracks.

* Source/WebCore/html/track/InbandGenericTextTrack.cpp: Remove unneeded include.

* Source/WebCore/html/track/TextTrack.cpp:
(WebCore::TextTrack::trackIndex): Use textTrackList() instead of m_textTrackList.
(WebCore::TextTrack::isRendered): Consider descriptions.
(WebCore::TextTrack::isSpoken):
(WebCore::TextTrack::trackIndexRelativeToRenderedTracks):  Use textTrackList()
instead of m_textTrackList.
(WebCore::TextTrack::speechSynthesis):
* Source/WebCore/html/track/TextTrack.h:

* Source/WebCore/html/track/TextTrackCue.cpp:
(WebCore::operator<<): All cues have a `text()` method, just use it.
* Source/WebCore/html/track/TextTrackCue.h:
(WebCore::TextTrackCue::text const):
(WebCore::TextTrackCue::speak):

* Source/WebCore/html/track/VTTCue.cpp:
(WebCore::VTTCue::updateDisplayTree): `track()` can return null, check it.
(WebCore::VTTCue::getDisplayTree): Ditto.
(WebCore::VTTCue::toJSON const): Drive-by: address a Darin FIXME.
(WebCore::VTTCue::speak):
* Source/WebCore/html/track/VTTCue.h:
(WebCore::VTTCue::speechUtterance const):
(WebCore::VTTCue::text const): Deleted.
* Source/WebCore/html/track/VTTCue.idl:

* Source/WebCore/page/CaptionUserPreferences.cpp:
(WebCore::CaptionUserPreferences::userPrefersTextDescriptions const): Check audioDescriptionsEnabled
setting.
(WebCore::CaptionUserPreferences::textTrackSelectionScore const): Consider description
tracks if the user preference is set. Clean up logic.

* Source/WebCore/page/CaptionUserPreferencesMediaAF.cpp:
(WebCore::CaptionUserPreferencesMediaAF::userPrefersCaptions const):
(WebCore::CaptionUserPreferencesMediaAF::userPrefersTextDescriptions const): check
MediaAccessibility framework preference.
* Source/WebCore/page/CaptionUserPreferencesMediaAF.h:
* Source/WebCore/platform/cf/MediaAccessibilitySoftLink.cpp:
* Source/WebCore/platform/cf/MediaAccessibilitySoftLink.h:

* Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm:
(-[WebSpeechSynthesisWrapper speakUtterance:]): If `utteranceVoice` is non-NULL
and the URI is empty, it is invalid so check the language.

* Source/WebCore/platform/graphics/InbandGenericCue.cpp:
(WebCore::InbandGenericCue::toJSONString const): Log cue text like VTTCue now does.

* Source/WebCore/platform/graphics/iso/ISOVTTCue.cpp:
(WebCore::ISOWebVTTCue::toJSONString const): Ditto.

* Source/WebCore/testing/Internals.cpp:
(WebCore::Internals::speechSynthesisUtteranceForCue):
(WebCore::Internals::mediaElementCurrentlySpokenCue):
* Source/WebCore/testing/Internals.h:
* Source/WebCore/testing/Internals.idl:

* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::speechSynthesisSpeak): Remove the `startTime` parameter name,
it isn't used.

Canonical link: https://commits.webkit.org/253931@main

2afe52d

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	~~🛠 mac~~	✅ 🛠 wpe	✅ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	~~🛠 mac-debug~~	✅ 🛠 gtk	~~🛠 wincairo~~
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	✅ 🛠 mac-AS-debug	✅ 🧪 gtk-wk2
	~~🧪 api-ios~~	~~🧪 api-mac~~	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc	✅ 🛠 tv	~~🧪 mac-wk1~~	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	~~🧪 mac-wk2~~	✅ 🧪 jsc-armv7-tests
✅ 🛠 🧪 merge	✅ 🛠 watch	~~🧪 mac-AS-debug-wk2~~	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	~~🧪 mac-wk2-stress~~	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-19T20:52:31Z

EWS run on previous version of this PR (hash 4ea8460)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	⏳ 🛠 mac	❌ 🛠 wpe	❌ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	⏳ 🛠 mac-debug	❌ 🛠 gtk	❌ 🛠 wincairo
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	⏳ 🛠 mac-AS-debug	❌ 🧪 gtk-wk2
	~~🧪 api-ios~~	⏳ 🧪 api-mac	❌ 🧪 api-gtk
✅ 🛠 🧪 jsc	❌ 🛠 tv	⏳ 🧪 mac-wk1	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	⏳ 🧪 mac-wk2	✅ 🧪 jsc-armv7-tests
	❌ 🛠 watch	⏳ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	⏳ 🧪 mac-wk2-stress	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-19T22:47:11Z

EWS run on previous version of this PR (hash cd29f04)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	⏳ 🛠 mac	⏳ 🛠 wpe	~~🛠 🧪 win~~
⏳ 🧪 bindings	✅ 🛠 ios-sim	⏳ 🛠 mac-debug	⏳ 🛠 gtk	⏳ 🛠 wincairo
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	⏳ 🛠 mac-AS-debug	⏳ 🧪 gtk-wk2
	~~🧪 api-ios~~	⏳ 🧪 api-mac	⏳ 🧪 api-gtk
⏳ 🛠 🧪 jsc	⏳ 🛠 tv	⏳ 🧪 mac-wk1	✅ 🛠 jsc-armv7
	⏳ 🛠 tv-sim	⏳ 🧪 mac-wk2	~~🧪 jsc-armv7-tests~~
	⏳ 🛠 watch	⏳ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-mips
	⏳ 🛠 watch-sim	⏳ 🧪 mac-wk2-stress	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-19T22:49:33Z

EWS run on previous version of this PR (hash dfb23ea)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	❌ 🛠 ios	❌ 🛠 mac	❌ 🛠 wpe	❌ 🛠 🧪 win
✅ 🧪 bindings	❌ 🛠 ios-sim	❌ 🛠 mac-debug	❌ 🛠 gtk	❌ 🛠 wincairo
✅ 🧪 webkitperl	❌ 🧪 ios-wk2	❌ 🛠 mac-AS-debug	❌ 🧪 gtk-wk2
	❌ 🧪 api-ios	❌ 🧪 api-mac	❌ 🧪 api-gtk
✅ 🛠 🧪 jsc	❌ 🛠 tv	❌ 🧪 mac-wk1	✅ 🛠 jsc-armv7
	❌ 🛠 tv-sim	❌ 🧪 mac-wk2	✅ 🧪 jsc-armv7-tests
	❌ 🛠 watch	❌ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-mips
	❌ 🛠 watch-sim	❌ 🧪 mac-wk2-stress	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-20T23:19:01Z

EWS run on previous version of this PR (hash 361d3b7)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	❌ 🛠 wpe	❌ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-debug	❌ 🛠 gtk	❌ 🛠 wincairo
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	✅ 🛠 mac-AS-debug	❌ 🧪 gtk-wk2
	✅ 🧪 api-ios	⏳ 🧪 api-mac	❌ 🧪 api-gtk
✅ 🛠 🧪 jsc	✅ 🛠 tv	~~🧪 mac-wk1~~	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	✅ 🧪 mac-wk2	✅ 🧪 jsc-armv7-tests
	✅ 🛠 watch	~~🧪 mac-AS-debug-wk2~~	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-21T02:43:41Z

EWS run on previous version of this PR (hash 22b2f19)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	❌ 🛠 wpe	❌ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-debug	❌ 🛠 gtk	❌ 🛠 wincairo
✅ 🧪 webkitperl	❌ 🧪 ios-wk2	✅ 🛠 mac-AS-debug	❌ 🧪 gtk-wk2
	✅ 🧪 api-ios	~~🧪 api-mac~~	❌ 🧪 api-gtk
✅ 🛠 🧪 jsc	✅ 🛠 tv	❌ 🧪 mac-wk1	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	✅ 🧪 mac-wk2	✅ 🧪 jsc-armv7-tests
	✅ 🛠 watch	✅ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-21T14:34:02Z

EWS run on previous version of this PR (hash 847fe9e)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	❌ 🛠 wpe	❌ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-debug	❌ 🛠 gtk	❌ 🛠 wincairo
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	✅ 🛠 mac-AS-debug	❌ 🧪 gtk-wk2
	✅ 🧪 api-ios	⏳ 🧪 api-mac	❌ 🧪 api-gtk
✅ 🛠 🧪 jsc	✅ 🛠 tv	~~🧪 mac-wk1~~	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	✅ 🧪 mac-wk2	✅ 🧪 jsc-armv7-tests
	✅ 🛠 watch	~~🧪 mac-AS-debug-wk2~~	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-21T17:13:27Z

EWS run on previous version of this PR (hash acee957)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-debug	✅ 🛠 gtk	✅ 🛠 wincairo
✅ 🧪 webkitperl	❌ 🧪 ios-wk2	✅ 🛠 mac-AS-debug	❌ 🧪 gtk-wk2
	✅ 🧪 api-ios	✅ 🧪 api-mac	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc	✅ 🛠 tv	❌ 🧪 mac-wk1	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	✅ 🧪 mac-wk2	✅ 🧪 jsc-armv7-tests
	✅ 🛠 watch	❌ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2022-08-28T16:39:34Z

EWS run on previous version of this PR (hash 46e92dc)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-debug	✅ 🛠 gtk	✅ 🛠 wincairo
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🛠 mac-AS-debug	✅ 🧪 gtk-wk2
	✅ 🧪 api-ios	✅ 🧪 api-mac	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc	✅ 🛠 tv	✅ 🧪 mac-wk1	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	✅ 🧪 mac-wk2	✅ 🧪 jsc-armv7-tests
	✅ 🛠 watch	✅ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	✅ 🧪 mac-wk2-stress	❌ 🧪 jsc-mips-tests

youssefsoli · 2022-08-29T21:01:07Z

Source/WebCore/html/HTMLMediaElement.cpp

-
-            auto exitEvent = Event::create(eventNames().exitEvent, Event::CanBubble::No, Event::IsCancelable::No);
-            scheduleEventOn(*eventTask.second, WTFMove(exitEvent));
+            executeCueEnterOrLeaveAction(*eventTask.second, CueAction::Enter);


nit: It might be cleaner to make eventTask a structured binding auto& [eventTime, eventCue]

Good point, we should do that as a followup.

youssefsoli · 2022-08-29T21:08:12Z

Source/WebCore/html/HTMLMediaElement.cpp

+    if (!m_cueBeingSpoken)
+        return false;
+
+    ALWAYS_LOG(LOGIDENTIFIER, "time = ", time, ", returning ", (time.toDouble() >= m_cueBeingSpoken->startTime() && time.toDouble() < m_cueBeingSpoken->endTime()));


(time.toDouble() >= m_cueBeingSpoken->startTime() && time.toDouble() < m_cueBeingSpoken->endTime()) can be its own variable like timeIsInRange

Yes, I'll change that.

youssefsoli · 2022-08-29T21:17:06Z

Source/WebCore/html/track/VTTCue.cpp

+        return;
+
+    auto& track = *this->track();
+    if (!m_speechUtterance) {


Is it possible for m_speechUtterance to exist at this point? If so, the completion handler won't be called.

It shouldn't be possible, I'll add an assert and will remove the test.

webkit-early-warning-system · 2022-08-30T00:00:50Z

2afe52d

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	~~🛠 mac~~	✅ 🛠 wpe	✅ 🛠 🧪 win
✅ 🧪 bindings	✅ 🛠 ios-sim	~~🛠 mac-debug~~	✅ 🛠 gtk	~~🛠 wincairo~~
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	✅ 🛠 mac-AS-debug	✅ 🧪 gtk-wk2
	~~🧪 api-ios~~	~~🧪 api-mac~~	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc	✅ 🛠 tv	~~🧪 mac-wk1~~	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim	~~🧪 mac-wk2~~	✅ 🧪 jsc-armv7-tests
✅ 🛠 🧪 merge	✅ 🛠 watch	~~🧪 mac-AS-debug-wk2~~	✅ 🛠 jsc-mips
	✅ 🛠 watch-sim	~~🧪 mac-wk2-stress~~	✅ 🧪 jsc-mips-tests

https://bugs.webkit.org/show_bug.cgi?id=243600 rdar://98206665 Reviewed by Jer Noble. * LayoutTests/media/track/captions-webvtt/captions-descriptions.vtt: Added. * LayoutTests/media/track/track-description-cue-expected.txt: Added. * LayoutTests/media/track/track-description-cue.html: Added. * LayoutTests/TestExpectations: Feature is Cocoa-specific so far, skip test globally. * LayoutTests/platform/ios/TestExpectations: Enable test on iOS. * LayoutTests/platform/mac/TestExpectations: Enable test on macOS. * Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml: Add 'AudioDescriptionsEnabled' setting. * Source/WebCore/Modules/speech/SpeechSynthesis.cpp: (WebCore::SpeechSynthesis::handleSpeakingCompleted): Call utterance to fire event. (WebCore::SpeechSynthesis::boundaryEventOccurred): Ditto. (WebCore::SpeechSynthesis::didStartSpeaking): Ditto. (WebCore::SpeechSynthesis::didPauseSpeaking): Ditto. (WebCore::SpeechSynthesis::didResumeSpeaking): Ditto. (WebCore::SpeechSynthesis::fireEvent const): Deleted. (WebCore::SpeechSynthesis::fireErrorEvent const): Deleted. * Source/WebCore/Modules/speech/SpeechSynthesis.h: (WebCore::SpeechSynthesis::userGestureRequiredForSpeechStart const): (WebCore::SpeechSynthesis::removeBehaviorRestriction): Make public. * Source/WebCore/Modules/speech/SpeechSynthesisUtterance.cpp: (WebCore::SpeechSynthesisUtterance::create): Add version that takes a completion handler. (WebCore::SpeechSynthesisUtterance::SpeechSynthesisUtterance): Add completion handler parameter. (WebCore::SpeechSynthesisUtterance::eventOccurred): New. Call completion handler or dispatch event. (WebCore::SpeechSynthesisUtterance::errorEventOccurred): Ditto. * Source/WebCore/Modules/speech/SpeechSynthesisUtterance.h: * Source/WebCore/Modules/speech/SpeechSynthesisUtterance.idl: JSGenerateToJSObject. * Source/WebCore/html/HTMLMediaElement.cpp: (WebCore::HTMLMediaElement::updateActiveTextTrackCues): Return early if a seek is pending. Call new executeCueEnterOrLeaveAction method instead of dispatching events directly. (WebCore::HTMLMediaElement::setSpeechSynthesisState): Maintain synthesis state. (WebCore::HTMLMediaElement::speakCueText): Speak a cue. (WebCore::HTMLMediaElement::pauseSpeakingCueText): (WebCore::HTMLMediaElement::resumeSpeakingCueText): (WebCore::HTMLMediaElement::cancelSpeakingCueText): (WebCore::HTMLMediaElement::shouldSpeakCueTextForTime): (WebCore::HTMLMediaElement::executeCueEnterOrLeaveAction): Trigger cue speech if the track contains descriptions and we are entering a cue range, schedule an enter or exit event. (WebCore::HTMLMediaElement::seekWithTolerance): INFO_LOG -> ALWAYS_LOG (WebCore::HTMLMediaElement::seekTask): Cancel speaking if necessary. (WebCore::HTMLMediaElement::finishSeek): Update logging. If there isn't a pending seek, queue a task to update text track cues. (WebCore::HTMLMediaElement::configureTextTrackGroup): When processing descriptions and the user wants text descriptions, set `fallbackTrack` to the first track seen in case none of the tracks matches the audio track language. (WebCore::HTMLMediaElement::playPlayer): Call resumeSpeakingCueText. (WebCore::HTMLMediaElement::pausePlayer): Call pauseSpeakingCueText. (WebCore::HTMLMediaElement::effectiveVolume const): Use the speech volume multiplier when calculating the effective volume. (WebCore::m_categoryAtMostRecentPlayback): Deleted. * Source/WebCore/html/HTMLMediaElement.h: (WebCore::HTMLMediaElement::cueBeingSpoken const): * Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.cpp: (WebCore::MediaControlTextTrackContainerElement::updateDisplay): Skip spoken tracks. * Source/WebCore/html/track/InbandGenericTextTrack.cpp: Remove unneeded include. * Source/WebCore/html/track/TextTrack.cpp: (WebCore::TextTrack::trackIndex): Use textTrackList() instead of m_textTrackList. (WebCore::TextTrack::isRendered): Consider descriptions. (WebCore::TextTrack::isSpoken): (WebCore::TextTrack::trackIndexRelativeToRenderedTracks): Use textTrackList() instead of m_textTrackList. (WebCore::TextTrack::speechSynthesis): * Source/WebCore/html/track/TextTrack.h: * Source/WebCore/html/track/TextTrackCue.cpp: (WebCore::operator<<): All cues have a `text()` method, just use it. * Source/WebCore/html/track/TextTrackCue.h: (WebCore::TextTrackCue::text const): (WebCore::TextTrackCue::speak): * Source/WebCore/html/track/VTTCue.cpp: (WebCore::VTTCue::updateDisplayTree): `track()` can return null, check it. (WebCore::VTTCue::getDisplayTree): Ditto. (WebCore::VTTCue::toJSON const): Drive-by: address a Darin FIXME. (WebCore::VTTCue::speak): * Source/WebCore/html/track/VTTCue.h: (WebCore::VTTCue::speechUtterance const): (WebCore::VTTCue::text const): Deleted. * Source/WebCore/html/track/VTTCue.idl: * Source/WebCore/page/CaptionUserPreferences.cpp: (WebCore::CaptionUserPreferences::userPrefersTextDescriptions const): Check audioDescriptionsEnabled setting. (WebCore::CaptionUserPreferences::textTrackSelectionScore const): Consider description tracks if the user preference is set. Clean up logic. * Source/WebCore/page/CaptionUserPreferencesMediaAF.cpp: (WebCore::CaptionUserPreferencesMediaAF::userPrefersCaptions const): (WebCore::CaptionUserPreferencesMediaAF::userPrefersTextDescriptions const): check MediaAccessibility framework preference. * Source/WebCore/page/CaptionUserPreferencesMediaAF.h: * Source/WebCore/platform/cf/MediaAccessibilitySoftLink.cpp: * Source/WebCore/platform/cf/MediaAccessibilitySoftLink.h: * Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm: (-[WebSpeechSynthesisWrapper speakUtterance:]): If `utteranceVoice` is non-NULL and the URI is empty, it is invalid so check the language. * Source/WebCore/platform/graphics/InbandGenericCue.cpp: (WebCore::InbandGenericCue::toJSONString const): Log cue text like VTTCue now does. * Source/WebCore/platform/graphics/iso/ISOVTTCue.cpp: (WebCore::ISOWebVTTCue::toJSONString const): Ditto. * Source/WebCore/testing/Internals.cpp: (WebCore::Internals::speechSynthesisUtteranceForCue): (WebCore::Internals::mediaElementCurrentlySpokenCue): * Source/WebCore/testing/Internals.h: * Source/WebCore/testing/Internals.idl: * Source/WebKit/UIProcess/WebPageProxy.cpp: (WebKit::WebPageProxy::speechSynthesisSpeak): Remove the `startTime` parameter name, it isn't used. Canonical link: https://commits.webkit.org/253931@main

webkit-commit-queue · 2022-08-30T01:59:53Z

Committed 253931@main (969ac99): https://commits.webkit.org/253931@main

Reviewed commits have been landed. Closing PR #3486 and removing active labels.

eric-carlson requested review from cdumez and rniwa as code owners August 19, 2022 20:52

eric-carlson self-assigned this Aug 19, 2022

eric-carlson added Accessibility For bugs related to accessibility. WebKit Nightly Build labels Aug 19, 2022

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Aug 19, 2022

eric-carlson force-pushed the audio-description-track branch from 4ea8460 to cd29f04 Compare August 19, 2022 22:45

eric-carlson force-pushed the audio-description-track branch from cd29f04 to dfb23ea Compare August 19, 2022 22:49

eric-carlson force-pushed the audio-description-track branch from dfb23ea to 361d3b7 Compare August 20, 2022 23:17

eric-carlson force-pushed the audio-description-track branch from 361d3b7 to 22b2f19 Compare August 21, 2022 02:43

eric-carlson force-pushed the audio-description-track branch from 22b2f19 to 847fe9e Compare August 21, 2022 14:33

eric-carlson force-pushed the audio-description-track branch from 847fe9e to acee957 Compare August 21, 2022 17:13

eric-carlson force-pushed the audio-description-track branch from acee957 to 46e92dc Compare August 28, 2022 16:37

eric-carlson requested a review from jernoble August 28, 2022 16:38

youssefsoli reviewed Aug 29, 2022

View reviewed changes

jernoble approved these changes Aug 29, 2022

View reviewed changes

eric-carlson force-pushed the audio-description-track branch from 46e92dc to 2afe52d Compare August 29, 2022 23:59

eric-carlson added merge-queue Applied to send a pull request to merge-queue and removed merging-blocked Applied to prevent a change from being merged labels Aug 30, 2022

webkit-early-warning-system force-pushed the audio-description-track branch from 2afe52d to 969ac99 Compare August 30, 2022 01:59

webkit-early-warning-system merged commit 969ac99 into WebKit:main Aug 30, 2022

webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Aug 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AX: Consider VTT-based audio descriptions with text-to-speech. #3486

AX: Consider VTT-based audio descriptions with text-to-speech. #3486

eric-carlson commented Aug 19, 2022 •

edited by webkit-early-warning-system

webkit-early-warning-system commented Aug 19, 2022 •

edited

webkit-early-warning-system commented Aug 19, 2022 •

edited

webkit-early-warning-system commented Aug 19, 2022 •

edited

webkit-early-warning-system commented Aug 20, 2022 •

edited

webkit-early-warning-system commented Aug 21, 2022 •

edited

webkit-early-warning-system commented Aug 21, 2022 •

edited

webkit-early-warning-system commented Aug 21, 2022 •

edited

webkit-early-warning-system commented Aug 28, 2022 •

edited

youssefsoli Aug 29, 2022

eric-carlson Aug 29, 2022

youssefsoli Aug 29, 2022

eric-carlson Aug 29, 2022

youssefsoli Aug 29, 2022

eric-carlson Aug 29, 2022

webkit-early-warning-system commented Aug 30, 2022 •

edited

webkit-commit-queue commented Aug 30, 2022

AX: Consider VTT-based audio descriptions with text-to-speech. #3486

AX: Consider VTT-based audio descriptions with text-to-speech. #3486

Conversation

eric-carlson commented Aug 19, 2022 • edited by webkit-early-warning-system

969ac99

webkit-early-warning-system commented Aug 19, 2022 • edited

webkit-early-warning-system commented Aug 19, 2022 • edited

webkit-early-warning-system commented Aug 19, 2022 • edited

webkit-early-warning-system commented Aug 20, 2022 • edited

webkit-early-warning-system commented Aug 21, 2022 • edited

webkit-early-warning-system commented Aug 21, 2022 • edited

webkit-early-warning-system commented Aug 21, 2022 • edited

webkit-early-warning-system commented Aug 28, 2022 • edited

youssefsoli Aug 29, 2022

Choose a reason for hiding this comment

eric-carlson Aug 29, 2022

Choose a reason for hiding this comment

youssefsoli Aug 29, 2022

Choose a reason for hiding this comment

eric-carlson Aug 29, 2022

Choose a reason for hiding this comment

youssefsoli Aug 29, 2022

Choose a reason for hiding this comment

eric-carlson Aug 29, 2022

Choose a reason for hiding this comment

webkit-early-warning-system commented Aug 30, 2022 • edited

webkit-commit-queue commented Aug 30, 2022

eric-carlson commented Aug 19, 2022 •

edited by webkit-early-warning-system

`969ac99`

webkit-early-warning-system commented Aug 19, 2022 •

edited

webkit-early-warning-system commented Aug 19, 2022 •

edited

webkit-early-warning-system commented Aug 19, 2022 •

edited

webkit-early-warning-system commented Aug 20, 2022 •

edited

webkit-early-warning-system commented Aug 21, 2022 •

edited

webkit-early-warning-system commented Aug 21, 2022 •

edited

webkit-early-warning-system commented Aug 21, 2022 •

edited

webkit-early-warning-system commented Aug 28, 2022 •

edited

webkit-early-warning-system commented Aug 30, 2022 •

edited