Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AX: Support VTT-based extended audio descriptions #4129

Conversation

eric-carlson
Copy link
Contributor

@eric-carlson eric-carlson commented Sep 8, 2022

c2f7594

AX: Support VTT-based extended audio descriptions
https://bugs.webkit.org/show_bug.cgi?id=244931
<rdar://99697422>

Reviewed by Jer Noble.

* LayoutTests/media/track/captions-webvtt/captions-extended-descriptions.vtt: Added.
* LayoutTests/media/track/track-description-cue.html:
* LayoutTests/media/track/track-extended-descriptions-expected.txt: Added.
* LayoutTests/media/track/track-extended-descriptions.html: Added.
* LayoutTests/TestExpectations: Feature is Cocoa-specific so far, skip test globally.
* LayoutTests/platform/ios/TestExpectations: Enable test on iOS.
* LayoutTests/platform/mac/TestExpectations: Enable test on macOS.

* Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml: Add 'ExtendedAudioDescriptionsEnabled'
setting.

* Source/WebCore/Modules/speech/SpeechSynthesis.cpp:
(WebCore::SpeechSynthesis::setPlatformSynthesizer): Take a Ref<> synthesizer instead of a
unique_ptr.
(WebCore::SpeechSynthesis::ensurePlatformSpeechSynthesizer): Use PlatformSpeechSynthesizer::create.
* Source/WebCore/Modules/speech/SpeechSynthesis.h:

* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::convertEnumerationToString): Added.
(WebCore::HTMLMediaElement::unregisterWithDocument): Cancel speech, clear synthesize.
(WebCore::HTMLMediaElement::updateActiveTextTrackCues): Cleanup for readability.
(WebCore::HTMLMediaElement::setSpeechSynthesisState): Update for extended descriptions.
(WebCore::HTMLMediaElement::speakCueText): Call cue.prepareToSpeak as we don't always want
a cue to begin speaking immediately.
(WebCore::HTMLMediaElement::pauseSpeakingCueText): Support completing an extended description.
(WebCore::HTMLMediaElement::resumeSpeakingCueText): Ditto.
(WebCore::HTMLMediaElement::pausePlaybackForExtendedTextDescription): New, pause playback
to allow an extended description to complete.
(WebCore::HTMLMediaElement::speechSynthesis): Have the media element own the synthesizer
since the logic for managing it is here.
(WebCore::HTMLMediaElement::executeCueEnterOrExitActionForTime): Support extended descriptions.
(WebCore::HTMLMediaElement::addTextTrack): Set m_userPrefersExtendedDescriptions.
(WebCore::HTMLMediaElement::configureTextTrackGroup): Use m_userPrefersTextDescriptions.
(WebCore::HTMLMediaElement::updatePlayState): Pause and resume speaking here, not in `playPlayer`
and `pausePlayer`.
(WebCore::HTMLMediaElement::playPlayer): Ditto.
(WebCore::HTMLMediaElement::pausePlayer): Ditto.
(WebCore::HTMLMediaElement::captionPreferencesChanged): set m_userPrefersExtendedDescriptions.
(WebCore::HTMLMediaElement::executeCueEnterOrLeaveAction): Deleted.
* Source/WebCore/html/HTMLMediaElement.h:
(WTF::LogArgument<WebCore::HTMLMediaElement::SpeechSynthesisState>::toString):

* Source/WebCore/html/track/TextTrack.cpp:
(WebCore::TextTrack::speechSynthesis): Deleted.
* Source/WebCore/html/track/TextTrack.h:

* Source/WebCore/html/track/TextTrackCue.h:
(WebCore::TextTrackCue::prepareToSpeak): Renamed from `speak`.
(WebCore::TextTrackCue::beginSpeaking): Added.
(WebCore::TextTrackCue::pauseSpeaking): Added.
(WebCore::TextTrackCue::cancelSpeaking): Added.
(WebCore::TextTrackCue::speak): Deleted.

* Source/WebCore/html/track/VTTCue.cpp:
(WebCore::mapVideoRateToSpeechRate): Convert playback rate to web speech rate.
(WebCore::VTTCue::prepareToSpeak): Call completion handler when there is nothing to speak
or when the track is null. Stash `speechSynthesis` for future use.
(WebCore::VTTCue::beginSpeaking): Begin or resume speaking.
(WebCore::VTTCue::pauseSpeaking):
(WebCore::VTTCue::cancelSpeaking):
(WebCore::VTTCue::speak): Deleted.
* Source/WebCore/html/track/VTTCue.h:

* Source/WebCore/page/CaptionUserPreferences.cpp:
(WebCore::CaptionUserPreferences::userPrefersTextDescriptions const): Consider setting for
extended descriptions.
* Source/WebCore/platform/PlatformSpeechSynthesizer.h: Make refcounted.

* Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm:
(-[WebSpeechSynthesisWrapper speakUtterance:]): `client()` is a ref, not a pointer.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didStartSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didFinishSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didPauseSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didContinueSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didCancelSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:willSpeakRangeOfSpeechString:utterance:]): Ditto.
(WebCore::PlatformSpeechSynthesizer::create): Client can never be null so take ref, not a pointer.
(WebCore::PlatformSpeechSynthesizer::PlatformSpeechSynthesizer):

* Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.cpp:
(WebCore::PlatformSpeechSynthesizerMock::create): Client can never be null so take ref, not a pointer.
(WebCore::PlatformSpeechSynthesizerMock::PlatformSpeechSynthesizerMock):
(WebCore::PlatformSpeechSynthesizerMock::speakingFinished): `client()` is a ref, not a pointer.
(WebCore::PlatformSpeechSynthesizerMock::speak): Ditto. Use configurable utterance duration.
(WebCore::PlatformSpeechSynthesizerMock::cancel): Ditto.
(WebCore::PlatformSpeechSynthesizerMock::pause): Ditto. Null-check `m_utterance`.
(WebCore::PlatformSpeechSynthesizerMock::resume): Ditto.
* Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.h:
(WebCore::PlatformSpeechSynthesizerMock::setUtteranceDuration):

* Source/WebCore/testing/Internals.cpp:
(WebCore::Internals::enableMockSpeechSynthesizer):
(WebCore::Internals::enableMockSpeechSynthesizerForMediaElement):
(WebCore::Internals::setSpeechUtteranceDuration):
* Source/WebCore/testing/Internals.h:
* Source/WebCore/testing/Internals.idl:

* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::resetSpeechSynthesizer): PlatformSpeechSynthesizer is a Ref, not
a unique_ptr.
(WebKit::WebPageProxy::speechSynthesisData): Ditto.
* Source/WebKit/UIProcess/WebPageProxy.h:

Canonical link: https://commits.webkit.org/254502@main

ebfd896

Misc iOS, tvOS & watchOS macOS Linux Windows
❌ πŸ§ͺ style βœ… πŸ›  ios βœ… πŸ›  mac βœ… πŸ›  wpe βœ… πŸ›  πŸ§ͺ win
βœ… πŸ§ͺ bindings βœ… πŸ›  ios-sim βœ… πŸ›  mac-debug βœ… πŸ›  gtk βœ… πŸ›  wincairo
βœ… πŸ§ͺ webkitperl βœ… πŸ§ͺ ios-wk2 βœ… πŸ›  mac-AS-debug βœ… πŸ§ͺ gtk-wk2
βœ… πŸ§ͺ api-ios βœ… πŸ§ͺ api-mac βœ… πŸ§ͺ api-gtk
βœ… πŸ›  πŸ§ͺ jsc βœ… πŸ›  tv βœ… πŸ§ͺ mac-wk1 βœ… πŸ›  jsc-armv7
βœ… πŸ›  tv-sim βœ… πŸ§ͺ mac-wk2 βœ… πŸ§ͺ jsc-armv7-tests
βœ… πŸ›  πŸ§ͺ merge βœ… πŸ›  watch βœ… πŸ§ͺ mac-AS-debug-wk2 βœ… πŸ›  jsc-mips
βœ… πŸ›  watch-sim βœ… πŸ§ͺ mac-wk2-stress βœ… πŸ§ͺ jsc-mips-tests

@eric-carlson eric-carlson self-assigned this Sep 8, 2022
@eric-carlson eric-carlson added Media Bugs related to the HTML 5 Media elements. WebKit Nightly Build labels Sep 8, 2022
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Sep 8, 2022
@eric-carlson eric-carlson removed the merging-blocked Applied to prevent a change from being merged label Sep 8, 2022
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Sep 9, 2022
@eric-carlson eric-carlson removed the merging-blocked Applied to prevent a change from being merged label Sep 12, 2022
@@ -74,9 +74,9 @@ SpeechSynthesis::SpeechSynthesis(ScriptExecutionContext& context)

SpeechSynthesis::~SpeechSynthesis() = default;

void SpeechSynthesis::setPlatformSynthesizer(std::unique_ptr<PlatformSpeechSynthesizer> synthesizer)
void SpeechSynthesis::setPlatformSynthesizer(Ref<PlatformSpeechSynthesizer> synthesizer)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should take a Ref<PlatformSpeechSynthesizer>&&.

{
m_platformSpeechSynthesizer = WTFMove(synthesizer);
m_platformSpeechSynthesizer = synthesizer.ptr();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...and then you can get rid of this.

@eric-carlson eric-carlson added the merge-queue Applied to send a pull request to merge-queue label Sep 15, 2022
https://bugs.webkit.org/show_bug.cgi?id=244931
<rdar://99697422>

Reviewed by Jer Noble.

* LayoutTests/media/track/captions-webvtt/captions-extended-descriptions.vtt: Added.
* LayoutTests/media/track/track-description-cue.html:
* LayoutTests/media/track/track-extended-descriptions-expected.txt: Added.
* LayoutTests/media/track/track-extended-descriptions.html: Added.
* LayoutTests/TestExpectations: Feature is Cocoa-specific so far, skip test globally.
* LayoutTests/platform/ios/TestExpectations: Enable test on iOS.
* LayoutTests/platform/mac/TestExpectations: Enable test on macOS.

* Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml: Add 'ExtendedAudioDescriptionsEnabled'
setting.

* Source/WebCore/Modules/speech/SpeechSynthesis.cpp:
(WebCore::SpeechSynthesis::setPlatformSynthesizer): Take a Ref<> synthesizer instead of a
unique_ptr.
(WebCore::SpeechSynthesis::ensurePlatformSpeechSynthesizer): Use PlatformSpeechSynthesizer::create.
* Source/WebCore/Modules/speech/SpeechSynthesis.h:

* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::convertEnumerationToString): Added.
(WebCore::HTMLMediaElement::unregisterWithDocument): Cancel speech, clear synthesize.
(WebCore::HTMLMediaElement::updateActiveTextTrackCues): Cleanup for readability.
(WebCore::HTMLMediaElement::setSpeechSynthesisState): Update for extended descriptions.
(WebCore::HTMLMediaElement::speakCueText): Call cue.prepareToSpeak as we don't always want
a cue to begin speaking immediately.
(WebCore::HTMLMediaElement::pauseSpeakingCueText): Support completing an extended description.
(WebCore::HTMLMediaElement::resumeSpeakingCueText): Ditto.
(WebCore::HTMLMediaElement::pausePlaybackForExtendedTextDescription): New, pause playback
to allow an extended description to complete.
(WebCore::HTMLMediaElement::speechSynthesis): Have the media element own the synthesizer
since the logic for managing it is here.
(WebCore::HTMLMediaElement::executeCueEnterOrExitActionForTime): Support extended descriptions.
(WebCore::HTMLMediaElement::addTextTrack): Set m_userPrefersExtendedDescriptions.
(WebCore::HTMLMediaElement::configureTextTrackGroup): Use m_userPrefersTextDescriptions.
(WebCore::HTMLMediaElement::updatePlayState): Pause and resume speaking here, not in `playPlayer`
and `pausePlayer`.
(WebCore::HTMLMediaElement::playPlayer): Ditto.
(WebCore::HTMLMediaElement::pausePlayer): Ditto.
(WebCore::HTMLMediaElement::captionPreferencesChanged): set m_userPrefersExtendedDescriptions.
(WebCore::HTMLMediaElement::executeCueEnterOrLeaveAction): Deleted.
* Source/WebCore/html/HTMLMediaElement.h:
(WTF::LogArgument<WebCore::HTMLMediaElement::SpeechSynthesisState>::toString):

* Source/WebCore/html/track/TextTrack.cpp:
(WebCore::TextTrack::speechSynthesis): Deleted.
* Source/WebCore/html/track/TextTrack.h:

* Source/WebCore/html/track/TextTrackCue.h:
(WebCore::TextTrackCue::prepareToSpeak): Renamed from `speak`.
(WebCore::TextTrackCue::beginSpeaking): Added.
(WebCore::TextTrackCue::pauseSpeaking): Added.
(WebCore::TextTrackCue::cancelSpeaking): Added.
(WebCore::TextTrackCue::speak): Deleted.

* Source/WebCore/html/track/VTTCue.cpp:
(WebCore::mapVideoRateToSpeechRate): Convert playback rate to web speech rate.
(WebCore::VTTCue::prepareToSpeak): Call completion handler when there is nothing to speak
or when the track is null. Stash `speechSynthesis` for future use.
(WebCore::VTTCue::beginSpeaking): Begin or resume speaking.
(WebCore::VTTCue::pauseSpeaking):
(WebCore::VTTCue::cancelSpeaking):
(WebCore::VTTCue::speak): Deleted.
* Source/WebCore/html/track/VTTCue.h:

* Source/WebCore/page/CaptionUserPreferences.cpp:
(WebCore::CaptionUserPreferences::userPrefersTextDescriptions const): Consider setting for
extended descriptions.
* Source/WebCore/platform/PlatformSpeechSynthesizer.h: Make refcounted.

* Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm:
(-[WebSpeechSynthesisWrapper speakUtterance:]): `client()` is a ref, not a pointer.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didStartSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didFinishSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didPauseSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didContinueSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didCancelSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:willSpeakRangeOfSpeechString:utterance:]): Ditto.
(WebCore::PlatformSpeechSynthesizer::create): Client can never be null so take ref, not a pointer.
(WebCore::PlatformSpeechSynthesizer::PlatformSpeechSynthesizer):

* Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.cpp:
(WebCore::PlatformSpeechSynthesizerMock::create): Client can never be null so take ref, not a pointer.
(WebCore::PlatformSpeechSynthesizerMock::PlatformSpeechSynthesizerMock):
(WebCore::PlatformSpeechSynthesizerMock::speakingFinished): `client()` is a ref, not a pointer.
(WebCore::PlatformSpeechSynthesizerMock::speak): Ditto. Use configurable utterance duration.
(WebCore::PlatformSpeechSynthesizerMock::cancel): Ditto.
(WebCore::PlatformSpeechSynthesizerMock::pause): Ditto. Null-check `m_utterance`.
(WebCore::PlatformSpeechSynthesizerMock::resume): Ditto.
* Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.h:
(WebCore::PlatformSpeechSynthesizerMock::setUtteranceDuration):

* Source/WebCore/testing/Internals.cpp:
(WebCore::Internals::enableMockSpeechSynthesizer):
(WebCore::Internals::enableMockSpeechSynthesizerForMediaElement):
(WebCore::Internals::setSpeechUtteranceDuration):
* Source/WebCore/testing/Internals.h:
* Source/WebCore/testing/Internals.idl:

* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::resetSpeechSynthesizer): PlatformSpeechSynthesizer is a Ref, not
a unique_ptr.
(WebKit::WebPageProxy::speechSynthesisData): Ditto.
* Source/WebKit/UIProcess/WebPageProxy.h:

Canonical link: https://commits.webkit.org/254502@main
@webkit-commit-queue
Copy link
Collaborator

Committed 254502@main (c2f7594): https://commits.webkit.org/254502@main

Reviewed commits have been landed. Closing PR #4129 and removing active labels.

@webkit-early-warning-system webkit-early-warning-system merged commit c2f7594 into WebKit:main Sep 15, 2022
@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Sep 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Media Bugs related to the HTML 5 Media elements.
Projects
None yet
5 participants