Refactor logic for muting autoplaying video elements in background tabs

https://bugs.webkit.org/show_bug.cgi?id=273633
rdar://125859510

Reviewed by Aditya Keerthi.

Refactor some logic related to muting media elements. See below for more details.

* LayoutTests/fast/element-targeting/target-video-in-subframe-expected.txt: Added.
* LayoutTests/fast/element-targeting/target-video-in-subframe.html: Added.
* LayoutTests/resources/ui-helper.js:
(window.UIHelper.resetVisibilityAdjustments):
(window.UIHelper):

Add a new layout test (and testing hooks) to exercise the change.

* Source/WebCore/dom/Document.cpp:
(WebCore::Document::visibilityAdjustmentStateDidChange):

Add plumbing to propagate visibility state change to all audio producers.

* Source/WebCore/dom/Document.h:
* Source/WebCore/dom/Element.cpp:
(WebCore::Element::setVisibilityAdjustment):

Set a flag on `Page` to indicate that we've applied visibility adjustment to at least one element
before. Used in `isInVisibilityAdjustmentSubtree()` below to exit right away, in the common case
where there has never been visibility adjustment.

(WebCore::Element::isInVisibilityAdjustmentSubtree const):

Add a helper method to compute whether or not the element is inside of a visibility adjustment
subtree, by traversing ancestors in the DOM.

* Source/WebCore/dom/Element.h:
* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::didFinishInsertingNode):
(WebCore::HTMLMediaElement::removedFromAncestor):
(WebCore::HTMLMediaElement::visibilityAdjustmentStateDidChange):

Recompute and cache the visibility adjustment subtree state whenever the media element is
unparented, re-parented, or visibility state otherwise changes.

(WebCore::HTMLMediaElement::effectiveMuted const):

Consult the cached state flag that's updated above when determining whether the media element should
be `effectivelyMuted()`.

* Source/WebCore/html/HTMLMediaElement.h:
* Source/WebCore/page/ElementTargetingController.cpp:
(WebCore::hasAudibleMedia):
(WebCore::targetedElementInfo):

Add support for a `hasAudibleMedia` flag on targeted element info.

(WebCore::findOnlyMainElement):
(WebCore::isNavigationalElement):
(WebCore::containsNavigationalElement):

Drive-by fix: mark the arguments to a few helper functions as `const`.

(WebCore::ElementTargetingController::adjustVisibility):
(WebCore::ElementTargetingController::adjustVisibilityInRepeatedlyTargetedRegions):
(WebCore::ElementTargetingController::resetVisibilityAdjustments):
(WebCore::ElementTargetingController::dispatchVisibilityAdjustmentStateDidChange):
* Source/WebCore/page/ElementTargetingController.h:
* Source/WebCore/page/ElementTargetingTypes.h:
* Source/WebCore/page/MediaProducer.h:
(WebCore::MediaProducer::visibilityAdjustmentStateDidChange):

Add a new delegate hook to inform media producers when visibility state changes. Currently, this is
only implemented by `HTMLMediaElement`, which allows it to adjust whether or not the underlying
media player should be muted, through `effectiveMuted()`.

* Source/WebCore/page/Page.cpp:
(WebCore::Page::didCommitLoad):
* Source/WebCore/page/Page.h:

Add a boolean flag to track whether or not there has ever been any visibility adjustment in the
page (this bit is cleared out in `didCommitLoad` above).

(WebCore::Page::hasEverSetVisibilityAdjustment const):
(WebCore::Page::didSetVisibilityAdjustment):
* Source/WebCore/testing/Internals.cpp:
(WebCore::Internals::isEffectivelyMuted):
* Source/WebCore/testing/Internals.h:
* Source/WebCore/testing/Internals.idl:

Add a new testing-only hook to ask for `HTMLMediaElement::effectiveMuted()`.

* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/UIProcess/API/APITargetedElementInfo.h:
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.h:
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.mm:
(-[_WKTargetedElementInfo hasAudibleMedia]):
* Tools/TestRunnerShared/UIScriptContext/Bindings/UIScriptController.idl:
* Tools/TestRunnerShared/UIScriptContext/UIScriptController.h:
(WTR::UIScriptController::resetVisibilityAdjustments):
* Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.h:
* Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.mm:
(WTR::UIScriptControllerCocoa::resetVisibilityAdjustments):

Canonical link: https://commits.webkit.org/278356@main

LayoutTests/fast/element-targeting/target-video-in-subframe-expected.txt

@@ -0,0 +1,14 @@
+PASS originalVideo.muted is false
+PASS originalVideo.volume is 1
+PASS internals.isEffectivelyMuted(originalVideo) is true
+PASS clonedVideo.muted is false
+PASS clonedVideo.volume is 1
+PASS internals.isEffectivelyMuted(clonedVideo) is true
+After reset:
+PASS internals.isEffectivelyMuted(originalVideo) is false
+PASS internals.isEffectivelyMuted(clonedVideo) is false
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
+This test requires WebKitTestRunner

LayoutTests/fast/element-targeting/target-video-in-subframe.html

@@ -0,0 +1,84 @@
+<!DOCTYPE html> <!-- webkit-test-runner [ useFlexibleViewport=true ] -->
+<html>
+<head>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<script src="../../resources/ui-helper.js"></script>
+<script src="../../resources/js-test.js"></script>
+<style>
+body, html {
+    margin: 0;
+}
+
+div.container {
+    width: 320px;
+    height: 240px;
+    border: 1px solid gray;
+    box-sizing: border-box;
+    text-align: center;
+}
+
+iframe {
+    width: 100%;
+    height: 100%;
+}
+</style>
+</head>
+<body>
+    <div class="container">
+        <iframe></iframe>
+    </div>
+    <div>This test requires WebKitTestRunner</div>
+</body>
+<script>
+jsTestIsAsync = true;
+
+addEventListener("load", async event => {
+    const frame = document.querySelector("iframe");
+    await new Promise(resolve => {
+        frame.addEventListener("load", resolve, { once: true });
+        frame.srcdoc = `
+            <head>
+                <style>
+                    video {
+                        width: 160px;
+                        height: 120px;
+                    }
+                </style>
+            </head>
+            <body>
+                <video src="../../media/content/audio-describes-video.mp4" loop autoplay />
+                <script>
+                    function cloneVideo() {
+                        const originalVideo = document.querySelector("video");
+                        const clonedVideo = document.createElement("video");
+                        clonedVideo.src = originalVideo.src;
+                        clonedVideo.autoplay = true;
+                        clonedVideo.loop = true;
+                        document.body.appendChild(clonedVideo);
+                        return [originalVideo, clonedVideo];
+                    }
+                </` + `script>
+            </body>`;
+    });
+
+    await UIHelper.adjustVisibilityForFrontmostTarget(100, 100);
+
+    [originalVideo, clonedVideo] = frame.contentWindow.cloneVideo();
+
+    shouldBeFalse("originalVideo.muted");
+    shouldBe("originalVideo.volume", "1");
+    shouldBeTrue("internals.isEffectivelyMuted(originalVideo)");
+
+    shouldBeFalse("clonedVideo.muted");
+    shouldBe("clonedVideo.volume", "1");
+    shouldBeTrue("internals.isEffectivelyMuted(clonedVideo)");
+
+    await UIHelper.resetVisibilityAdjustments();
+
+    debug("After reset:");
+    shouldBeFalse("internals.isEffectivelyMuted(originalVideo)");
+    shouldBeFalse("internals.isEffectivelyMuted(clonedVideo)");
+    finishJSTest();
+});
+</script>
+</html>

LayoutTests/resources/ui-helper.js

@@ -2184,6 +2184,15 @@ window.UIHelper = class UIHelper {
             })()`, resolve);
         });
     }
+
+    static resetVisibilityAdjustments() {
+        if (!this.isWebKit2())
+            return Promise.resolve();
+
+        return new Promise(resolve => {
+            testRunner.runUIScript("uiController.resetVisibilityAdjustments(result => uiController.uiScriptComplete(result));", resolve);
+        });
+    }
 }
 
 UIHelper.EventStreamBuilder = class {

Source/WebCore/dom/Document.cpp

@@ -5253,6 +5253,12 @@ void Document::updateIsPlayingMedia()
 #endif
 }
 
+void Document::visibilityAdjustmentStateDidChange()
+{
+    for (auto& audioProducer : m_audioProducers)
+        audioProducer.visibilityAdjustmentStateDidChange();
+}
+
 void Document::pageMutedStateDidChange()
 {
     for (auto& audioProducer : m_audioProducers)

Source/WebCore/dom/Document.h

@@ -1623,6 +1623,7 @@ class Document
     inline bool isCapturing() const;
     WEBCORE_EXPORT void updateIsPlayingMedia();
     void pageMutedStateDidChange();
+    void visibilityAdjustmentStateDidChange();
 
     bool hasEverHadSelectionInsideTextFormControl() const { return m_hasEverHadSelectionInsideTextFormControl; }
     void setHasEverHadSelectionInsideTextFormControl() { m_hasEverHadSelectionInsideTextFormControl = true; }

Source/WebCore/dom/Element.cpp

@@ -70,6 +70,7 @@
 #include "HTMLCanvasElement.h"
 #include "HTMLDialogElement.h"
 #include "HTMLDocument.h"
+#include "HTMLFrameOwnerElement.h"
 #include "HTMLHtmlElement.h"
 #include "HTMLImageElement.h"
 #include "HTMLInputElement.h"
@@ -5653,6 +5654,40 @@ OptionSet<VisibilityAdjustment> Element::visibilityAdjustment() const
 void Element::setVisibilityAdjustment(OptionSet<VisibilityAdjustment> adjustment)
 {
     ensureElementRareData().setVisibilityAdjustment(adjustment);
+
+    if (!adjustment)
+        return;
+
+    if (RefPtr page = document().page())
+        page->didSetVisibilityAdjustment();
+}
+
+bool Element::isInVisibilityAdjustmentSubtree() const
+{
+    RefPtr page = document().page();
+    if (!page)
+        return false;
+
+    if (!page->hasEverSetVisibilityAdjustment())
+        return false;
+
+    auto lineageIsInAdjustmentSubtree = [this] {
+        for (auto& element : lineageOfType<Element>(*this)) {
+            if (element.visibilityAdjustment().contains(VisibilityAdjustment::Subtree))
+                return true;
+        }
+        return false;
+    };
+
+    if (RefPtr owner = document().ownerElement()) {
+        if (owner->isInVisibilityAdjustmentSubtree())
+            return true;
+
+        ASSERT(!lineageIsInAdjustmentSubtree());
+        return false;
+    }
+
+    return lineageIsInAdjustmentSubtree();
 }
 
 TextStream& operator<<(TextStream& ts, ContentRelevancy relevancy)

Source/WebCore/dom/Element.h

@@ -642,6 +642,7 @@ class Element : public ContainerNode {
 
     OptionSet<VisibilityAdjustment> visibilityAdjustment() const;
     void setVisibilityAdjustment(OptionSet<VisibilityAdjustment>);
+    bool isInVisibilityAdjustmentSubtree() const;
 
     bool isSpellCheckingEnabled() const;
     WEBCORE_EXPORT bool isWritingSuggestionsEnabled() const;

Source/WebCore/html/HTMLMediaElement.cpp

@@ -131,6 +131,7 @@
 #include "VideoTrackConfiguration.h"
 #include "VideoTrackList.h"
 #include "VideoTrackPrivate.h"
+#include "VisibilityAdjustment.h"
 #include "WebCoreJSClientData.h"
 #include <JavaScriptCore/Uint8Array.h>
 #include <limits>
@@ -963,6 +964,8 @@ void HTMLMediaElement::didFinishInsertingNode()
     if (m_inActiveDocument && m_networkState == NETWORK_EMPTY && !attributeWithoutSynchronization(srcAttr).isEmpty())
         prepareForLoad();
 
+    visibilityAdjustmentStateDidChange();
+
     if (!m_explicitlyMuted) {
         m_explicitlyMuted = true;
         m_muted = hasAttributeWithoutSynchronization(mutedAttr);
@@ -1048,6 +1051,8 @@ void HTMLMediaElement::removedFromAncestor(RemovalType removalType, ContainerNod
         m_mediaSession->clientCharacteristicsChanged(false);
 
     HTMLElement::removedFromAncestor(removalType, oldParentOfRemovedTree);
+
+    visibilityAdjustmentStateDidChange();
 }
 
 void HTMLMediaElement::willAttachRenderers()
@@ -9009,6 +9014,25 @@ void HTMLMediaElement::setAutoplayEventPlaybackState(AutoplayEventPlaybackState
     }
 }
 
+void HTMLMediaElement::visibilityAdjustmentStateDidChange()
+{
+    auto currentValue = isInVisibilityAdjustmentSubtree();
+    if (m_cachedIsInVisibilityAdjustmentSubtree == currentValue)
+        return;
+
+    bool wasMuted = effectiveMuted();
+    m_cachedIsInVisibilityAdjustmentSubtree = currentValue;
+    bool muted = effectiveMuted();
+    if (wasMuted == muted)
+        return;
+
+    RefPtr player = m_player;
+    if (!player)
+        return;
+
+    player->setMuted(muted);
+}
+
 void HTMLMediaElement::pageMutedStateDidChange()
 {
     if (RefPtr page = document().page()) {
@@ -9031,7 +9055,19 @@ double HTMLMediaElement::effectiveVolume() const
 
 bool HTMLMediaElement::effectiveMuted() const
 {
-    return muted() || (m_mediaController && m_mediaController->muted()) || (document().page() && document().page()->isAudioMuted());
+    if (muted())
+        return true;
+
+    if (m_mediaController && m_mediaController->muted())
+        return true;
+
+    if (RefPtr page = document().page(); page && page->isAudioMuted())
+        return true;
+
+    if (m_cachedIsInVisibilityAdjustmentSubtree)
+        return true;
+
+    return false;
 }
 
 bool HTMLMediaElement::doesHaveAttribute(const AtomString& attribute, AtomString* value) const

Source/WebCore/html/HTMLMediaElement.h

@@ -599,6 +599,7 @@ class HTMLMediaElement
 
     bool willLog(WTFLogLevel) const;
 
+    bool isAudible() const final { return canProduceAudio(); }
     bool isSuspended() const final;
 
     WEBCORE_EXPORT void didBecomeFullscreenElement() final;
@@ -998,7 +999,6 @@ class HTMLMediaElement
     bool shouldOverrideBackgroundPlaybackRestriction(PlatformMediaSession::InterruptionType) const override;
     bool shouldOverrideBackgroundLoadingRestriction() const override;
     bool canProduceAudio() const final;
-    bool isAudible() const final { return canProduceAudio(); };
     bool isEnded() const final { return ended(); }
     MediaTime mediaSessionDuration() const final;
     bool hasMediaStreamSource() const final;
@@ -1008,6 +1008,7 @@ class HTMLMediaElement
     std::optional<NowPlayingInfo> nowPlayingInfo() const final;
     WeakPtr<PlatformMediaSession> selectBestMediaSession(const Vector<WeakPtr<PlatformMediaSession>>&, PlatformMediaSession::PlaybackControlsPurpose) final;
 
+    void visibilityAdjustmentStateDidChange() final;
     void pageMutedStateDidChange() override;
 
 #if USE(AUDIO_SESSION) && PLATFORM(MAC)
@@ -1019,7 +1020,7 @@ class HTMLMediaElement
 
     bool processingUserGestureForMedia() const;
 
-    bool effectiveMuted() const;
+    WEBCORE_EXPORT bool effectiveMuted() const;
     double effectiveVolume() const;
 
     void registerWithDocument(Document&);
@@ -1255,6 +1256,7 @@ class HTMLMediaElement
     bool m_shouldAudioPlaybackRequireUserGesture : 1;
     bool m_shouldVideoPlaybackRequireUserGesture : 1;
     bool m_volumeLocked : 1;
+    bool m_cachedIsInVisibilityAdjustmentSubtree : 1 { false };
 
     enum class ControlsState : uint8_t { None, Initializing, Ready, PartiallyDeinitialized };
     friend String convertEnumerationToString(HTMLMediaElement::ControlsState enumerationValue);