Skip to content
Permalink
Browse files
Media continues loading after rendered invisible (removed from DOM; s…
…crolled off screen)

https://bugs.webkit.org/show_bug.cgi?id=185487

Reviewed by Eric Carlson.

Source/WebCore:

Test: media/video-buffering-allowed.html

When a media element is removed from the dom (e.g. through innerHTML=""), it doesn't
necessarily stop loading media data; it will continue to do so until its destructor is
called through garbage collection. Similarly, when a media element is rendered not-visible
by being scrolled off-screen or being made display:none, media loading continues. There
are legitimate use cases for out-of-DOM media loading, so only temporarily block loading
when the element transitions out of the document. Similarly, only block loading for non-visible
media elements when returning from the "page is hidden" state, and only until the media
element is asked to play or is otherwise made visible.

Note: this refactors a lot of code out of PlatformMediaSession and into MediaElementSession,
since this code is specific to "media elements".

* html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::HTMLMediaElement):
(WebCore::HTMLMediaElement::insertedIntoAncestor):
(WebCore::HTMLMediaElement::removedFromAncestor):
(WebCore::HTMLMediaElement::playInternal):
(WebCore::HTMLMediaElement::stopWithoutDestroyingMediaPlayer):
(WebCore::HTMLMediaElement::resume):
(WebCore::HTMLMediaElement::visibilityStateChanged):
(WebCore::HTMLMediaElement::createMediaPlayer):
(WebCore::HTMLMediaElement::setShouldBufferData):
(WebCore::HTMLMediaElement::purgeBufferedDataIfPossible):
(WebCore::HTMLMediaElement::isVisibleInViewportChanged):
(WebCore::HTMLMediaElement::fullscreenModeChanged):
(WebCore::HTMLMediaElement::setInActiveDocument):
* html/HTMLMediaElement.h:
(WebCore::HTMLMediaElement::shouldBufferData const):
(WebCore::HTMLMediaElement::elementIsHidden const):
* html/MediaElementSession.cpp:
(WebCore::MediaElementSession::MediaElementSession):
(WebCore::MediaElementSession::clientWillBeginAutoplaying):
(WebCore::MediaElementSession::clientWillBeginPlayback):
(WebCore::MediaElementSession::clientWillPausePlayback):
(WebCore::MediaElementSession::visibilityChanged):
(WebCore::MediaElementSession::isVisibleInViewportChanged):
(WebCore::MediaElementSession::inActiveDocumentChanged):
(WebCore::MediaElementSession::scheduleClientDataBufferingCheck):
(WebCore::MediaElementSession::clientDataBufferingTimerFired):
(WebCore::MediaElementSession::updateClientDataBuffering):
(WebCore::MediaElementSession::dataBufferingPermitted const):
(WebCore::MediaElementSession::wantsToObserveViewportVisibilityForAutoplay const):
* html/MediaElementSession.h:
* platform/audio/PlatformMediaSession.cpp:
(WebCore::PlatformMediaSession::PlatformMediaSession):
(WebCore::PlatformMediaSession::clientWillBeginAutoplaying):
(WebCore::PlatformMediaSession::clientWillBeginPlayback):
(WebCore::PlatformMediaSession::clientWillPausePlayback):
(): Deleted.
(WebCore::PlatformMediaSession::visibilityChanged): Deleted.
(WebCore::PlatformMediaSession::scheduleClientDataBufferingCheck): Deleted.
(WebCore::PlatformMediaSession::clientDataBufferingTimerFired): Deleted.
(WebCore::PlatformMediaSession::updateClientDataBuffering): Deleted.
(WebCore::PlatformMediaSession::isHidden const): Deleted.
* platform/audio/PlatformMediaSession.h:
(WebCore::PlatformMediaSessionClient::setShouldBufferData): Deleted.
(WebCore::PlatformMediaSessionClient::elementIsHidden const): Deleted.
* platform/audio/PlatformMediaSessionManager.cpp:
(WebCore::PlatformMediaSessionManager::sessionCanLoadMedia const): Deleted.
* platform/audio/PlatformMediaSessionManager.h:
* platform/audio/ios/MediaSessionManagerIOS.h:
* platform/audio/ios/MediaSessionManagerIOS.mm:
(WebCore::MediaSessionManageriOS::sessionCanLoadMedia const): Deleted.
* rendering/RenderVideo.cpp:
(WebCore::RenderVideo::willBeDestroyed):
* testing/Internals.cpp:
(WebCore::Internals::elementShouldBufferData):
* testing/Internals.h:
* testing/Internals.idl:

LayoutTests:

* media/video-buffering-allowed-expected.txt: Added.
* media/video-buffering-allowed.html: Added.
* media/video-test.js:
(compare):
(testExpected):
(sleepFor):
(testArraysEqual): Deleted.


Canonical link: https://commits.webkit.org/201128@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@231817 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
jernoble committed May 15, 2018
1 parent 8d9f3f1 commit 8e0916aa35b39eed39b87f40bb9467811f750792
Show file tree
Hide file tree
Showing 20 changed files with 436 additions and 131 deletions.
@@ -1,3 +1,18 @@
2018-05-15 Jer Noble <jer.noble@apple.com>

Media continues loading after rendered invisible (removed from DOM; scrolled off screen)
https://bugs.webkit.org/show_bug.cgi?id=185487

Reviewed by Eric Carlson.

* media/video-buffering-allowed-expected.txt: Added.
* media/video-buffering-allowed.html: Added.
* media/video-test.js:
(compare):
(testExpected):
(sleepFor):
(testArraysEqual): Deleted.

2018-05-15 Charles Vazac <cvazac@gmail.com>

Add the PerformanceServerTiming Interface which makes Server-Timing header timing values available to JavaScript running in the browser.
@@ -1,48 +1,48 @@
CONSOLE MESSAGE: line 176: The language 'a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language '1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language '1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language '1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language '1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language '1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language '1a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language '1a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language '1a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language '1a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language '-a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language '1a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language '-a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language '-a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language '-a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'a-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language '-a' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'a-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'a-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'a-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'a1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'a-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'a1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'a1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'a1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'aa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'a1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'aa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'aa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'aa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'aaaa' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'aa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'aaaa' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'aaaa' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'aaaa' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'aaa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'aaaa' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'aaa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'aaa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'aaa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'aaa1' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 106: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 176: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
CONSOLE MESSAGE: line 221: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
Test that only BCP47 language tags are accepted as valid but still reflected.


@@ -0,0 +1,32 @@

RUN(video.src = findMediaFile("video", "content/test"))
EVENT(canplaythrough)
EXPECTED (internals.elementShouldBufferData(video) == 'true') OK
* Remove the video element from the document.
RUN(video.parentNode.removeChild(video))
EXPECTED (internals.elementShouldBufferData(video) == 'false') OK
* Play the video.
RUN(video.play())
EVENT(playing)
EXPECTED (internals.elementShouldBufferData(video) == 'true') OK
* Pause the video.
RUN(video.pause())
EVENT(pause)
EXPECTED (internals.elementShouldBufferData(video) == 'true') OK
* Re-insert the video element into the document.
RUN(document.body.insertBefore(video, document.body.firstChild))
EXPECTED (internals.elementShouldBufferData(video) == 'true') OK
* display:none the video element.
RUN(video.style.display = "none")
EXPECTED (internals.elementShouldBufferData(video) == 'true') OK
* Simulate the view becoming invisible.
RUN(internals.setPageVisibility(false))
EXPECTED (internals.elementShouldBufferData(video) == 'false') OK
* Simulate the view becoming visible.
RUN(internals.setPageVisibility(true))
EXPECTED (internals.elementShouldBufferData(video) == 'false') OK
* Remove display:none from the video element.
RUN(video.style.removeProperty("display"))
EXPECTED (internals.elementShouldBufferData(video) == 'true') OK
END OF TEST

@@ -0,0 +1,58 @@
<!DOCTYPE html>
<html>
<head>
<title>video-buffering-allowed</title>
<script src=media-file.js></script>
<script src=video-test.js></script>
<script>
async function runTest() {
findMediaElement();
failTestIn(1000);

run('video.src = findMediaFile("video", "content/test")');
await waitFor(video, 'canplaythrough');
testExpected('internals.elementShouldBufferData(video)', true);

consoleWrite('* Remove the video element from the document.');
run('video.parentNode.removeChild(video)');
await testExpectedEventually('internals.elementShouldBufferData(video)', false);

consoleWrite('* Play the video.');
run ('video.play()');
await waitFor(video, 'playing');
testExpected('internals.elementShouldBufferData(video)', true);

consoleWrite('* Pause the video.');
run('video.pause()');
await waitFor(video, 'pause');
testExpected('internals.elementShouldBufferData(video)', true);

consoleWrite('* Re-insert the video element into the document.');
run('document.body.insertBefore(video, document.body.firstChild)');
await testExpectedEventually('internals.elementShouldBufferData(video)', true);

consoleWrite('* display:none the video element.');
run('video.style.display = "none"');
await testExpectedEventually('internals.elementShouldBufferData(video)', true);

consoleWrite('* Simulate the view becoming invisible.');
run('internals.setPageVisibility(false)');
await testExpectedEventually('internals.elementShouldBufferData(video)', false);

consoleWrite('* Simulate the view becoming visible.');
run('internals.setPageVisibility(true)');
await testExpectedEventually('internals.elementShouldBufferData(video)', false);

consoleWrite('* Remove display:none from the video element.');
run('video.style.removeProperty("display")');
await testExpectedEventually('internals.elementShouldBufferData(video)', true);

endTest();
}
window.addEventListener('load', runTest);
</script>
</head>
<body>
<video controls></video>
</body>
</html>
@@ -62,17 +62,9 @@ function test(testFuncString, endit)
endTest();
}

function testExpected(testFuncString, expected, comparison)
function compare(testFuncString, expected, comparison)
{
try {
var observed = eval(testFuncString);
} catch (ex) {
consoleWrite(ex);
return;
}

if (comparison === undefined)
comparison = '==';
var observed = eval(testFuncString);

var success = false;
switch (comparison)
@@ -87,7 +79,51 @@ function testExpected(testFuncString, expected, comparison)
case 'instanceof': success = observed instanceof expected; break;
}

reportExpected(success, testFuncString, comparison, expected, observed)
return {success:success, observed:observed};
}

function testExpected(testFuncString, expected, comparison)
{
if (comparison === undefined)
comparison = '==';

try {
let {success, observed} = compare(testFuncString, expected, comparison);
reportExpected(success, testFuncString, comparison, expected, observed)
} catch (ex) {
consoleWrite(ex);
}
}

function sleepFor(duration) {
return new Promise(resolve => {
setTimeout(resolve, duration);
});
}

function testExpectedEventually(testFuncString, expected, comparison)
{
return new Promise(async resolve => {
var success;
var observed;
if (comparison === undefined)
comparison = '==';
while (true) {
try {
let {success, observed} = compare(testFuncString, expected, comparison);
if (success) {
reportExpected(success, testFuncString, comparison, expected, observed);
resolve();
return;
}
await sleepFor(1);
} catch (ex) {
consoleWrite(ex);
resolve();
return;
}
}
});
}

function testArraysEqual(testFuncString, expected)
@@ -150,6 +186,15 @@ function run(testFuncString)
}
}

function waitFor(element, event) {
return new Promise(resolve => {
element.addEventListener(event, event => {
consoleWrite(`EVENT(${event.type})`);
resolve(event);
}, { once: true });
});
}

function waitForEventOnce(eventName, func, endit)
{
waitForEvent(eventName, func, endit, true)
@@ -1,3 +1,82 @@
2018-05-15 Jer Noble <jer.noble@apple.com>

Media continues loading after rendered invisible (removed from DOM; scrolled off screen)
https://bugs.webkit.org/show_bug.cgi?id=185487

Reviewed by Eric Carlson.

Test: media/video-buffering-allowed.html

When a media element is removed from the dom (e.g. through innerHTML=""), it doesn't
necessarily stop loading media data; it will continue to do so until its destructor is
called through garbage collection. Similarly, when a media element is rendered not-visible
by being scrolled off-screen or being made display:none, media loading continues. There
are legitimate use cases for out-of-DOM media loading, so only temporarily block loading
when the element transitions out of the document. Similarly, only block loading for non-visible
media elements when returning from the "page is hidden" state, and only until the media
element is asked to play or is otherwise made visible.

Note: this refactors a lot of code out of PlatformMediaSession and into MediaElementSession,
since this code is specific to "media elements".

* html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::HTMLMediaElement):
(WebCore::HTMLMediaElement::insertedIntoAncestor):
(WebCore::HTMLMediaElement::removedFromAncestor):
(WebCore::HTMLMediaElement::playInternal):
(WebCore::HTMLMediaElement::stopWithoutDestroyingMediaPlayer):
(WebCore::HTMLMediaElement::resume):
(WebCore::HTMLMediaElement::visibilityStateChanged):
(WebCore::HTMLMediaElement::createMediaPlayer):
(WebCore::HTMLMediaElement::setShouldBufferData):
(WebCore::HTMLMediaElement::purgeBufferedDataIfPossible):
(WebCore::HTMLMediaElement::isVisibleInViewportChanged):
(WebCore::HTMLMediaElement::fullscreenModeChanged):
(WebCore::HTMLMediaElement::setInActiveDocument):
* html/HTMLMediaElement.h:
(WebCore::HTMLMediaElement::shouldBufferData const):
(WebCore::HTMLMediaElement::elementIsHidden const):
* html/MediaElementSession.cpp:
(WebCore::MediaElementSession::MediaElementSession):
(WebCore::MediaElementSession::clientWillBeginAutoplaying):
(WebCore::MediaElementSession::clientWillBeginPlayback):
(WebCore::MediaElementSession::clientWillPausePlayback):
(WebCore::MediaElementSession::visibilityChanged):
(WebCore::MediaElementSession::isVisibleInViewportChanged):
(WebCore::MediaElementSession::inActiveDocumentChanged):
(WebCore::MediaElementSession::scheduleClientDataBufferingCheck):
(WebCore::MediaElementSession::clientDataBufferingTimerFired):
(WebCore::MediaElementSession::updateClientDataBuffering):
(WebCore::MediaElementSession::dataBufferingPermitted const):
(WebCore::MediaElementSession::wantsToObserveViewportVisibilityForAutoplay const):
* html/MediaElementSession.h:
* platform/audio/PlatformMediaSession.cpp:
(WebCore::PlatformMediaSession::PlatformMediaSession):
(WebCore::PlatformMediaSession::clientWillBeginAutoplaying):
(WebCore::PlatformMediaSession::clientWillBeginPlayback):
(WebCore::PlatformMediaSession::clientWillPausePlayback):
(): Deleted.
(WebCore::PlatformMediaSession::visibilityChanged): Deleted.
(WebCore::PlatformMediaSession::scheduleClientDataBufferingCheck): Deleted.
(WebCore::PlatformMediaSession::clientDataBufferingTimerFired): Deleted.
(WebCore::PlatformMediaSession::updateClientDataBuffering): Deleted.
(WebCore::PlatformMediaSession::isHidden const): Deleted.
* platform/audio/PlatformMediaSession.h:
(WebCore::PlatformMediaSessionClient::setShouldBufferData): Deleted.
(WebCore::PlatformMediaSessionClient::elementIsHidden const): Deleted.
* platform/audio/PlatformMediaSessionManager.cpp:
(WebCore::PlatformMediaSessionManager::sessionCanLoadMedia const): Deleted.
* platform/audio/PlatformMediaSessionManager.h:
* platform/audio/ios/MediaSessionManagerIOS.h:
* platform/audio/ios/MediaSessionManagerIOS.mm:
(WebCore::MediaSessionManageriOS::sessionCanLoadMedia const): Deleted.
* rendering/RenderVideo.cpp:
(WebCore::RenderVideo::willBeDestroyed):
* testing/Internals.cpp:
(WebCore::Internals::elementShouldBufferData):
* testing/Internals.h:
* testing/Internals.idl:

2018-05-15 Charles Vazac <cvazac@gmail.com>

Add the PerformanceServerTiming Interface which makes Server-Timing header timing values available to JavaScript running in the browser.

0 comments on commit 8e0916a

Please sign in to comment.