Skip to content

Commit

Permalink
Merge r228416 - Resource Load Statistics: Classify resources as preva…
Browse files Browse the repository at this point in the history
…lent based on redirects to other prevalent resources

https://bugs.webkit.org/show_bug.cgi?id=182664
<rdar://problem/37372572>

Reviewed by Brent Fulgham.

Source/WebCore:

Tests: http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-redirect-collusion.html
       http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion.html
       http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-unique-redirects-to.html

* loader/ResourceLoadObserver.cpp:
(WebCore::ResourceLoadObserver::logFrameNavigation):
    We now collect statistics on top frame navigational redirects too, both to and from.
(WebCore::ResourceLoadObserver::logSubresourceLoading):
    We now collect statistics on redirects from in addition to the existing redirects to.
* loader/ResourceLoadStatistics.cpp:
(WebCore::ResourceLoadStatistics::encode const):
(WebCore::ResourceLoadStatistics::decode):
(WebCore::ResourceLoadStatistics::toString const):
(WebCore::ResourceLoadStatistics::merge):
* loader/ResourceLoadStatistics.h:
    Three new fields:
    - topFrameUniqueRedirectsTo
    - topFrameUniqueRedirectsFrom
    - subresourceUniqueRedirectsFrom

Source/WebKit:

* Platform/classifier/ResourceLoadStatisticsClassifier.cpp:
(WebKit::ResourceLoadStatisticsClassifier::hasPrevalentResourceCharacteristics):
    Unique top frame redirects now counts toward classification as prevalent.
* Shared/WebCoreArgumentCoders.cpp:
(IPC::ArgumentCoder<ResourceLoadStatistics>::encode):
(IPC::ArgumentCoder<ResourceLoadStatistics>::decode):
* UIProcess/API/C/WKWebsiteDataStoreRef.cpp:
(WKWebsiteDataStoreSetStatisticsSubresourceUniqueRedirectFrom):
(WKWebsiteDataStoreSetStatisticsTopFrameUniqueRedirectTo):
(WKWebsiteDataStoreSetStatisticsTopFrameUniqueRedirectFrom):
    Test infrastructure.
* UIProcess/API/C/WKWebsiteDataStoreRef.h:
* UIProcess/WebResourceLoadStatisticsStore.cpp:
    Bumped the statistics model to 11 because of the added fields.
(WebKit::WebResourceLoadStatisticsStore::recursivelyGetAllDomainsThatHaveRedirectedToThisDomain):
    This function back traces redirect chains to classify domains
    that have redirected to a newly classified prevalent resource.
(WebKit::WebResourceLoadStatisticsStore::processStatisticsAndDataRecords):
    Now uses the new convenience function WebResourceLoadStatisticsStore::setPrevalentResource().
(WebKit::WebResourceLoadStatisticsStore::setPrevalentResource):
    New convenience function to make sure we always call
    WebResourceLoadStatisticsStore::recursivelyGetAllDomainsThatHaveRedirectedToThisDomain()
    and capture redirect chains backward.
(WebKit::WebResourceLoadStatisticsStore::setSubframeUnderTopFrameOrigin):
(WebKit::WebResourceLoadStatisticsStore::setSubresourceUnderTopFrameOrigin):
(WebKit::WebResourceLoadStatisticsStore::setSubresourceUniqueRedirectTo):
(WebKit::WebResourceLoadStatisticsStore::setSubresourceUniqueRedirectFrom):
(WebKit::WebResourceLoadStatisticsStore::setTopFrameUniqueRedirectTo):
(WebKit::WebResourceLoadStatisticsStore::setTopFrameUniqueRedirectFrom):
    Test infrastructure.
* UIProcess/WebResourceLoadStatisticsStore.h:

Tools:

* WebKitTestRunner/InjectedBundle/Bindings/TestRunner.idl:
* WebKitTestRunner/InjectedBundle/TestRunner.cpp:
(WTR::TestRunner::setStatisticsSubresourceUniqueRedirectFrom):
(WTR::TestRunner::setStatisticsTopFrameUniqueRedirectTo):
(WTR::TestRunner::setStatisticsTopFrameUniqueRedirectFrom):
* WebKitTestRunner/InjectedBundle/TestRunner.h:
* WebKitTestRunner/TestController.cpp:
(WTR::TestController::setStatisticsSubresourceUniqueRedirectFrom):
(WTR::TestController::setStatisticsTopFrameUniqueRedirectTo):
(WTR::TestController::setStatisticsTopFrameUniqueRedirectFrom):
* WebKitTestRunner/TestController.h:
* WebKitTestRunner/TestInvocation.cpp:
(WTR::TestInvocation::didReceiveSynchronousMessageFromInjectedBundle):

LayoutTests:

* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-redirect-collusion-expected.txt: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-redirect-collusion.html: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion-expected.txt: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion.html: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-unique-redirects-to-expected.txt: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-unique-redirects-to.html: Added.
* platform/wk2/TestExpectations:
    New tests added as [ Pass ].
  • Loading branch information
johnwilander authored and carlosgcampos committed Feb 20, 2018
1 parent 2fa283a commit f8e9930
Show file tree
Hide file tree
Showing 26 changed files with 676 additions and 18 deletions.
17 changes: 17 additions & 0 deletions LayoutTests/ChangeLog
@@ -1,3 +1,20 @@
2018-02-12 John Wilander <wilander@apple.com>

Resource Load Statistics: Classify resources as prevalent based on redirects to other prevalent resources
https://bugs.webkit.org/show_bug.cgi?id=182664
<rdar://problem/37372572>

Reviewed by Brent Fulgham.

* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-redirect-collusion-expected.txt: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-redirect-collusion.html: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion-expected.txt: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion.html: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-unique-redirects-to-expected.txt: Added.
* http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-unique-redirects-to.html: Added.
* platform/wk2/TestExpectations:
New tests added as [ Pass ].

2018-02-09 Philippe Normand <pnormand@igalia.com>

[GStreamer][WebAudio] Winamp2-js woes
Expand Down
@@ -0,0 +1,14 @@
Tests for classification based on subresource redirect collusion.

On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".


PASS Colluding host 1 got set as prevalent resource.
PASS Colluding host 2 got set as prevalent resource.
PASS Colluding host 3 got set as prevalent resource.
PASS Colluding host 4 got set as prevalent resource.
PASS Colluding localhost got set as prevalent resource after actual sub frame redirect.
PASS successfullyParsed is true

TEST COMPLETE

@@ -0,0 +1,62 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<script src="/js-test-resources/js-test.js"></script>
</head>
<body>
<script>
description("Tests for classification based on subresource redirect collusion.");
jsTestIsAsync = true;

const hostUnderTest = "127.0.0.1:8000";
const statisticsUrl = "http://" + hostUnderTest + "/temp";
const subresourceOrigin1 = "http://127.0.0.2:8000/temp";
const subresourceOrigin2 = "http://127.0.0.3:8000/temp";
const subresourceOrigin3 = "http://127.0.0.4:8000/temp";
const subresourceOrigin4 = "http://127.0.0.5:8000/temp";

function setEnableFeature(enable) {
if (!enable)
testRunner.statisticsResetToConsistentState();
internals.setResourceLoadStatisticsEnabled(enable);
testRunner.setCookieStoragePartitioningEnabled(enable);
}

function runTest() {
testRunner.statisticsNotifyObserver();

testRunner.setStatisticsSubresourceUniqueRedirectFrom(statisticsUrl, subresourceOrigin1);
testRunner.setStatisticsSubresourceUniqueRedirectFrom(subresourceOrigin1, subresourceOrigin2);
testRunner.setStatisticsSubresourceUniqueRedirectFrom(subresourceOrigin2, subresourceOrigin3);
testRunner.setStatisticsSubresourceUniqueRedirectFrom(subresourceOrigin3, subresourceOrigin4);

testRunner.setStatisticsPrevalentResource(statisticsUrl, true);
if (!testRunner.isStatisticsPrevalentResource(statisticsUrl))
testFailed("Host explicity set did not get set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(subresourceOrigin1))
testPassed("Colluding host 1 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(subresourceOrigin2))
testPassed("Colluding host 2 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(subresourceOrigin3))
testPassed("Colluding host 3 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(subresourceOrigin4))
testPassed("Colluding host 4 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource("http://localhost:8000"))
testPassed("Colluding localhost got set as prevalent resource after actual sub frame redirect.");

setEnableFeature(false);
finishJSTest();
}

if (document.location.host === hostUnderTest && window.testRunner && window.internals) {
setEnableFeature(true);

testRunner.setStatisticsPrevalentResource(statisticsUrl, false);
if (testRunner.isStatisticsPrevalentResource(statisticsUrl))
testFailed("Host did not get set as non-prevalent resource.");
}
</script>
<iframe onload="runTest()" src="http://localhost:8000/resourceLoadStatistics/resources/redirect.php?redirectTo=http://127.0.0.1:8000/resourceLoadStatistics/onclick.html"></iframe>
</body>
</html>
@@ -0,0 +1,14 @@
Tests for classification based on top frame redirect collusion.

On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".


PASS Colluding host 1 got set as prevalent resource.
PASS Colluding host 2 got set as prevalent resource.
PASS Colluding host 3 got set as prevalent resource.
PASS Colluding host 4 got set as prevalent resource.
PASS Colluding localhost got set as prevalent resource after actual navigational redirect.
PASS successfullyParsed is true

TEST COMPLETE

@@ -0,0 +1,65 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<script src="/js-test-resources/js-test.js"></script>
</head>
<body>
<script>
description("Tests for classification based on top frame redirect collusion.");
jsTestIsAsync = true;

const hostUnderTest = "127.0.0.1:8000";
const statisticsUrl = "http://" + hostUnderTest + "/temp";
const topFrameOrigin1 = "http://127.0.0.2:8000/temp";
const topFrameOrigin2 = "http://127.0.0.3:8000/temp";
const topFrameOrigin3 = "http://127.0.0.4:8000/temp";
const topFrameOrigin4 = "http://127.0.0.5:8000/temp";
const topFrameOrigin5 = "http://localhost:8000/";

function setEnableFeature(enable) {
if (!enable)
testRunner.statisticsResetToConsistentState();
internals.setResourceLoadStatisticsEnabled(enable);
testRunner.setCookieStoragePartitioningEnabled(enable);
}

function runTest() {
testRunner.setStatisticsTopFrameUniqueRedirectFrom(statisticsUrl, topFrameOrigin1);
testRunner.setStatisticsTopFrameUniqueRedirectFrom(topFrameOrigin1, topFrameOrigin2);
testRunner.setStatisticsTopFrameUniqueRedirectFrom(topFrameOrigin2, topFrameOrigin3);
testRunner.setStatisticsTopFrameUniqueRedirectFrom(topFrameOrigin3, topFrameOrigin4);

testRunner.setStatisticsPrevalentResource(statisticsUrl, true);
if (!testRunner.isStatisticsPrevalentResource(statisticsUrl))
testFailed("Host explicity set did not get set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(topFrameOrigin1))
testPassed("Colluding host 1 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(topFrameOrigin2))
testPassed("Colluding host 2 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(topFrameOrigin3))
testPassed("Colluding host 3 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(topFrameOrigin4))
testPassed("Colluding host 4 got set as prevalent resource.");
if (testRunner.isStatisticsPrevalentResource(topFrameOrigin5))
testPassed("Colluding localhost got set as prevalent resource after actual navigational redirect.");

setEnableFeature(false);
finishJSTest();
}

if (document.location.hash === "" && window.testRunner && window.internals) {
setEnableFeature(true);

testRunner.setStatisticsPrevalentResource(statisticsUrl, false);
if (testRunner.isStatisticsPrevalentResource(statisticsUrl))
testFailed("Host did not get set as non-prevalent resource.");

document.location.href = topFrameOrigin5 + "resourceLoadStatistics/resources/redirect.php?redirectTo=http://127.0.0.1:8000/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion.html#continue";
} else {
testRunner.statisticsNotifyObserver();
runTest();
}
</script>
</body>
</html>
@@ -0,0 +1,10 @@
Tests for classification based on top frame unique redirects to.

On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".


PASS Host classified as prevalent resource.
PASS successfullyParsed is true

TEST COMPLETE

@@ -0,0 +1,58 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<script src="/js-test-resources/js-test.js"></script>
</head>
<body>
<script>
description("Tests for classification based on top frame unique redirects to.");
jsTestIsAsync = true;

const hostUnderTest = "127.0.0.1:8000";
const statisticsUrl = "http://" + hostUnderTest + "/temp";
const topFrameOrigin1 = "http://127.0.0.2:8000/temp";
const topFrameOrigin2 = "http://127.0.0.3:8000/temp";
const topFrameOrigin3 = "http://127.0.0.4:8000/temp";
const topFrameOrigin4 = "http://127.0.0.5:8000/temp";

function setEnableFeature(enable) {
if (!enable)
testRunner.statisticsResetToConsistentState();
internals.setResourceLoadStatisticsEnabled(enable);
testRunner.setCookieStoragePartitioningEnabled(enable);
}

function completeTest() {
if (!testRunner.isStatisticsPrevalentResource(statisticsUrl))
testFailed("Host did not get classified as prevalent resource.");
else
testPassed("Host classified as prevalent resource.");

setEnableFeature(false);
finishJSTest();
}

function runTest() {
testRunner.setStatisticsTopFrameUniqueRedirectTo(statisticsUrl, topFrameOrigin1);
testRunner.setStatisticsTopFrameUniqueRedirectTo(statisticsUrl, topFrameOrigin2);
testRunner.setStatisticsTopFrameUniqueRedirectTo(statisticsUrl, topFrameOrigin3);
testRunner.setStatisticsTopFrameUniqueRedirectTo(statisticsUrl, topFrameOrigin4);

testRunner.installStatisticsDidScanDataRecordsCallback(completeTest);

testRunner.statisticsProcessStatisticsAndDataRecords();
}

if (document.location.host === hostUnderTest && window.testRunner && window.internals) {
setEnableFeature(true);
testRunner.setStatisticsNotifyPagesWhenDataRecordsWereScanned(true);

testRunner.setStatisticsPrevalentResource(statisticsUrl, false);
if (testRunner.isStatisticsPrevalentResource(statisticsUrl))
testFailed("Host did not get set as non-prevalent resource.");
runTest();
}
</script>
</body>
</html>
3 changes: 3 additions & 0 deletions LayoutTests/platform/wk2/TestExpectations
Expand Up @@ -690,6 +690,9 @@ http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-mixed-statistic
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-sub-frame-under-top-frame-origins.html [ Pass ]
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-under-top-frame-origins.html [ Pass ]
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-unique-redirects-to.html [ Pass ]
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-redirect-collusion.html [ Pass ]
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion.html [ Pass ]
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-unique-redirects-to.html [ Pass ]
webkit.org/b/181223 http/tests/resourceLoadStatistics/clear-in-memory-and-persistent-store.html [ Skip ]
http/tests/resourceLoadStatistics/grandfathering.html [ Pass ]
webkit.org/b/180703 http/tests/resourceLoadStatistics/telemetry-generation.html [ Pass Failure ]
Expand Down
28 changes: 28 additions & 0 deletions Source/WebCore/ChangeLog
@@ -1,3 +1,31 @@
2018-02-12 John Wilander <wilander@apple.com>

Resource Load Statistics: Classify resources as prevalent based on redirects to other prevalent resources
https://bugs.webkit.org/show_bug.cgi?id=182664
<rdar://problem/37372572>

Reviewed by Brent Fulgham.

Tests: http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-subresource-redirect-collusion.html
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-redirect-collusion.html
http/tests/resourceLoadStatistics/classify-as-prevalent-based-on-top-frame-unique-redirects-to.html

* loader/ResourceLoadObserver.cpp:
(WebCore::ResourceLoadObserver::logFrameNavigation):
We now collect statistics on top frame navigational redirects too, both to and from.
(WebCore::ResourceLoadObserver::logSubresourceLoading):
We now collect statistics on redirects from in addition to the existing redirects to.
* loader/ResourceLoadStatistics.cpp:
(WebCore::ResourceLoadStatistics::encode const):
(WebCore::ResourceLoadStatistics::decode):
(WebCore::ResourceLoadStatistics::toString const):
(WebCore::ResourceLoadStatistics::merge):
* loader/ResourceLoadStatistics.h:
Three new fields:
- topFrameUniqueRedirectsTo
- topFrameUniqueRedirectsFrom
- subresourceUniqueRedirectsFrom

2018-02-12 Zalan Bujtas <zalan@apple.com>

[RenderTreeBuilder] Move RenderMenuList::takeChild() to RenderTreeBuilder
Expand Down
26 changes: 20 additions & 6 deletions Source/WebCore/loader/ResourceLoadObserver.cpp
Expand Up @@ -138,9 +138,6 @@ void ResourceLoadObserver::logFrameNavigation(const Frame& frame, const Frame& t
ASSERT(topFrame.document());
ASSERT(topFrame.page());

if (frame.isMainFrame())
return;

auto* page = topFrame.page();
if (!shouldLog(page))
return;
Expand Down Expand Up @@ -177,8 +174,21 @@ void ResourceLoadObserver::logFrameNavigation(const Frame& frame, const Frame& t

if (isRedirect
&& !areDomainsAssociated(page, sourcePrimaryDomain, targetPrimaryDomain)) {
auto& redirectingOriginStatistics = ensureResourceStatisticsForPrimaryDomain(sourcePrimaryDomain);
if (redirectingOriginStatistics.subresourceUniqueRedirectsTo.add(targetPrimaryDomain).isNewEntry)
bool isNewRedirectToEntry = false;
bool isNewRedirectFromEntry = false;
if (frame.isMainFrame()) {
auto& redirectingOriginStatistics = ensureResourceStatisticsForPrimaryDomain(sourcePrimaryDomain);
isNewRedirectToEntry = redirectingOriginStatistics.topFrameUniqueRedirectsTo.add(targetPrimaryDomain).isNewEntry;
auto& targetStatistics = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain);
isNewRedirectFromEntry = targetStatistics.topFrameUniqueRedirectsFrom.add(sourcePrimaryDomain).isNewEntry;
} else {
auto& redirectingOriginStatistics = ensureResourceStatisticsForPrimaryDomain(sourcePrimaryDomain);
isNewRedirectToEntry = redirectingOriginStatistics.subresourceUniqueRedirectsTo.add(targetPrimaryDomain).isNewEntry;
auto& targetStatistics = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain);
isNewRedirectFromEntry = targetStatistics.subresourceUniqueRedirectsFrom.add(sourcePrimaryDomain).isNewEntry;
}

if (isNewRedirectToEntry || isNewRedirectFromEntry)
shouldCallNotificationCallback = true;
}

Expand Down Expand Up @@ -234,7 +244,11 @@ void ResourceLoadObserver::logSubresourceLoading(const Frame* frame, const Resou

if (isRedirect) {
auto& redirectingOriginStatistics = ensureResourceStatisticsForPrimaryDomain(sourcePrimaryDomain);
if (redirectingOriginStatistics.subresourceUniqueRedirectsTo.add(targetPrimaryDomain).isNewEntry)
bool isNewRedirectToEntry = redirectingOriginStatistics.subresourceUniqueRedirectsTo.add(targetPrimaryDomain).isNewEntry;
auto& targetStatistics = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain);
bool isNewRedirectFromEntry = targetStatistics.subresourceUniqueRedirectsFrom.add(sourcePrimaryDomain).isNewEntry;

if (isNewRedirectToEntry || isNewRedirectFromEntry)
shouldCallNotificationCallback = true;
}

Expand Down

0 comments on commit f8e9930

Please sign in to comment.