Skip to content
Permalink
Browse files
Migrate PrivateClickMeasurements from ResourceLoadStatistics database…
… to new database

https://bugs.webkit.org/show_bug.cgi?id=229646

Patch by Alex Christensen <achristensen@webkit.org> on 2021-08-30
Reviewed by Kate Cheney.

Source/WebKit:

In r281721 I moved the PCM database to be separate from the ResourceLoadStatistics database.
This patch extracts data from the old database and sends it to the new database.

I added some tests that create databases in each of the 3 historical schemas then verify that
data is successfully migrated to the new database.

* NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:
(WebKit::ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema):
(WebKit::ResourceLoadStatisticsDatabaseStore::tableExists):
(WebKit::ResourceLoadStatisticsDatabaseStore::deleteTable):
(WebKit::ResourceLoadStatisticsDatabaseStore::migrateDataToPCMDatabaseIfNecessary):
(WebKit::ResourceLoadStatisticsDatabaseStore::openAndUpdateSchemaIfNecessary):
(WebKit::ResourceLoadStatisticsDatabaseStore::destroyStatements):
* NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h:
* NetworkProcess/DatabaseUtilities.cpp:
(WebKit::DatabaseUtilities::buildPrivateClickMeasurementFromDatabase):
* NetworkProcess/DatabaseUtilities.h:
* NetworkProcess/PrivateClickMeasurement/PrivateClickMeasurementDatabase.cpp:
(WebKit::PCM::Database::insertPrivateClickMeasurement):
(WebKit::PCM::Database::findPrivateClickMeasurement):
(WebKit::PCM::Database::allAttributedPrivateClickMeasurement):
(WebKit::PCM::Database::getDomainStringFromDomainID const):
(WebKit::PCM::Database::buildPrivateClickMeasurementFromDatabase): Deleted.
(WebKit::PCM::Database::getDomainStringFromDomainID): Deleted.
* NetworkProcess/PrivateClickMeasurement/PrivateClickMeasurementDatabase.h:
* UIProcess/API/Cocoa/WKWebViewPrivateForTesting.h:
* UIProcess/API/Cocoa/WKWebViewTesting.mm:
(-[WKWebView _dumpPrivateClickMeasurement:]):

Tools:

* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WebKitCocoa/EventAttribution.mm:
* TestWebKitAPI/Tests/WebKitCocoa/PrivateClickMeasurement.mm:
(webViewWithResourceLoadStatisticsEnabledInNetworkProcess):
(addValuesToTable):
(earliestTimeToSend):
(addAttributedPCMv1):
(addUnattributedPCMv1):
(addAttributedPCMv2):
(addUnattributedPCMv2):
(addAttributedPCMv3):
(addUnattributedPCMv3):
(dumpedPCM):
(pollUntilPCMIsMigrated):
(emptyObservationsDBPath):
(cleanUp):
(createAndPopulateObservedDomainTable):
(setUp):
(TEST):
* TestWebKitAPI/Tests/WebKitCocoa/pcmWithoutFraudPreventionDatabase.db: Removed.
* TestWebKitAPI/Tests/WebKitCocoa/pcmWithoutFraudPreventionDatabase.db-shm: Removed.
* TestWebKitAPI/Tests/WebKitCocoa/pcmWithoutFraudPreventionDatabase.db-wal: Removed.
* TestWebKitAPI/Tests/WebKitCocoa/pcmWithoutReportingColumns.db: Removed.
* TestWebKitAPI/Tests/WebKitCocoa/pcmWithoutReportingColumns.db-shm: Removed.
* TestWebKitAPI/Tests/WebKitCocoa/pcmWithoutReportingColumns.db-wal: Removed.

Canonical link: https://commits.webkit.org/241116@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@281779 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
Alex Christensen authored and webkit-commit-queue committed Aug 30, 2021
1 parent f93d412 commit a6ac5511f8fe324f1fa09ded04cede243b738e77
Showing 19 changed files with 439 additions and 190 deletions.
@@ -1,3 +1,39 @@
2021-08-30 Alex Christensen <achristensen@webkit.org>

Migrate PrivateClickMeasurements from ResourceLoadStatistics database to new database
https://bugs.webkit.org/show_bug.cgi?id=229646

Reviewed by Kate Cheney.

In r281721 I moved the PCM database to be separate from the ResourceLoadStatistics database.
This patch extracts data from the old database and sends it to the new database.

I added some tests that create databases in each of the 3 historical schemas then verify that
data is successfully migrated to the new database.

* NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:
(WebKit::ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema):
(WebKit::ResourceLoadStatisticsDatabaseStore::tableExists):
(WebKit::ResourceLoadStatisticsDatabaseStore::deleteTable):
(WebKit::ResourceLoadStatisticsDatabaseStore::migrateDataToPCMDatabaseIfNecessary):
(WebKit::ResourceLoadStatisticsDatabaseStore::openAndUpdateSchemaIfNecessary):
(WebKit::ResourceLoadStatisticsDatabaseStore::destroyStatements):
* NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h:
* NetworkProcess/DatabaseUtilities.cpp:
(WebKit::DatabaseUtilities::buildPrivateClickMeasurementFromDatabase):
* NetworkProcess/DatabaseUtilities.h:
* NetworkProcess/PrivateClickMeasurement/PrivateClickMeasurementDatabase.cpp:
(WebKit::PCM::Database::insertPrivateClickMeasurement):
(WebKit::PCM::Database::findPrivateClickMeasurement):
(WebKit::PCM::Database::allAttributedPrivateClickMeasurement):
(WebKit::PCM::Database::getDomainStringFromDomainID const):
(WebKit::PCM::Database::buildPrivateClickMeasurementFromDatabase): Deleted.
(WebKit::PCM::Database::getDomainStringFromDomainID): Deleted.
* NetworkProcess/PrivateClickMeasurement/PrivateClickMeasurementDatabase.h:
* UIProcess/API/Cocoa/WKWebViewPrivateForTesting.h:
* UIProcess/API/Cocoa/WKWebViewTesting.mm:
(-[WKWebView _dumpPrivateClickMeasurement:]):

2021-08-30 Alex Christensen <achristensen@webkit.org>

WKWebViewConfiguration._loadsSubresources=NO should prevent preconnecting
@@ -330,29 +330,42 @@ void ResourceLoadStatisticsDatabaseStore::openITPDatabase()
std::optional<Vector<String>> ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema()
{
Vector<String> missingTables;
auto statement = m_database.prepareStatement("SELECT 1 from sqlite_master WHERE type='table' and tbl_name=?"_s);
if (!statement) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema failed to prepare, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
return std::nullopt;
}

for (auto& table : expectedTableAndIndexQueries().keys()) {
if (statement->bindText(1, table) != SQLITE_OK) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema failed to bind, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
return std::nullopt;
}
if (statement->step() != SQLITE_ROW) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema schema is missing table: %s", this, table.ascii().data());
if (!this->tableExists(table))
missingTables.append(String(table));
}
statement->reset();
}
if (missingTables.isEmpty())
return std::nullopt;

return missingTables;
}

bool ResourceLoadStatisticsDatabaseStore::tableExists(StringView tableName)
{
constexpr auto checkIfTableExistsQuery = "SELECT 1 from sqlite_master WHERE type='table' and tbl_name=?"_s;
auto scopedStatement = this->scopedStatement(m_checkIfTableExistsStatement, checkIfTableExistsQuery, "tableExists"_s);
if (!scopedStatement) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::tableExists failed to prepare, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
return false;
}
if (scopedStatement->bindText(1, tableName) != SQLITE_OK) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::tableExists failed to bind, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
return false;
}
return scopedStatement->step() == SQLITE_ROW;
}

void ResourceLoadStatisticsDatabaseStore::deleteTable(StringView tableName)
{
ASSERT(tableExists(tableName));
auto dropTableQuery = m_database.prepareStatementSlow(makeString("DROP TABLE ", tableName));
ASSERT(dropTableQuery);
if (!dropTableQuery || dropTableQuery->step() != SQLITE_DONE) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::deleteTable failed to drop temporary tables, error message: %s", this, m_database.lastErrorMsg());
ASSERT_NOT_REACHED();
}
}

TableAndIndexPair ResourceLoadStatisticsDatabaseStore::currentTableAndIndexQueries(const String& tableName)
{
auto getTableStatement = m_database.prepareStatement("SELECT sql FROM sqlite_master WHERE tbl_name=? AND type = 'table'"_s);
@@ -504,6 +517,54 @@ void ResourceLoadStatisticsDatabaseStore::migrateDataToNewTablesIfNecessary()
}
}

void ResourceLoadStatisticsDatabaseStore::migrateDataToPCMDatabaseIfNecessary()
{
if (!tableExists("UnattributedPrivateClickMeasurement")
&& !tableExists("AttributedPrivateClickMeasurement"))
return;

Vector<WebCore::PrivateClickMeasurement> unattributed;
{
constexpr auto allUnattributedPrivateClickMeasurementAttributionsQuery = "SELECT * FROM UnattributedPrivateClickMeasurement"_s;
auto unattributedScopedStatement = m_database.prepareStatement(allUnattributedPrivateClickMeasurementAttributionsQuery);
if (!unattributedScopedStatement) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::migrateDataToPCMDatabaseIfNecessary failed to prepare unattributed statement, error message: %s", this, m_database.lastErrorMsg());
ASSERT_NOT_REACHED();
return;
}
while (unattributedScopedStatement->step() == SQLITE_ROW)
unattributed.append(buildPrivateClickMeasurementFromDatabase(unattributedScopedStatement.value(), PrivateClickMeasurementAttributionType::Unattributed));
}

Vector<WebCore::PrivateClickMeasurement> attributed;
{
constexpr auto allAttributedPrivateClickMeasurementQuery = "SELECT * FROM AttributedPrivateClickMeasurement"_s;
auto attributedScopedStatement = m_database.prepareStatement(allAttributedPrivateClickMeasurementQuery);
if (!attributedScopedStatement) {
RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::migrateDataToPCMDatabaseIfNecessary failed to prepare attributed statement, error message: %s", this, m_database.lastErrorMsg());
ASSERT_NOT_REACHED();
return;
}
while (attributedScopedStatement->step() == SQLITE_ROW)
attributed.append(buildPrivateClickMeasurementFromDatabase(attributedScopedStatement.value(), PrivateClickMeasurementAttributionType::Attributed));
}

if (!unattributed.isEmpty() || !attributed.isEmpty()) {
RunLoop::main().dispatch([store = makeRef(store()), unattributed = unattributed.isolatedCopy(), attributed = attributed.isolatedCopy()] () mutable {
auto& pcmStore = store->privateClickMeasurementStore();
for (auto& pcm : WTFMove(attributed))
pcmStore.insertPrivateClickMeasurement(WTFMove(pcm), PrivateClickMeasurementAttributionType::Attributed);
for (auto& pcm : WTFMove(unattributed))
pcmStore.insertPrivateClickMeasurement(WTFMove(pcm), PrivateClickMeasurementAttributionType::Unattributed);
});

}

auto transactionScope = beginTransactionIfNecessary();
deleteTable("UnattributedPrivateClickMeasurement");
deleteTable("AttributedPrivateClickMeasurement");
}

Vector<String> ResourceLoadStatisticsDatabaseStore::columnsForTable(const String& tableName)
{
auto statement = m_database.prepareStatementSlow(makeString("PRAGMA table_info(", tableName, ")"));
@@ -582,7 +643,7 @@ void ResourceLoadStatisticsDatabaseStore::openAndUpdateSchemaIfNecessary()
return;
}

// Renaming and adding columns should be done before migrating to avoid mismatched or missing columns.
migrateDataToPCMDatabaseIfNecessary();
migrateDataToNewTablesIfNecessary();
}

@@ -743,6 +804,7 @@ void ResourceLoadStatisticsDatabaseStore::destroyStatements()
m_uniqueRedirectExistsStatement = nullptr;
m_observedDomainsExistsStatement = nullptr;
m_removeAllDataStatement = nullptr;
m_checkIfTableExistsStatement = nullptr;
}

bool ResourceLoadStatisticsDatabaseStore::insertObservedDomain(const ResourceLoadStatistics& loadStatistics)
@@ -140,12 +140,16 @@ class ResourceLoadStatisticsDatabaseStore final : public ResourceLoadStatisticsS
TableAndIndexPair currentTableAndIndexQueries(const String&);
bool missingReferenceToObservedDomains();
void migrateDataToNewTablesIfNecessary();
void migrateDataToPCMDatabaseIfNecessary();
bool tableExists(StringView);
void deleteTable(StringView);

void destroyStatements() final;

bool hasStorageAccess(const TopFrameDomain&, const SubFrameDomain&) const;
Vector<WebResourceLoadStatisticsStore::ThirdPartyDataForSpecificFirstParty> getThirdPartyDataForSpecificFirstPartyDomains(unsigned, const RegistrableDomain&) const;
void openAndUpdateSchemaIfNecessary();
String getDomainStringFromDomainID(unsigned) const;
String getDomainStringFromDomainID(unsigned) const final;
ASCIILiteral getSubStatisticStatement(const String&) const;
void appendSubStatisticList(StringBuilder&, const String& tableName, const String& domain) const;
void mergeStatistic(const ResourceLoadStatistics&);
@@ -254,6 +258,7 @@ class ResourceLoadStatisticsDatabaseStore final : public ResourceLoadStatisticsS
mutable std::unique_ptr<WebCore::SQLiteStatement> m_uniqueRedirectExistsStatement;
mutable std::unique_ptr<WebCore::SQLiteStatement> m_observedDomainsExistsStatement;
mutable std::unique_ptr<WebCore::SQLiteStatement> m_removeAllDataStatement;
mutable std::unique_ptr<WebCore::SQLiteStatement> m_checkIfTableExistsStatement;

PAL::SessionID m_sessionID;
bool m_isNewResourceLoadStatisticsDatabaseFile { false };
@@ -27,6 +27,8 @@
#include "DatabaseUtilities.h"

#include "Logging.h"
#include "PrivateClickMeasurementManager.h"
#include <WebCore/PrivateClickMeasurement.h>
#include <WebCore/SQLiteStatement.h>
#include <WebCore/SQLiteStatementAutoResetScope.h>
#include <wtf/FileSystem.h>
@@ -135,4 +137,44 @@ void DatabaseUtilities::interrupt()
m_database.interrupt();
}

WebCore::PrivateClickMeasurement DatabaseUtilities::buildPrivateClickMeasurementFromDatabase(WebCore::SQLiteStatement& statement, PrivateClickMeasurementAttributionType attributionType)
{
ASSERT(!RunLoop::isMain());
auto sourceSiteDomain = getDomainStringFromDomainID(statement.columnInt(0));
auto destinationSiteDomain = getDomainStringFromDomainID(statement.columnInt(1));
auto sourceID = statement.columnInt(2);
auto timeOfAdClick = attributionType == PrivateClickMeasurementAttributionType::Attributed ? statement.columnDouble(5) : statement.columnDouble(3);
auto token = attributionType == PrivateClickMeasurementAttributionType::Attributed ? statement.columnText(7) : statement.columnText(4);
auto signature = attributionType == PrivateClickMeasurementAttributionType::Attributed ? statement.columnText(8) : statement.columnText(5);
auto keyID = attributionType == PrivateClickMeasurementAttributionType::Attributed ? statement.columnText(9) : statement.columnText(6);

WebCore::PrivateClickMeasurement attribution(WebCore::PrivateClickMeasurement::SourceID(sourceID), WebCore::PrivateClickMeasurement::SourceSite(WebCore::RegistrableDomain::uncheckedCreateFromRegistrableDomainString(sourceSiteDomain)), WebCore::PrivateClickMeasurement::AttributionDestinationSite(WebCore::RegistrableDomain::uncheckedCreateFromRegistrableDomainString(destinationSiteDomain)), { }, { }, WallTime::fromRawSeconds(timeOfAdClick));

if (attributionType == PrivateClickMeasurementAttributionType::Attributed) {
auto attributionTriggerData = statement.columnInt(3);
auto priority = statement.columnInt(4);
auto sourceEarliestTimeToSendValue = statement.columnDouble(6);
auto destinationEarliestTimeToSendValue = statement.columnDouble(10);

if (attributionTriggerData != -1)
attribution.setAttribution(WebCore::PrivateClickMeasurement::AttributionTriggerData { static_cast<uint32_t>(attributionTriggerData), WebCore::PrivateClickMeasurement::Priority(priority) });

std::optional<WallTime> sourceEarliestTimeToSend;
std::optional<WallTime> destinationEarliestTimeToSend;

// A value of 0.0 indicates that the report has been sent to the respective site.
if (sourceEarliestTimeToSendValue > 0.0)
sourceEarliestTimeToSend = WallTime::fromRawSeconds(sourceEarliestTimeToSendValue);

if (destinationEarliestTimeToSendValue > 0.0)
destinationEarliestTimeToSend = WallTime::fromRawSeconds(destinationEarliestTimeToSendValue);

attribution.setTimesToSend({ sourceEarliestTimeToSend, destinationEarliestTimeToSend });
}

attribution.setSourceSecretToken({ token, signature, keyID });

return attribution;
}

} // namespace WebKit
@@ -30,11 +30,15 @@
#include <wtf/Scope.h>

namespace WebCore {
class PrivateClickMeasurement;
class SQLiteStatement;
class SQLiteStatementAutoResetScope;
}

namespace WebKit {

enum class PrivateClickMeasurementAttributionType : bool;

class DatabaseUtilities {
protected:
DatabaseUtilities(String&& storageFilePath);
@@ -49,6 +53,9 @@ class DatabaseUtilities {
void interrupt();
virtual bool createSchema() = 0;
virtual void destroyStatements() = 0;
virtual String getDomainStringFromDomainID(unsigned) const = 0;

WebCore::PrivateClickMeasurement buildPrivateClickMeasurementFromDatabase(WebCore::SQLiteStatement&, PrivateClickMeasurementAttributionType);

const String m_storageFilePath;
mutable WebCore::SQLiteDatabase m_database;

0 comments on commit a6ac551

Please sign in to comment.