Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

optimize uuid building #4279

Merged
merged 3 commits into from Mar 11, 2021
Merged

optimize uuid building #4279

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f3bac56
optimize uuid building
bradymiller Mar 10, 2021
fbcfde6
minor optimization
bradymiller Mar 10, 2021
a4fe06d
documentation updates
bradymiller Mar 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
70 changes: 42 additions & 28 deletions src/Common/Uuid/UuidRegistry.php
View file
Open in desktop
Expand Up @@ -199,46 +199,60 @@ private static function appendPopulateLog($table, $count, &$logEntry)
}
}

// Generic function to create missing uuids in a sql table (table needs an `id` column to work)
// Generic function to create missing uuids in a sql table
// This function works in "blocks" of 1000 to avoid mysql/mariadb death when updating a super large (>500K) amount of uuids in one table
// This function returns the number of missing uuids that were populated
public function createMissingUuids()
{
// set up counter
$counter = 0;

// Empty should be NULL, but to be safe also checking for empty and null bytes
$resultSet = sqlStatementNoLog("SELECT * FROM `" . $this->table_name . "` WHERE `uuid` IS NULL OR `uuid` = '' OR `uuid` = '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0'");
while ($row = sqlFetchArray($resultSet)) {
$done = false;
while (!$done) {
// just maximum of 1000 at a time to attempt to speed things up and not break when inserting a large number of uuids
Copy link
Sponsor Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plan to try 100, 500, 1000, 10000 to see what is best (or if there is even any difference)

Copy link
Sponsor Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just gonna stick with 1000 for now. this is not really speeding things up (or slowing things down), but it definitely prevents breaking when doing very large updates (when having to do more than 500,000 uuid updates on a table).

Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addams_family

if (!$this->table_vertical) {
// standard case, add missing uuid
sqlQueryNoLog("UPDATE `" . $this->table_name . "` SET `uuid` = ? WHERE `" . $this->table_id . "` = ?", [$this->createUuid(), $row[$this->table_id]]);
$counter++;
// in this standard case, can decrease memory use by just collecting the id
$resultSet = sqlStatementNoLog("SELECT `" . $this->table_id . "` FROM `" . $this->table_name . "` WHERE `uuid` IS NULL OR `uuid` = '' OR `uuid` = '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' LIMIT 1000");
} else {
// more complicated case where setting uuid in a vertical table. In this case need a uuid for each combination of table columns stored in $this->table_vertical array
$stringQuery = "";
$arrayQuery = [];
$prependAnd = false;
foreach ($this->table_vertical as $column) {
if ($prependAnd) {
$stringQuery .= " AND `" . $column . "` = ? ";
// in this more complicated case (ie. vertical table), need to collect all columns
$resultSet = sqlStatementNoLog("SELECT * FROM `" . $this->table_name . "` WHERE `uuid` IS NULL OR `uuid` = '' OR `uuid` = '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' LIMIT 1000");
}
if (sqlNumRows($resultSet) < 1000) {
$done = true;
}
while ($row = sqlFetchArray($resultSet)) {
if (!$this->table_vertical) {
// standard case, add missing uuid
sqlQueryNoLog("UPDATE `" . $this->table_name . "` SET `uuid` = ? WHERE `" . $this->table_id . "` = ?", [$this->createUuid(), $row[$this->table_id]]);
$counter++;
} else {
// more complicated case where setting uuid in a vertical table. In this case need a uuid for each combination of table columns stored in $this->table_vertical array
$stringQuery = "";
$arrayQuery = [];
$prependAnd = false;
foreach ($this->table_vertical as $column) {
if ($prependAnd) {
$stringQuery .= " AND `" . $column . "` = ? ";
} else {
$stringQuery .= " `" . $column . "` = ? ";
}
$arrayQuery[] = $row[$column];
$prependAnd = true;
}
// First, see if it has already been set
$setUuid = sqlQueryNoLog("SELECT `uuid` FROM `" . $this->table_name . "` WHERE `uuid` IS NOT NULL AND `uuid` != '' AND `uuid` != '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' AND " . $stringQuery, $arrayQuery)['uuid'];
if (!empty($setUuid)) {
// Already set
array_unshift($arrayQuery, $setUuid);
} else {
$stringQuery .= " `" . $column . "` = ? ";
// Not already set, so create
array_unshift($arrayQuery, $this->createUuid());
}
$arrayQuery[] = $row[$column];
$prependAnd = true;
}
// First, see if it has already been set
$setUuid = sqlQueryNoLog("SELECT `uuid` FROM `" . $this->table_name . "` WHERE `uuid` IS NOT NULL AND `uuid` != '' AND `uuid` != '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' AND " . $stringQuery, $arrayQuery)['uuid'];
if (!empty($setUuid)) {
// Already set
array_unshift($arrayQuery, $setUuid);
} else {
// Not already set, so create
array_unshift($arrayQuery, $this->createUuid());
// Now populate
sqlQueryNoLog("UPDATE `" . $this->table_name . "` SET `uuid` = ? WHERE " . $stringQuery, $arrayQuery);
$counter++;
}
// Now populate
sqlQueryNoLog("UPDATE `" . $this->table_name . "` SET `uuid` = ? WHERE " . $stringQuery, $arrayQuery);
$counter++;
}
}

Expand Down