Skip to content

Commit 087cc08

Browse files
author
epriestley
committedApr 30, 2012
Make SQL patch management DAG-based and provide namespace support
Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
1 parent 68b597f commit 087cc08

37 files changed

+2168
-354
lines changed
 

‎bin/storage

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../scripts/sql/manage_storage.php

‎scripts/__init_script__.php

+11-1
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,17 @@
4242
? $_SERVER['PHABRICATOR_ENV']
4343
: getenv('PHABRICATOR_ENV');
4444
if (!$env) {
45-
echo "Define PHABRICATOR_ENV before running this script.\n";
45+
phutil_require_module('phutil', 'console');
46+
echo phutil_console_wrap(
47+
phutil_console_format(
48+
"**ERROR**: PHABRICATOR_ENV Not Set\n\n".
49+
"Define the __PHABRICATOR_ENV__ environment variable before running ".
50+
"this script. You can do it on the command line like this:\n\n".
51+
" $ PHABRICATOR_ENV=__custom/myconfig__ %s ...\n\n".
52+
"Replace __custom/myconfig__ with the path to your configuration file. ".
53+
"For more information, see the 'Configuration Guide' in the ".
54+
"Phabricator documentation.\n\n",
55+
$argv[0]));
4656
exit(1);
4757
}
4858

‎scripts/install/update_phabricator.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ $ROOT/phabricator/bin/phd stop
4949
sudo /etc/init.d/httpd stop
5050

5151
# Upgrade the database schema.
52-
$ROOT/phabricator/scripts/sql/upgrade_schema.php -f
52+
$ROOT/phabricator/bin/storage upgrade --force
5353

5454
# Restart apache.
5555
sudo /etc/init.d/httpd start

‎scripts/sql/manage_storage.php

+123
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env php
2+
<?php
3+
4+
/*
5+
* Copyright 2012 Facebook, Inc.
6+
*
7+
* Licensed under the Apache License, Version 2.0 (the "License");
8+
* you may not use this file except in compliance with the License.
9+
* You may obtain a copy of the License at
10+
*
11+
* http://www.apache.org/licenses/LICENSE-2.0
12+
*
13+
* Unless required by applicable law or agreed to in writing, software
14+
* distributed under the License is distributed on an "AS IS" BASIS,
15+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
* See the License for the specific language governing permissions and
17+
* limitations under the License.
18+
*/
19+
20+
$root = dirname(dirname(dirname(__FILE__)));
21+
require_once $root.'/scripts/__init_script__.php';
22+
23+
$args = new PhutilArgumentParser($argv);
24+
$args->setTagline('manage Phabricator storage and schemata');
25+
$args->setSynopsis(<<<EOHELP
26+
**storage** __workflow__ [__options__]
27+
Manage Phabricator database storage and schema versioning.
28+
29+
**storage** upgrade
30+
Initialize or upgrade Phabricator storage.
31+
32+
**storage** upgrade --user __root__ --password __hunter2__
33+
Use administrative credentials for schema changes.
34+
EOHELP
35+
);
36+
$args->parseStandardArguments();
37+
38+
$conf = PhabricatorEnv::newObjectFromConfig('mysql.configuration-provider');
39+
40+
$default_user = $conf->getUser();
41+
$default_password = $conf->getPassword();
42+
$default_host = $conf->getHost();
43+
$default_namespace = 'phabricator';
44+
45+
try {
46+
$args->parsePartial(
47+
array(
48+
array(
49+
'name' => 'force',
50+
'short' => 'f',
51+
'help' => 'Do not prompt before performing dangerous operations.',
52+
),
53+
array(
54+
'name' => 'user',
55+
'short' => 'u',
56+
'param' => 'username',
57+
'default' => $default_user,
58+
'help' => "Connect with __username__ instead of the configured ".
59+
"default ('{$default_user}').",
60+
),
61+
array(
62+
'name' => 'password',
63+
'short' => 'p',
64+
'param' => 'password',
65+
'default' => $default_password,
66+
'help' => 'Use __password__ instead of the configured default.',
67+
),
68+
array(
69+
'name' => 'namespace',
70+
'param' => 'name',
71+
'default' => $default_namespace,
72+
'help' => "Use namespace __namespace__ instead of the configured ".
73+
"default ('{$default_namespace}'). This is an advanced ".
74+
"feature used by unit tests; you should not normally ".
75+
"use this flag.",
76+
),
77+
array(
78+
'name' => 'dryrun',
79+
'help' => 'Do not actually change anything, just show what would be '.
80+
'changed.',
81+
),
82+
));
83+
} catch (PhutilArgumentUsageException $ex) {
84+
$args->printUsageException($ex);
85+
exit(77);
86+
}
87+
88+
$api = new PhabricatorStorageManagementAPI();
89+
$api->setUser($args->getArg('user'));
90+
$api->setHost($default_host);
91+
$api->setPassword($args->getArg('password'));
92+
$api->setNamespace($args->getArg('namespace'));
93+
94+
try {
95+
queryfx(
96+
$api->getConn('meta_data', $select_database = false),
97+
'SELECT 1');
98+
} catch (AphrontQueryException $ex) {
99+
echo phutil_console_format(
100+
"**%s**: %s\n",
101+
'Unable To Connect',
102+
$ex->getMessage());
103+
exit(1);
104+
}
105+
106+
$workflows = array(
107+
new PhabricatorStorageManagementDatabasesWorkflow(),
108+
new PhabricatorStorageManagementDestroyWorkflow(),
109+
new PhabricatorStorageManagementDumpWorkflow(),
110+
new PhabricatorStorageManagementStatusWorkflow(),
111+
new PhabricatorStorageManagementUpgradeWorkflow(),
112+
);
113+
114+
$patches = PhabricatorSQLPatchList::buildAllPatches();
115+
116+
foreach ($workflows as $workflow) {
117+
$workflow->setAPI($api);
118+
$workflow->setPatches($patches);
119+
}
120+
121+
$workflows[] = new PhutilHelpArgumentWorkflow();
122+
123+
$args->parseWorkflows($workflows);

‎scripts/sql/upgrade_schema.php

+7-201
Original file line numberDiff line numberDiff line change
@@ -17,204 +17,10 @@
1717
* limitations under the License.
1818
*/
1919

20-
$root = dirname(dirname(dirname(__FILE__)));
21-
require_once $root.'/scripts/__init_script__.php';
22-
23-
phutil_require_module('phutil', 'console');
24-
phutil_require_module('phabricator', 'infrastructure/setup/sql');
25-
26-
define('SCHEMA_VERSION_TABLE_NAME', 'schema_version');
27-
28-
// TODO: getopt() is super terrible, move to something less terrible.
29-
$options = getopt('fhdv:u:p:m:') + array(
30-
'v' => null, // Upgrade from specific version
31-
'u' => null, // Override MySQL User
32-
'p' => null, // Override MySQL Pass
33-
'm' => null, // Specify max version to upgrade to
34-
);
35-
36-
foreach (array('h', 'f', 'd') as $key) {
37-
// By default, these keys are set to 'false' to indicate that the flag was
38-
// passed.
39-
if (array_key_exists($key, $options)) {
40-
$options[$key] = true;
41-
}
42-
}
43-
44-
if (!empty($options['h']) || ($options['v'] && !is_numeric($options['v']))
45-
|| ($options['m'] && !is_numeric($options['m']))) {
46-
usage();
47-
}
48-
49-
if (empty($options['f']) && empty($options['d'])) {
50-
echo phutil_console_wrap(
51-
"Before running this script, you should take down the Phabricator web ".
52-
"interface and stop any running Phabricator daemons.");
53-
54-
if (!phutil_console_confirm('Are you ready to continue?')) {
55-
echo "Cancelled.\n";
56-
exit(1);
57-
}
58-
}
59-
60-
// Use always the version from the commandline if it is defined
61-
$next_version = isset($options['v']) ? (int)$options['v'] : null;
62-
$max_version = isset($options['m']) ? (int)$options['m'] : null;
63-
64-
$conf = PhabricatorEnv::newObjectFromConfig('mysql.configuration-provider');
65-
66-
if ($options['u']) {
67-
$conn_user = $options['u'];
68-
$conn_pass = $options['p'];
69-
} else {
70-
$conn_user = $conf->getUser();
71-
$conn_pass = $conf->getPassword();
72-
}
73-
$conn_host = $conf->getHost();
74-
75-
// Split out port information, since the command-line client requires a
76-
// separate flag for the port.
77-
$uri = new PhutilURI('mysql://'.$conn_host);
78-
if ($uri->getPort()) {
79-
$conn_port = $uri->getPort();
80-
$conn_bare_hostname = $uri->getDomain();
81-
} else {
82-
$conn_port = null;
83-
$conn_bare_hostname = $conn_host;
84-
}
85-
86-
$conn = PhabricatorEnv::newObjectFromConfig(
87-
'mysql.implementation',
88-
array(
89-
array(
90-
'user' => $conn_user,
91-
'pass' => $conn_pass,
92-
'host' => $conn_host,
93-
'database' => null,
94-
),
95-
));
96-
97-
try {
98-
99-
$create_sql = <<<END
100-
CREATE DATABASE IF NOT EXISTS `phabricator_meta_data`;
101-
END;
102-
queryfx($conn, $create_sql);
103-
104-
$create_sql = <<<END
105-
CREATE TABLE IF NOT EXISTS phabricator_meta_data.`schema_version` (
106-
`version` INTEGER not null
107-
);
108-
END;
109-
queryfx($conn, $create_sql);
110-
111-
// Get the version only if commandline argument wasn't given
112-
if ($next_version === null) {
113-
$version = queryfx_one(
114-
$conn,
115-
'SELECT * FROM phabricator_meta_data.%T',
116-
SCHEMA_VERSION_TABLE_NAME);
117-
118-
if (!$version) {
119-
print "*** No version information in the database ***\n";
120-
print "*** Give the first patch version which to ***\n";
121-
print "*** apply as the command line argument ***\n";
122-
exit(-1);
123-
}
124-
125-
$next_version = $version['version'] + 1;
126-
}
127-
128-
$patches = PhabricatorSQLPatchList::getPatchList();
129-
130-
$patch_applied = false;
131-
foreach ($patches as $patch) {
132-
if ($patch['version'] < $next_version) {
133-
continue;
134-
}
135-
136-
if ($max_version && $patch['version'] > $max_version) {
137-
continue;
138-
}
139-
140-
$short_name = basename($patch['path']);
141-
print "Applying patch {$short_name}...\n";
142-
143-
if (!empty($options['d'])) {
144-
$patch_applied = true;
145-
continue;
146-
}
147-
148-
if ($conn_port) {
149-
$port = '--port='.(int)$conn_port;
150-
} else {
151-
$port = null;
152-
}
153-
154-
if (preg_match('/\.php$/', $patch['path'])) {
155-
$schema_conn = $conn;
156-
require_once $patch['path'];
157-
} else {
158-
list($stdout, $stderr) = execx(
159-
"mysql --user=%s --password=%s --host=%s {$port} ".
160-
"--default-character-set=utf8 < %s",
161-
$conn_user,
162-
$conn_pass,
163-
$conn_bare_hostname,
164-
$patch['path']);
165-
166-
if ($stderr) {
167-
print $stderr;
168-
exit(-1);
169-
}
170-
}
171-
172-
// Patch was successful, update the db with the latest applied patch version
173-
// 'DELETE' and 'INSERT' instead of update, because the table might be empty
174-
queryfx(
175-
$conn,
176-
'DELETE FROM phabricator_meta_data.%T',
177-
SCHEMA_VERSION_TABLE_NAME);
178-
queryfx(
179-
$conn,
180-
'INSERT INTO phabricator_meta_data.%T VALUES (%d)',
181-
SCHEMA_VERSION_TABLE_NAME,
182-
$patch['version']);
183-
184-
$patch_applied = true;
185-
}
186-
187-
if (!$patch_applied) {
188-
print "Your database is already up-to-date.\n";
189-
}
190-
191-
} catch (AphrontQueryAccessDeniedException $ex) {
192-
echo
193-
"ACCESS DENIED\n".
194-
"The user '{$conn_user}' does not have sufficient MySQL privileges to\n".
195-
"execute the schema upgrade. Use the -u and -p flags to run as a user\n".
196-
"with more privileges (e.g., root).".
197-
"\n\n".
198-
"EXCEPTION:\n".
199-
$ex->getMessage().
200-
"\n\n";
201-
exit(1);
202-
}
203-
204-
function usage() {
205-
echo
206-
"usage: upgrade_schema.php [-v version] [-u user -p pass] [-f] [-h]".
207-
"\n\n".
208-
"Run 'upgrade_schema.php -u root -p hunter2' to override the configured ".
209-
"default user.\n".
210-
"Run 'upgrade_schema.php -v 12' to apply all patches starting from ".
211-
"version 12. It is very unlikely you need to do this.\n".
212-
"Run 'upgrade_schema.php -m 110' to apply all patches up to and ".
213-
"including version 110 (but nothing past).\n".
214-
"Use the -f flag to upgrade noninteractively, without prompting.\n".
215-
"Use the -d flag to do a dry run - patches that would be applied ".
216-
"will be listed, but not applied.\n".
217-
"Use the -h flag to show this help.\n";
218-
exit(1);
219-
}
220-
20+
echo "This script has been replaced by 'phabricator/bin/storage'.\n\n".
21+
"Run:\n\n".
22+
" phabricator/bin $ ./storage help\n\n".
23+
"...for help, or:\n\n".
24+
" phabricator/bin $ ./storage upgrade\n\n".
25+
"...to upgrade storage.\n\n";
26+
exit(1);

0 commit comments

Comments
 (0)
Failed to load comments.