From a49ab239d6859dfd4746b804b5160a7a161a3501 Mon Sep 17 00:00:00 2001 From: Max Bronsema Date: Thu, 28 May 2015 17:23:55 -0700 Subject: [PATCH 1/4] Added in rough copy of navigation for importing sites. --- mkdocs.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index c69f66d..6410fde 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,4 +9,7 @@ pages: - ['install/debian.md', 'Install', 'Debian/Ubuntu'] - ['install/manual.md', 'Install', 'Manual'] - ['install/common-problems.md', 'Install', 'Common Problems'] - +- ['import/importing-sites.md', 'Importing Sites'] +- ['import/aegir-backups.md', 'Aegir Backups'] +- ['import/single-site.md', 'Single Site Import'] +- ['import/complete-drupal-install.md', 'Complete Drupal Install'] From d0c1735be52b22f25adff1a5ba36746c48015844 Mon Sep 17 00:00:00 2001 From: Max Bronsema Date: Fri, 29 May 2015 16:13:37 -0700 Subject: [PATCH 2/4] Skeleton for the importing section of the manual. --- docs/import/aegir-backups.md | 0 docs/import/complete-drupal-install.md | 0 docs/import/importing-sites.md | 12 ++++++++++++ docs/import/single-site.md | 0 mkdocs.yml | 6 +++--- 5 files changed, 15 insertions(+), 3 deletions(-) create mode 100644 docs/import/aegir-backups.md create mode 100644 docs/import/complete-drupal-install.md create mode 100644 docs/import/importing-sites.md create mode 100644 docs/import/single-site.md diff --git a/docs/import/aegir-backups.md b/docs/import/aegir-backups.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/import/complete-drupal-install.md b/docs/import/complete-drupal-install.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/import/importing-sites.md b/docs/import/importing-sites.md new file mode 100644 index 0000000..6b132eb --- /dev/null +++ b/docs/import/importing-sites.md @@ -0,0 +1,12 @@ +Importing sites +=============== + +It is possible to import existing sites into the Aegir hosting system, so that they can be managed by Aegir. + +These instructions assume you've set up a new server with Aegir on it, and you want to import sites into Aegir from another server, or even from the same server, using backups. + +There are three standard ways of importing sites into Aegir: + +* from [Aegir backups](import/aegir-backups)Aegir backups - simplest and fastest, scriptable +* from any [single site](import/single-site) manually - most reliable, can import even non-aegir sites, hard to automate +* from a [complete drupal install](import/complete-drupal-install) - fairly reliable, can import non-aegir sites, could be automated but requires access to the database servers currently in use by the site diff --git a/docs/import/single-site.md b/docs/import/single-site.md new file mode 100644 index 0000000..e69de29 diff --git a/mkdocs.yml b/mkdocs.yml index 6410fde..e5d0ff9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,6 +10,6 @@ pages: - ['install/manual.md', 'Install', 'Manual'] - ['install/common-problems.md', 'Install', 'Common Problems'] - ['import/importing-sites.md', 'Importing Sites'] -- ['import/aegir-backups.md', 'Aegir Backups'] -- ['import/single-site.md', 'Single Site Import'] -- ['import/complete-drupal-install.md', 'Complete Drupal Install'] +- ['import/aegir-backups.md', 'Importing Sites', 'From Aegir Backups'] +- ['import/single-site.md', 'Importing Sites', 'Single Site Import'] +- ['import/complete-drupal-install.md', 'Importing Sites', 'Complete Drupal Install'] From 20830d96f693948645218e82d0aeab93be379f20 Mon Sep 17 00:00:00 2001 From: Max Bronsema Date: Tue, 2 Jun 2015 20:43:55 -0700 Subject: [PATCH 3/4] First draft port of site import docs. --- docs/import/complete-drupal-install.md | 108 +++++++++++++++++++++++++ docs/import/importing-sites.md | 6 +- docs/import/single-site.md | 89 ++++++++++++++++++++ 3 files changed, 200 insertions(+), 3 deletions(-) diff --git a/docs/import/complete-drupal-install.md b/docs/import/complete-drupal-install.md index e69de29..1443ab6 100644 --- a/docs/import/complete-drupal-install.md +++ b/docs/import/complete-drupal-install.md @@ -0,0 +1,108 @@ +Importing a complete Drupal platform +==================================== + +Table of Contents + +1. Transfer the codebase +2. Transfer the Database +3. Setup Platform in Aegir +4. Import and Verify Sites In Aegir +5. Migrate To Another Platform +6. Troubleshooting + +This way of importing sites is often considered the safest whereby you transfer across an entire codebase containing Drupal core, and define it as a Platform first. This makes sure you bring any dependencies, contrib modules etc, in exactly the same version and configuration that is referenced in the database for the site. + +1. Transfer the codebase + + Create a tarball of the codebase on the old server. Within the drupal root directory type: tar -zcvf example.com.tar.gz * .htaccess + + Note that we have to specify including the .htaccess because it isn't picked up by the * wildcard due to being prefixed with a full stop. It is important to bring along the .htaccess for a platform so that Aegir can consider these settings when it generates Apache configuration for this platform. + + Then, on the new server you can create a directory for it in the codebase you want to put it under: + + cd /var/aegir/platforms + mkdir your-new-platform + cd your-new-platform + wget http://example.com/example.com.tar.gz + tar -zxvf example.com.tar.gz + + Instead of using wget you could of course use SCP or FTP to transfer the tarball onto the Aegir server. + + Once the file is unpacked, check the ownership and group of each file by using ls -al. It should be either 'aegir:aegir' or 'aegir:www-data'. Change it if necessary. We have to assume you have basic knowledge of Linux and POSIX permissions - we can't be responsible for teaching you sysadmin skills :) + + In particular, files that may have been uploaded on your site via modules like imagecache, upload, Profile pics etc, may need to have their ownership changed: sudo chown -R aegir $platform/sites/*/files + + If your site previously resided in sites/defau1t you need to move it because Aegir ignores the default directory (it has no way of understanding what URL this would be accessed by, so it is impossible to 'manage' it). Each site needs its own directory with the correct domain in typical Drupal multisite design: + + mv sites/defau1t sites/example.com + + If you do this, you may need to also update your filepath's stored in the files table by running the following query on your database: + + UPDATE files SET filepath = REPLACE(filepath, 'sites/defau1t', 'sites/example.com'); + + There are many other places in the database that can suffer the same problem, the node_revisions table for example. If you have phpMyAdmin available you can easily search the entire database for 'sites/defau1t'. Select the database, then click the Search tab. Once you know which tables are affected you can run a variation of the above query to correct these records. + +2. Transfer the Database + + Note: This step is not necessary if you are moving your site from one directory on your server (e.g., /var/www/html) to a newly created Aegir directory on the same server (e.g., /var/aegir). + + Make a backup of the database on your old server Backup and Migrate module is good for this, or you can use phpmyadmin or your favourite MySQL management tool. It's a good idea to truncate the cache, search and watchdog tables first to reduce the size of the database (Backup and Migrate does this for you). + + Aegir does not provide support for importing databases with prefixed tables so it is best to remove prefixes form the database. However, a [workaround is available](http://drupal.org/node/483346#comment-3113516) if you want to keep prefixes on your tables. + + Unlike traditional Drupal projects the database settings cannot be found in settings.php. The database credentials are stored in the Apache vhost config of the associated site. This is a security measure implemented by the Aegir project. + + In the vhost directory on your old server, /var/aegir/config/server_master/apache/vhost.d/, there is a vhost file for every site on your server. The contents of the file can be displayed in your terminal with the command + + cat thenameofthefile + + At the top of the file you will find the credentials needed to create a database on the new server. + + On your new server, manually create a new database and upload the .sql file from your backup. + + Then create the mysql user that your site accesses the database as, and grant it all permissions on that database except 'GRANT'. + + Connect to the database on the old server: + + mysql -u root -p + + Create the new database: + + create database yourdatabasename + + Add the user: + + CREATE USER 'yourdatabaseuser'@'localhost' identified by 'youruserpassword'; + + Give the necessary privileges: + + GRANT ALL PRIVILEGES ON yourdatabasename.* TO 'yourdatabaseuser'@'localhost'; + + Import the database: + mysql -u yourdatabaseuser -p -h localhost yourdatabasename < thenameofthedatabase.sql + +3. Setup Platform in Aegir + + In the Aegir web interface click on 'Create Content' and 'Platform'. This step is often easiest when using the admin-menu on the top of your screen. + + Enter the name you want to use for the platform (eg 'My Platform import'), and the path to the platform on the server (eg '/var/aegir/platforms/your-new-platform') + + Click Save to submit the form, and Aegir will spawn a Verify task for this platform into the queue. + + Once the task is complete this the task should turn green (click through to the homepage to refesh and check. If not, see 'Troubleshooting' below). + +4. Import and Verify Sites In Aegir + + Upon completion of the above Platform verification task, Aegir will discover your new site and spawn a new 'Import' task for the site. + + Once it's done this (it may take a couple of cron runs to dispatch these tasks in the queue) the tasks should turn green (if not, see 'Troubleshooting' below). + + At this point, presuming you have updated DNS or are overriding DNS via an entry in your /etc/hosts file to access the site via the new Aegir server, you can visit site in a browser and check around to see that it has worked. Pay particular attention to any links in node content that pointed to paths referencing /sites/community.aegirproject.org if your site was not part of a multisite setup. (Drupal has a habit of storing these paths in the database, or they may have been hard-coded into nodes by users). + + It's a good idea to clear the caches, and you may need to get imagecache to rebuild its thumbnails if you use it. + +5. Migrate To Another Platform + + Now that you have your sites under Aegir's control you can take advantage of its power, and easily migrate them to another platform. In the event that your sites were on old versions of Drupal core and a new one is available and present as another Platform on your Aegir server, you can use the Migrate process to upgrade those sites onto the new core. + + For details on how to Migrate sites, consult the [Migrate documentation](@todo). \ No newline at end of file diff --git a/docs/import/importing-sites.md b/docs/import/importing-sites.md index 6b132eb..d59b4e9 100644 --- a/docs/import/importing-sites.md +++ b/docs/import/importing-sites.md @@ -7,6 +7,6 @@ These instructions assume you've set up a new server with Aegir on it, and you w There are three standard ways of importing sites into Aegir: -* from [Aegir backups](import/aegir-backups)Aegir backups - simplest and fastest, scriptable -* from any [single site](import/single-site) manually - most reliable, can import even non-aegir sites, hard to automate -* from a [complete drupal install](import/complete-drupal-install) - fairly reliable, can import non-aegir sites, could be automated but requires access to the database servers currently in use by the site +* from [Aegir backups](aegir-backups)Aegir backups - simplest and fastest, scriptable +* from any [single site](single-site) manually - most reliable, can import even non-aegir sites, hard to automate +* from a [complete drupal install](complete-drupal-install) - fairly reliable, can import non-aegir sites, could be automated but requires access to the database servers currently in use by the site diff --git a/docs/import/single-site.md b/docs/import/single-site.md index e69de29..c12c231 100644 --- a/docs/import/single-site.md +++ b/docs/import/single-site.md @@ -0,0 +1,89 @@ +Importing a single site manually +================================ + +If you already have a platform setup on Aegir with EXACTLY the same codebase as your existing site, then you don't need to transfer the entire old codebase - you can just transfer the sites/example.com directory. However, you also need to make sure any dependencies on contrib modules that your site has, are covered on the codebase or Platform that you're importing it into. + +In general it may be considered safer/less prone to confusing errors to [transfer the entire old codebase](complete-drupal-install) into Aegir as a whole Platform, whereby the site will be imported automatically under that Platform. You can then migrate it in Aegir to one of your existing Platforms later. + +Also, it is much faster to just "deploy" an Aegir backup than go through the manual procedure here, so you should generally follow that procedure instead of the one defined here unless you have a very hairy setup. + +To transer a site manually into Aegir, those are the steps you should follow: + +Table of Contents + + 1. Create the site database, user and directory + 2. Transfer the files + 3. Transfer the Database + 4. Verify In Aegir + +1. Create the site database, user and directory + +In order to import the site, you need to create a database and database user for the site, along with a directory in the platform. + +The simplest way to do this is to create an empty site in the right platform and overwrite it. Through the commandline: + + drush provision-save @example.com --context_type=site --platform=@platform_d6 --uri=example.com --db_server=@server_localhost --client_name=admin + + drush provision-install @example.com + +Through the UI, this can be done simply by creating a site in Create content. + +If this is not working for some reason, you can create the mysql user and database manually using the following: + + GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES, CREATE TEMPORARY TABLES ON databasename.* TO 'databaseuser'@'localhost' IDENTIFIED BY 'password'; + + CREATE DATABASE databasename; + +2. Transfer the files + +You need to copy the sites/example.com directory from your old server to the new Aegir server. + +To do this, first create a tarball of the site on the old server. Within the sites/example.com directory type: + + tar -zcvf example.com.tar.gz . + +Then, on the Aegir server, you can create a directory for it in the codebase (platform) you want to put it under: + + cd /var/aegir/platforms/drupal-X.Y/sites + mkdir example.com + cd example.com + wget http://example.com/example.com.tar.gz + tar -zxvf example.com.tar.gz ./files ./modules ./libraries ./themes + +Instead of using wget you could of course use SCP or FTP to transfer the tarball onto the Aegir server. + +In the above we assume that you have created an empty site as directed above, if not, you will also need to extract the settings.php file, so that last command should instead be simply: + + tar -zxvf example.com.tar.gz + +Then you will need to modify the settings.php file to follow the user and database created in the first step. Again, this is not necessary if you let Aegir create the site for you. + +Once the file is unpacked, check the ownership and group of each file by using ls -al. It should be either 'aegir:aegir' or 'aegir:www-data'. Change it if necessary. In particular, files that may have been uploaded on your site via modules like imagecache, upload, Profile pics etc, may need to have their ownership changed: + + sudo chown -R aegir:www-data $platform/sites/*/files + +3. Transfer the Database + +Make a backup of the database on your old server Backup and Migrate module is good for this, or you can use phpmyadmin or your favourite MySQL management tool. It's a good idea to truncate the cache, search and watchdog tables first to reduce the size of the database. The "Backup and Migrate" module does this for you. + +If you are importing a site from a previous Aegir installation somewhere, such as from a backup tarball, the database.sql will already be within the tarball you unpacked. You can use this to import into a new database on your new server. + +You can import the database with drush using the following command: + + drush @example.com sqlc < database.sql + +4. Verify In Aegir + +In the Aegir web interface click on the name of the platform you have added your site to. You can access the list of platforms via the 'Platforms' tab in the primary links menu. + +Execute a new Verify task on this platform via the 'Verify' button in the task list. + +Aegir will now re-verify the platform. In the process of this, it will discover your new site and spawn a new 'Import' task for the site. + +Once it's done this (it may take a couple of cron runs to dispatch these tasks in the queue) the tasks should turn green. + +At this point, presuming you have updated DNS or are overriding DNS via an entry in your /etc/hosts file to access the site via the new Aegir server, you can visit the site in a browser and check around to see that it has worked. Pay particular attention to any links in node content that pointed to paths referencing /sites/community.aegirproject.org if your site was not part of a multisite setup. (Drupal has a habit of storing these paths in the database, or they may have been hard-coded into nodes by users). + +Aegir will not go through all your database and update all URLs, so some images or links may be broken. This will need to be done manually. Aegir does attempt to update paths stored in tables such as 'files' though. + +It's a good idea to clear the caches, and you may need to get imagecache to rebuild its thumbnails if you use it. \ No newline at end of file From 7f50090de979dfcbedf970dac2fe331b66d71889 Mon Sep 17 00:00:00 2001 From: Max Bronsema Date: Mon, 8 Jun 2015 17:36:12 -0700 Subject: [PATCH 4/4] Modified documentation to be aligned with https://www.drupal.org/node/2500095. --- docs/import/aegir-backups.md | 0 docs/import/importing-sites.md | 2 +- docs/import/remote-import.md | 4 ++++ mkdocs.yml | 2 +- 4 files changed, 6 insertions(+), 2 deletions(-) delete mode 100644 docs/import/aegir-backups.md create mode 100644 docs/import/remote-import.md diff --git a/docs/import/aegir-backups.md b/docs/import/aegir-backups.md deleted file mode 100644 index e69de29..0000000 diff --git a/docs/import/importing-sites.md b/docs/import/importing-sites.md index d59b4e9..2a2dcf8 100644 --- a/docs/import/importing-sites.md +++ b/docs/import/importing-sites.md @@ -7,6 +7,6 @@ These instructions assume you've set up a new server with Aegir on it, and you w There are three standard ways of importing sites into Aegir: -* from [Aegir backups](aegir-backups)Aegir backups - simplest and fastest, scriptable +* from [Remote Import](remote-import)Aegir backups - simplest and fastest, scriptable * from any [single site](single-site) manually - most reliable, can import even non-aegir sites, hard to automate * from a [complete drupal install](complete-drupal-install) - fairly reliable, can import non-aegir sites, could be automated but requires access to the database servers currently in use by the site diff --git a/docs/import/remote-import.md b/docs/import/remote-import.md new file mode 100644 index 0000000..2455413 --- /dev/null +++ b/docs/import/remote-import.md @@ -0,0 +1,4 @@ +Remote Import +============= + +@todo Document how this works once [Remote Import](https://www.drupal.org/project/hosting_remote_import) is officially part of 3.x via [Golden Contrib](https://www.drupal.org/node/2180053) \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index acb28be..e3e35ed 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,6 +10,6 @@ pages: - ['install/manual.md', 'Install', 'Manual'] - ['install/common-problems.md', 'Install', 'Common Problems'] - ['import/importing-sites.md', 'Importing Sites'] -- ['import/aegir-backups.md', 'Importing Sites', 'From Aegir Backups'] +- ['import/remote-import.md', 'Importing Sites', 'Remote Import'] - ['import/single-site.md', 'Importing Sites', 'Single Site Import'] - ['import/complete-drupal-install.md', 'Importing Sites', 'Complete Drupal Install']