Db import speed up pr devel (#290)

* Defer nightly backups, disable ASG processes during syncs and run syncs with backup (#94)

* Change location of nightly backup script and delegate the cron that runs it to the deploy server, if required (for ASGs).

* Set up nightly backup crons in separate files in /etc/cron.d

* Can't put site cron files in /etc/cron.d because the deploy user doesn't have perms.

* Try and add the ability to sync a site using a nightly backup instead of a fresh DB dump.

* Use Ansible modules to look up RDS host and to copy the nightly backup into place.

* Delegate PATH setup in db backup cron to localhost.

* Shell bad. Command good. But makes it convoluted. Oh well.

* Used wrong database name in source database copy.

* Try and disable the ReplaceUnhealthy auto scale process during syncs.

* Fix Drupal cron roles when deferring to deploy server.

* Added deploy.yml examples for Drupal 9 and Localgov. Updated tests (#97)

* Adding a new SimpleSAMLphp meta role. (#100)

* Allowing users to set cachetool version properly. (#102)

* Deploy ami pr 1.x (#106)

* Including an ami.yml for packing AMIs on build finale.

* New api_call role, focused on GitLab for now.

* Optionally trigger an infra build with an API call.

* Making the MySQL dump command for routine back-ups less aggressive. (#104)

* Making the MySQL dump command for routine back-ups less aggressive.

* Making max_allowed_packets a variable we can set.

* Adding dump command to a re-usable central var.

* Renaming _init var so it's easier to distinguish in the code.

* Fix database backups pr 1.x (#109)

* Making the MySQL dump command for routine back-ups less aggressive.

* Making max_allowed_packets a variable we can set.

* Adding dump command to a re-usable central var.

* Renaming _init var so it's easier to distinguish in the code.

* Defaults file must be the first param for mysqldump.

* Fix MySQL backup deferral. (#110)

* Files recurse fix pr 1.x (#112)

* Don't recurse through site directory when setting permissions during config_generate step.

* Update drupal7 config_generate perms update task and use true/false instead of yes/no.

* Improve multisite support (#115)

* Pass -l option in drush commands to specify site name, which should match the site folder name.

* Move -l option in drush commands to the first option instead of the last.

* Static credentials handling fix pr 1.x (#119)

* Attempt to fix static credentials handling when deploying to an ASG.

* Include build_type in static password file location.

* Move to using new location for static credentials handling, but try to catch any legacy stuff too, for now.

* Add a couple of comments to legacy handling of static credentials.

* Making contents of deploy tar 'ownerless'. (#117)

* Implement file syncing (#124)

* Add a files_sync role.

* Clear up some comments, add temp_dir variable and use rsync with command module instead of synchronize module, in files_sync role.

* Need trailing slash on src when syncing files to destination server so the contents get synced and not the parent directory itself.

* Use rsync instead of copy when syncing files to target server as copy is so slow.

* Ensure file sync tasks are run only once.

* Create Drupal-specific sync roles (#128)

* Add a drupal_sync_tasks role to run Drupal specific tasks during syncs.

* Move sync database_apply files into tasks subdirectory.

* Woops, used bad role names in sync database update roles.

* Remove feature reverting from drupal7 database_apply sync role.

* Fixing GRANT query for MySQL > 8.0. (#131)

* Fixing GRANT query for MySQL > 8.0.

* Create what though?

* Use IF NOT EXISTS when creating database user as that command fails if static database users are being used. (#133)

* Attempt to fix syncs whenever the 'dump' type is used for source or target. (#149)

* Squashfs pr 1.x (#150)

* Adding SquashFS option to syncing.

* Make 'tarball' the default mount type so nothing existing breaks.

* Altering deploy_path so we can build in another location for SquashFS builds.

* Ensuring the build dir exists if doing a SquashFS build.

* Force symlink creation as the deploy dir may not exist yet.

* Running mount commands with sudo.

* Tweaking link command and destination for live links.

* Slight bug in link path handling for SquashFS.

* Checking for existing mount and using remount operation.

* Stop ce-deploy trying to delete from read-only SquashFS mount.

* Formatting error, these special jinja2 things are not filters - no spaces.

* Reloading services to make sure mounting doesn't fail.

* Picking more sensible loop var name.

* Moving to shell for mount check and fixing jinja2 filter names.

* Trying with the posix mount module instead of command.

* Working through user/sudo issues.

* Altering symlink handling slightly so we have the deploy directory always set.

* Simplifying 'when' checks.

* Check deploy_code.mount_type is defined when setting facts in init role. (#155)

* Make config imports during syncs optional (#157)

* Use a different variable for config imports during a sync so they can be optional.

* Actually, no. Use a variable to determine if config should be imported during a sync.

* Fix some logic with config import during syncs and add a comment to explain when the task runs.

* Squashfs pr 1.x (#153)

* Adding SquashFS option to syncing.

* Make 'tarball' the default mount type so nothing existing breaks.

* Altering deploy_path so we can build in another location for SquashFS builds.

* Ensuring the build dir exists if doing a SquashFS build.

* Force symlink creation as the deploy dir may not exist yet.

* Running mount commands with sudo.

* Tweaking link command and destination for live links.

* Slight bug in link path handling for SquashFS.

* Checking for existing mount and using remount operation.

* Stop ce-deploy trying to delete from read-only SquashFS mount.

* Formatting error, these special jinja2 things are not filters - no spaces.

* Reloading services to make sure mounting doesn't fail.

* Picking more sensible loop var name.

* Moving to shell for mount check and fixing jinja2 filter names.

* Trying with the posix mount module instead of command.

* Working through user/sudo issues.

* Altering symlink handling slightly so we have the deploy directory always set.

* Simplifying 'when' checks.

* Adding a revert behaviour for SquashFS.

* Making some wording a little less ambiguous.

* Fixing bad image path and adding clauses to stat check.

* Trying to make remounting SquashFS images a bit safer.

* Adding when clauses to SquashFS image path check.

* Removing revert code, as it cannot work.

* Add cache clears to Drupal deployments, before DB updates and stuff (#159)

* Add cache clears to Drupal deployments, before database updates and config import are run.

* Move cache clears out of meta roles and only run in database_apply role on non-initial builds.

* Add cache clear stuff to drupal7 database_apply role.

* Avoid leaving exponentially growing sqsh files in build locations! (#164)

* Avoid leaving exponentially growing sqsh files in build locations!

* Use mount point instead of /tmp for stowing previous sqsh file.

* Exclude sqsh file pr 1.x (#167)

* Avoid leaving exponentially growing sqsh files in build locations!

* Use mount point instead of /tmp for stowing previous sqsh file.

* Making sure image/tarball filenames are build specific in /tmp.

* Removing unnecessary lines in Drupal config generation. (#169)

* Removing unnecessary lines in Drupal config generation.

* Drupal 7 lacked the install_command var.

* Ensuring dump directory exists on backup step. (#172)

* Ensuring dump directory exists on backup step.

* Acts on shared storage, so can run once.

* Allowing Drupal 7 jobs to disable cron. (#174)

* Suppress db revert pr 1.x (#177)

* Supporting option to suppress reverting backups.

* Fixing namespacing in database_backup role.

* Fixing bad assumption that databases will have TCP connections. (#179)

* Handling the 'drush deploy' command more elegantly for Drupal 8+. (#180)

* Handling the 'drush deploy' command more elegantly for Drupal 8+.

* Changing logic to AND for cache rebuild.

* Attempt to clear the opcache during Drupal deployments. (#182)

Co-authored-by: Emlyn Kinzett <emlyn.kinzett@codeenigma.com>

* Better drush deploy support pr 1.x (#185)

* Handling the 'drush deploy' command more elegantly for Drupal 8+.

* Changing logic to AND for cache rebuild.

* Em's clear cache code needs to run for each 'site' too.

* Cron job schedule params pr 1.x (#190)

* cron job schedule params, namespaces

* drupal7-cron-params

* cron-job-schedule-params-fix-defaults

* Adding option to stop services that might interfere with a squashfs mount. (#193)

* Drush refactor pr 1.x (#197)

* Updating docs.

* Consistent drush handling in cron.

* Linting cruft and removing unimplemented drush.yml handling from D7.

* Refactor drush role to support new GitHub release format.

* Updating drush README.

* Casting the first part of the drush version string as an integer.

* Refactoring integer handling.

* Major version string will always be a string, assuming below 8 will never exist.

* Downloaded drush archive is remote, not local.

* Refactoring handling of .bin directory.

* Supporting drush-launcher.

* Removing drush installation support for Drupal 8 and above.

* Bad variable name in Drupal cron jobs.

* Fixing maintenance mode command linting and consistency.

* Moving chdir to args to see if it helps.

* Trying an actual 'cd' command inline.

* Cannot '&&' in a cmd.

* Trying drush's --root flag to set Drupal path.

* With --root we do not need chdir for drush.

* Revert "With --root we do not need chdir for drush."

This reverts commit d07aaf7.

* Better deploy_code role docs. (#213)

* Better deploy_code role docs.

* roles path error in docs.

* roles path error in docs.

* Adding a note about deploy_previous handling for squashfs.

* Cachetool settings and drupal7 tasks fix pr 1.x (#211)

* cachetool_settings_and_drupal7_tasks_fix

* cachetool_settings_and_drupal7_tasks_fix

* cachetool_settings_and_drupal7_tasks_fix

* cachetool settings fixing condition and namespace

* fixing cachetool adapter options

* fixed var

* fixed var

* fixed var

* previous_cachetool_remove_before_install_new (#224)

* previous_cachetool_remove_before_install_new

* namespace fix

* Fix cachetool removal pr 1.x (#227)

* fix_cachetool_removal_step

* fix task name

* Fix cachetool removal pr 1.x (#229)

* fix_cachetool_removal_step

* fix task name

* fix_cachetool_removal_task

* fix_cachetool_removal_task

* Documentation enhancements pr 1.x (#218)

* Better deploy_code role docs.

* roles path error in docs.

* roles path error in docs.

* Adding a note about deploy_previous handling for squashfs.

* Reference incorrect role for deploy user sudo perms.

* Minor edits to frontpage README.

* Rebuilt docs.

* Accidentally overwrote docs change.

* Drush refactor pr 1.x (#231)

* Fixing sync jobs.

* Fixing bad task title.

* Drush refactor pr 1.x (#234)

* Fixing sync jobs.

* Fixing bad task title.

* Trying to fix D7 drush issues with chdir.

* Drush refactor pr 1.x (#236)

* Fixing sync jobs.

* Fixing bad task title.

* Trying to fix D7 drush issues with chdir.

* Moving D7 drush commands back to 'shell'.

* Change php detection for fastcgi and cachetool pr 1.x (#238)

* change_php_detection_for_fastcgi_and_cachetool

* fix_typo

* change_php_detection_for_fastcgi_and_cachetool_no_sudo

* Cachetool bin per site pr 1.x (#243)

* cachetool_bin_per_project

* cachetool_bin_per_site_fix

* cachetool_bin_per_site_fix2

* apcu no longer has an 'all' option. (#245)

* Refactoring cachetool path setting to match other squashfs vars. (#247)

* Cachetool pr 1.x (#249)

* Refactoring cachetool path setting to match other squashfs vars.

* Refactoring cachetool_bin handling.

* Ensuring we can prevent features and ctools deployments in Drupal 7. (#240)

* Cachetool pr 1.x (#251)

* Refactoring cachetool path setting to match other squashfs vars.

* Refactoring cachetool_bin handling.

* Forgot to remove the default() filter from squashfs var setting.

* Refactoring clauses slightly to simplify.

* improving_old_builds_cleanup (#254)

* improving_old_builds_cleanup

* improving_old_builds_cleanup_fix

* Adding escaped backticks to db names to be safe. (#256)

* Mysql db name pr 1.x (#258)

* Adding escaped backticks to db names to be safe.

* Looks like Ansible auto-escapes backticks.

* Mysql db name pr 1.x (#260)

* Adding escaped backticks to db names to be safe.

* Looks like Ansible auto-escapes backticks.

* Adding backticks to database names for creating MySQL users too.

* Mysql db name pr 1.x (#262)

* Adding escaped backticks to db names to be safe.

* Looks like Ansible auto-escapes backticks.

* Adding backticks to database names for creating MySQL users too.

* Adding extra mysqldump flags to stop restores failing on RDS.

* Mysql db name pr 1.x (#264)

* Adding escaped backticks to db names to be safe.

* Looks like Ansible auto-escapes backticks.

* Adding backticks to database names for creating MySQL users too.

* Adding extra mysqldump flags to stop restores failing on RDS.

* Removing ignore_errors.

* Updating mysqldump query.

* Bug fixes pr 1.x (#266)

* Fixing linting so CI can work.

* More linting fixes.

* Downgrading Ubuntu due to Docker issues with ce-dev and latest.

* Making pipefail shell commands use /bin/bash.

* Adding pipefail code to cachetool installer block.

* Adding executable for drush install checker.

* Bug fixes pr 1.x (#268)

* Fixing linting so CI can work.

* More linting fixes.

* Downgrading Ubuntu due to Docker issues with ce-dev and latest.

* Making pipefail shell commands use /bin/bash.

* Adding pipefail code to cachetool installer block.

* Adding executable for drush install checker.

* Fixing D7 ctools behaviour to match features handling.

* Making sync paths build unique.

* Adding --host parameter to build.sh for pre-deploy host checking with Ansible. (#271)

* Making syncs safer and more efficient. (#273)

* Bug fixes pr 1.x (#278)

* Fixing linting so CI can work.

* More linting fixes.

* Downgrading Ubuntu due to Docker issues with ce-dev and latest.

* Making pipefail shell commands use /bin/bash.

* Adding pipefail code to cachetool installer block.

* Adding executable for drush install checker.

* Fixing D7 ctools behaviour to match features handling.

* Jinja2 template looking for settings.php using wrong path.

* Fixing config_generate for Drupal.

* Making host checking more robust.

* Removing variable declaration that will break host check.

* Allow us to use deploy_code with a completely custom build and do nothing.

* Bug fixes pr 1.x (#282)

* Fixing linting so CI can work.

* More linting fixes.

* Downgrading Ubuntu due to Docker issues with ce-dev and latest.

* Making pipefail shell commands use /bin/bash.

* Adding pipefail code to cachetool installer block.

* Adding executable for drush install checker.

* Fixing D7 ctools behaviour to match features handling.

* Jinja2 template looking for settings.php using wrong path.

* Fixing config_generate for Drupal.

* Making host checking more robust.

* Removing variable declaration that will break host check.

* Allow us to use deploy_code with a completely custom build and do nothing.

* Make PHP cachetool install optional.

* Bug fixes pr 1.x (#284)

* Fixing linting so CI can work.

* More linting fixes.

* Downgrading Ubuntu due to Docker issues with ce-dev and latest.

* Making pipefail shell commands use /bin/bash.

* Adding pipefail code to cachetool installer block.

* Adding executable for drush install checker.

* Fixing D7 ctools behaviour to match features handling.

* Jinja2 template looking for settings.php using wrong path.

* Fixing config_generate for Drupal.

* Making host checking more robust.

* Removing variable declaration that will break host check.

* Allow us to use deploy_code with a completely custom build and do nothing.

* Make PHP cachetool install optional.

* Fixing bug introduced by failed_when - need to check for return code on mount points.

* Bug fixes pr 1.x (#287)

* Fixing linting so CI can work.

* More linting fixes.

* Downgrading Ubuntu due to Docker issues with ce-dev and latest.

* Making pipefail shell commands use /bin/bash.

* Adding pipefail code to cachetool installer block.

* Adding executable for drush install checker.

* Fixing D7 ctools behaviour to match features handling.

* Jinja2 template looking for settings.php using wrong path.

* Fixing config_generate for Drupal.

* Making host checking more robust.

* Removing variable declaration that will break host check.

* Allow us to use deploy_code with a completely custom build and do nothing.

* Make PHP cachetool install optional.

* Fixing bug introduced by failed_when - need to check for return code on mount points.

* Moving where opcache clears get called so they can be excluded if necessary.

* db_import_speed_up

* db_import_speed_up_fixed

* db_import_speed_up_fix2

* db_import_speed_up_fix3

---------

Co-authored-by: EmlynK <emlyn.kinzett@codeenigma.com>
Co-authored-by: Dionisio <dionisiofernandez83@gmail.com>
Co-authored-by: Greg Harvey <greg.harvey@gmail.com>

Author: 4 people

roles/database_backup/database_backup-mysql/tasks/revert-dump.yml

@@ -1,7 +1,21 @@
 ---
+- name: Unpack dump file.
+  ansible.builtin.unarchive:
+    src: "{{ mysql_backup.dumps_directory }}/{{ _mysql_host }}/{{ database.database }}-{{ previous_build_number }}.sql.bz2"
+    dest: "/tmp/{{ database.database }}-{{ previous_build_number }}.sql"
+  run_once: true
+  when: previous_build_number > 0
+
 - name: Revert database from dump.
-  ansible.builtin.shell: "set -o pipefail && bzcat {{ mysql_backup.dumps_directory }}/{{ _mysql_host }}/{{ database.database }}-{{ previous_build_number }}.sql.bz2 | mysql --defaults-extra-file={{ database.credentials_file }} {{ database.database }}"
+  ansible.builtin.shell: "mysql --defaults-extra-file={{ database.credentials_file }} {{ database.database }} < /tmp/{{ database.database }}-{{ previous_build_number }}.sql"
   args:
     executable: /bin/bash
+  run_once: true
   when: previous_build_number > 0
+
+- name: Delete unpacked dump file.
+  ansible.builtin.file:
+    path: "/tmp/{{ database.database }}-{{ previous_build_number }}.sql"
+    state: absent
   run_once: true
+  when: previous_build_number > 0

roles/sync/database_sync/database_sync-mysql/tasks/sync.yml

@@ -93,6 +93,10 @@
   ansible.builtin.set_fact:
     mysql_sync_target_dump_path: "/tmp/{{ database.target.database }}_{{ build_number }}_target.sql.bz2"
 
+- name: Register tmp unpacked target dump name.
+  ansible.builtin.set_fact:
+    mysql_sync_target_dump_unpacked_path: "/tmp/{{ database.target.database }}_{{ build_number }}_target.sql"
+
 - name: Get target last known good build number.
   ansible.builtin.command:
     argv:
@@ -126,6 +130,16 @@
     src: "{{ _ce_deploy_build_tmp_dir }}/{{ database.target.database }}.sql.bz2"
     dest: "{{ mysql_sync_target_dump_path }}"
 
+- name: Unpack dump file.
+  ansible.builtin.unarchive:
+    src: "{{ mysql_sync_target_dump_path }}"
+    dest: "{{ mysql_sync_target_dump_unpacked_path }}"
+
+- name: Delete temporary dump file on target.
+  ansible.builtin.file:
+    path: "{{ mysql_sync_target_dump_path }}"
+    state: absent
+
 - name: Drop target database.
   ansible.builtin.command:
     cmd: "mysql --defaults-extra-file={{ database.target.credentials_file }} -e 'drop database if exists {{ mysql_sync_target_database }};'"
@@ -135,13 +149,13 @@
     cmd: "mysql --defaults-extra-file={{ database.target.credentials_file }} -e 'create database {{ mysql_sync_target_database }};'"
 
 - name: Repopulate database from dump.
-  ansible.builtin.shell: "set -o pipefail && bzcat {{ mysql_sync_target_dump_path }} | mysql --defaults-extra-file={{ database.target.credentials_file }} {{ mysql_sync_target_database }}"
+  ansible.builtin.shell: "mysql --defaults-extra-file={{ database.target.credentials_file }} {{ mysql_sync_target_database }} < {{ mysql_sync_target_dump_unpacked_path }}"
   args:
     executable: /bin/bash
 
-- name: Delete temporary dump file on target.
+- name: Delete temporary unpacked dump file on target.
   ansible.builtin.file:
-    path: "{{ mysql_sync_target_dump_path }}"
+    path: "{{ mysql_sync_target_dump_unpacked_path }}"
     state: absent
 
 - name: Delete temporary dump file on source.