Skip to content

Commit 4e17c10

Browse files
gregharveyEmlynKDionisioFGtymofiisobchenko
authored
Ecs deployments pr devel (#299)
* Defer nightly backups, disable ASG processes during syncs and run syncs with backup (#94) * Change location of nightly backup script and delegate the cron that runs it to the deploy server, if required (for ASGs). * Set up nightly backup crons in separate files in /etc/cron.d * Can't put site cron files in /etc/cron.d because the deploy user doesn't have perms. * Try and add the ability to sync a site using a nightly backup instead of a fresh DB dump. * Use Ansible modules to look up RDS host and to copy the nightly backup into place. * Delegate PATH setup in db backup cron to localhost. * Shell bad. Command good. But makes it convoluted. Oh well. * Used wrong database name in source database copy. * Try and disable the ReplaceUnhealthy auto scale process during syncs. * Fix Drupal cron roles when deferring to deploy server. * Added deploy.yml examples for Drupal 9 and Localgov. Updated tests (#97) * Adding a new SimpleSAMLphp meta role. (#100) * Allowing users to set cachetool version properly. (#102) * Deploy ami pr 1.x (#106) * Including an ami.yml for packing AMIs on build finale. * New api_call role, focused on GitLab for now. * Optionally trigger an infra build with an API call. * Making the MySQL dump command for routine back-ups less aggressive. (#104) * Making the MySQL dump command for routine back-ups less aggressive. * Making max_allowed_packets a variable we can set. * Adding dump command to a re-usable central var. * Renaming _init var so it's easier to distinguish in the code. * Fix database backups pr 1.x (#109) * Making the MySQL dump command for routine back-ups less aggressive. * Making max_allowed_packets a variable we can set. * Adding dump command to a re-usable central var. * Renaming _init var so it's easier to distinguish in the code. * Defaults file must be the first param for mysqldump. * Fix MySQL backup deferral. (#110) * Files recurse fix pr 1.x (#112) * Don't recurse through site directory when setting permissions during config_generate step. * Update drupal7 config_generate perms update task and use true/false instead of yes/no. * Improve multisite support (#115) * Pass -l option in drush commands to specify site name, which should match the site folder name. * Move -l option in drush commands to the first option instead of the last. * Static credentials handling fix pr 1.x (#119) * Attempt to fix static credentials handling when deploying to an ASG. * Include build_type in static password file location. * Move to using new location for static credentials handling, but try to catch any legacy stuff too, for now. * Add a couple of comments to legacy handling of static credentials. * Making contents of deploy tar 'ownerless'. (#117) * Implement file syncing (#124) * Add a files_sync role. * Clear up some comments, add temp_dir variable and use rsync with command module instead of synchronize module, in files_sync role. * Need trailing slash on src when syncing files to destination server so the contents get synced and not the parent directory itself. * Use rsync instead of copy when syncing files to target server as copy is so slow. * Ensure file sync tasks are run only once. * Create Drupal-specific sync roles (#128) * Add a drupal_sync_tasks role to run Drupal specific tasks during syncs. * Move sync database_apply files into tasks subdirectory. * Woops, used bad role names in sync database update roles. * Remove feature reverting from drupal7 database_apply sync role. * Fixing GRANT query for MySQL > 8.0. (#131) * Fixing GRANT query for MySQL > 8.0. * Create what though? * Use IF NOT EXISTS when creating database user as that command fails if static database users are being used. (#133) * Attempt to fix syncs whenever the 'dump' type is used for source or target. (#149) * Squashfs pr 1.x (#150) * Adding SquashFS option to syncing. * Make 'tarball' the default mount type so nothing existing breaks. * Altering deploy_path so we can build in another location for SquashFS builds. * Ensuring the build dir exists if doing a SquashFS build. * Force symlink creation as the deploy dir may not exist yet. * Running mount commands with sudo. * Tweaking link command and destination for live links. * Slight bug in link path handling for SquashFS. * Checking for existing mount and using remount operation. * Stop ce-deploy trying to delete from read-only SquashFS mount. * Formatting error, these special jinja2 things are not filters - no spaces. * Reloading services to make sure mounting doesn't fail. * Picking more sensible loop var name. * Moving to shell for mount check and fixing jinja2 filter names. * Trying with the posix mount module instead of command. * Working through user/sudo issues. * Altering symlink handling slightly so we have the deploy directory always set. * Simplifying 'when' checks. * Check deploy_code.mount_type is defined when setting facts in init role. (#155) * Make config imports during syncs optional (#157) * Use a different variable for config imports during a sync so they can be optional. * Actually, no. Use a variable to determine if config should be imported during a sync. * Fix some logic with config import during syncs and add a comment to explain when the task runs. * Squashfs pr 1.x (#153) * Adding SquashFS option to syncing. * Make 'tarball' the default mount type so nothing existing breaks. * Altering deploy_path so we can build in another location for SquashFS builds. * Ensuring the build dir exists if doing a SquashFS build. * Force symlink creation as the deploy dir may not exist yet. * Running mount commands with sudo. * Tweaking link command and destination for live links. * Slight bug in link path handling for SquashFS. * Checking for existing mount and using remount operation. * Stop ce-deploy trying to delete from read-only SquashFS mount. * Formatting error, these special jinja2 things are not filters - no spaces. * Reloading services to make sure mounting doesn't fail. * Picking more sensible loop var name. * Moving to shell for mount check and fixing jinja2 filter names. * Trying with the posix mount module instead of command. * Working through user/sudo issues. * Altering symlink handling slightly so we have the deploy directory always set. * Simplifying 'when' checks. * Adding a revert behaviour for SquashFS. * Making some wording a little less ambiguous. * Fixing bad image path and adding clauses to stat check. * Trying to make remounting SquashFS images a bit safer. * Adding when clauses to SquashFS image path check. * Removing revert code, as it cannot work. * Add cache clears to Drupal deployments, before DB updates and stuff (#159) * Add cache clears to Drupal deployments, before database updates and config import are run. * Move cache clears out of meta roles and only run in database_apply role on non-initial builds. * Add cache clear stuff to drupal7 database_apply role. * Avoid leaving exponentially growing sqsh files in build locations! (#164) * Avoid leaving exponentially growing sqsh files in build locations! * Use mount point instead of /tmp for stowing previous sqsh file. * Exclude sqsh file pr 1.x (#167) * Avoid leaving exponentially growing sqsh files in build locations! * Use mount point instead of /tmp for stowing previous sqsh file. * Making sure image/tarball filenames are build specific in /tmp. * Removing unnecessary lines in Drupal config generation. (#169) * Removing unnecessary lines in Drupal config generation. * Drupal 7 lacked the install_command var. * Ensuring dump directory exists on backup step. (#172) * Ensuring dump directory exists on backup step. * Acts on shared storage, so can run once. * Allowing Drupal 7 jobs to disable cron. (#174) * Suppress db revert pr 1.x (#177) * Supporting option to suppress reverting backups. * Fixing namespacing in database_backup role. * Fixing bad assumption that databases will have TCP connections. (#179) * Handling the 'drush deploy' command more elegantly for Drupal 8+. (#180) * Handling the 'drush deploy' command more elegantly for Drupal 8+. * Changing logic to AND for cache rebuild. * Attempt to clear the opcache during Drupal deployments. (#182) Co-authored-by: Emlyn Kinzett <emlyn.kinzett@codeenigma.com> * Better drush deploy support pr 1.x (#185) * Handling the 'drush deploy' command more elegantly for Drupal 8+. * Changing logic to AND for cache rebuild. * Em's clear cache code needs to run for each 'site' too. * Cron job schedule params pr 1.x (#190) * cron job schedule params, namespaces * drupal7-cron-params * cron-job-schedule-params-fix-defaults * Adding option to stop services that might interfere with a squashfs mount. (#193) * Drush refactor pr 1.x (#197) * Updating docs. * Consistent drush handling in cron. * Linting cruft and removing unimplemented drush.yml handling from D7. * Refactor drush role to support new GitHub release format. * Updating drush README. * Casting the first part of the drush version string as an integer. * Refactoring integer handling. * Major version string will always be a string, assuming below 8 will never exist. * Downloaded drush archive is remote, not local. * Refactoring handling of .bin directory. * Supporting drush-launcher. * Removing drush installation support for Drupal 8 and above. * Bad variable name in Drupal cron jobs. * Fixing maintenance mode command linting and consistency. * Moving chdir to args to see if it helps. * Trying an actual 'cd' command inline. * Cannot '&&' in a cmd. * Trying drush's --root flag to set Drupal path. * With --root we do not need chdir for drush. * Revert "With --root we do not need chdir for drush." This reverts commit d07aaf7. * Better deploy_code role docs. (#213) * Better deploy_code role docs. * roles path error in docs. * roles path error in docs. * Adding a note about deploy_previous handling for squashfs. * Cachetool settings and drupal7 tasks fix pr 1.x (#211) * cachetool_settings_and_drupal7_tasks_fix * cachetool_settings_and_drupal7_tasks_fix * cachetool_settings_and_drupal7_tasks_fix * cachetool settings fixing condition and namespace * fixing cachetool adapter options * fixed var * fixed var * fixed var * previous_cachetool_remove_before_install_new (#224) * previous_cachetool_remove_before_install_new * namespace fix * Fix cachetool removal pr 1.x (#227) * fix_cachetool_removal_step * fix task name * Fix cachetool removal pr 1.x (#229) * fix_cachetool_removal_step * fix task name * fix_cachetool_removal_task * fix_cachetool_removal_task * Documentation enhancements pr 1.x (#218) * Better deploy_code role docs. * roles path error in docs. * roles path error in docs. * Adding a note about deploy_previous handling for squashfs. * Reference incorrect role for deploy user sudo perms. * Minor edits to frontpage README. * Rebuilt docs. * Accidentally overwrote docs change. * Drush refactor pr 1.x (#231) * Fixing sync jobs. * Fixing bad task title. * Drush refactor pr 1.x (#234) * Fixing sync jobs. * Fixing bad task title. * Trying to fix D7 drush issues with chdir. * Drush refactor pr 1.x (#236) * Fixing sync jobs. * Fixing bad task title. * Trying to fix D7 drush issues with chdir. * Moving D7 drush commands back to 'shell'. * Change php detection for fastcgi and cachetool pr 1.x (#238) * change_php_detection_for_fastcgi_and_cachetool * fix_typo * change_php_detection_for_fastcgi_and_cachetool_no_sudo * Cachetool bin per site pr 1.x (#243) * cachetool_bin_per_project * cachetool_bin_per_site_fix * cachetool_bin_per_site_fix2 * apcu no longer has an 'all' option. (#245) * Refactoring cachetool path setting to match other squashfs vars. (#247) * Cachetool pr 1.x (#249) * Refactoring cachetool path setting to match other squashfs vars. * Refactoring cachetool_bin handling. * Ensuring we can prevent features and ctools deployments in Drupal 7. (#240) * Cachetool pr 1.x (#251) * Refactoring cachetool path setting to match other squashfs vars. * Refactoring cachetool_bin handling. * Forgot to remove the default() filter from squashfs var setting. * Refactoring clauses slightly to simplify. * improving_old_builds_cleanup (#254) * improving_old_builds_cleanup * improving_old_builds_cleanup_fix * Adding escaped backticks to db names to be safe. (#256) * Mysql db name pr 1.x (#258) * Adding escaped backticks to db names to be safe. * Looks like Ansible auto-escapes backticks. * Mysql db name pr 1.x (#260) * Adding escaped backticks to db names to be safe. * Looks like Ansible auto-escapes backticks. * Adding backticks to database names for creating MySQL users too. * Mysql db name pr 1.x (#262) * Adding escaped backticks to db names to be safe. * Looks like Ansible auto-escapes backticks. * Adding backticks to database names for creating MySQL users too. * Adding extra mysqldump flags to stop restores failing on RDS. * Mysql db name pr 1.x (#264) * Adding escaped backticks to db names to be safe. * Looks like Ansible auto-escapes backticks. * Adding backticks to database names for creating MySQL users too. * Adding extra mysqldump flags to stop restores failing on RDS. * Removing ignore_errors. * Updating mysqldump query. * Bug fixes pr 1.x (#266) * Fixing linting so CI can work. * More linting fixes. * Downgrading Ubuntu due to Docker issues with ce-dev and latest. * Making pipefail shell commands use /bin/bash. * Adding pipefail code to cachetool installer block. * Adding executable for drush install checker. * Bug fixes pr 1.x (#268) * Fixing linting so CI can work. * More linting fixes. * Downgrading Ubuntu due to Docker issues with ce-dev and latest. * Making pipefail shell commands use /bin/bash. * Adding pipefail code to cachetool installer block. * Adding executable for drush install checker. * Fixing D7 ctools behaviour to match features handling. * Making sync paths build unique. * Adding --host parameter to build.sh for pre-deploy host checking with Ansible. (#271) * Making syncs safer and more efficient. (#273) * Bug fixes pr 1.x (#278) * Fixing linting so CI can work. * More linting fixes. * Downgrading Ubuntu due to Docker issues with ce-dev and latest. * Making pipefail shell commands use /bin/bash. * Adding pipefail code to cachetool installer block. * Adding executable for drush install checker. * Fixing D7 ctools behaviour to match features handling. * Jinja2 template looking for settings.php using wrong path. * Fixing config_generate for Drupal. * Making host checking more robust. * Removing variable declaration that will break host check. * Allow us to use deploy_code with a completely custom build and do nothing. * Bug fixes pr 1.x (#282) * Fixing linting so CI can work. * More linting fixes. * Downgrading Ubuntu due to Docker issues with ce-dev and latest. * Making pipefail shell commands use /bin/bash. * Adding pipefail code to cachetool installer block. * Adding executable for drush install checker. * Fixing D7 ctools behaviour to match features handling. * Jinja2 template looking for settings.php using wrong path. * Fixing config_generate for Drupal. * Making host checking more robust. * Removing variable declaration that will break host check. * Allow us to use deploy_code with a completely custom build and do nothing. * Make PHP cachetool install optional. * Bug fixes pr 1.x (#284) * Fixing linting so CI can work. * More linting fixes. * Downgrading Ubuntu due to Docker issues with ce-dev and latest. * Making pipefail shell commands use /bin/bash. * Adding pipefail code to cachetool installer block. * Adding executable for drush install checker. * Fixing D7 ctools behaviour to match features handling. * Jinja2 template looking for settings.php using wrong path. * Fixing config_generate for Drupal. * Making host checking more robust. * Removing variable declaration that will break host check. * Allow us to use deploy_code with a completely custom build and do nothing. * Make PHP cachetool install optional. * Fixing bug introduced by failed_when - need to check for return code on mount points. * Bug fixes pr 1.x (#287) * Fixing linting so CI can work. * More linting fixes. * Downgrading Ubuntu due to Docker issues with ce-dev and latest. * Making pipefail shell commands use /bin/bash. * Adding pipefail code to cachetool installer block. * Adding executable for drush install checker. * Fixing D7 ctools behaviour to match features handling. * Jinja2 template looking for settings.php using wrong path. * Fixing config_generate for Drupal. * Making host checking more robust. * Removing variable declaration that will break host check. * Allow us to use deploy_code with a completely custom build and do nothing. * Make PHP cachetool install optional. * Fixing bug introduced by failed_when - need to check for return code on mount points. * Moving where opcache clears get called so they can be excluded if necessary. * Adding verbose output for drush. (#289) * Adding container push and build code. --------- Co-authored-by: EmlynK <emlyn.kinzett@codeenigma.com> Co-authored-by: Dionisio <dionisiofernandez83@gmail.com> Co-authored-by: tymofiisobchenko <104431720+tymofiisobchenko@users.noreply.github.com>
1 parent 59aad1c commit 4e17c10

File tree

6 files changed

+130
-0
lines changed

6 files changed

+130
-0
lines changed

docs/_Sidebar.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
- [Data backups](/roles/database_backup)
3232
- [MySQL backups](/roles/database_backup/database_backup-mysql)
3333
- [Deploy](/roles/deploy_code)
34+
- [Deploy container](/roles/deploy_container)
3435
- [Init](/roles/_init)
3536
- [LHCI run](/roles/lhci_run)
3637
- ["Meta"](/roles/_meta)

docs/roles/deploy_container.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Deploy container
2+
Step that deploys the codebase in a Docker container image.
3+
4+
<!--TOC-->
5+
<!--ENDTOC-->
6+
7+
<!--ROLEVARS-->
8+
## Default variables
9+
```yaml
10+
---
11+
deploy_container:
12+
container_name: "example/example"
13+
container_tag: latest # tag will take format container_name:container_tag
14+
docker_registry_url: https://index.docker.io/v1/
15+
docker_registry_user: example
16+
docker_registry_pass: asdf1234
17+
docker_base_command: "docker image build"
18+
docker_build_dir: "{{ _ce_deploy_build_dir }}"
19+
environment_vars: {} # dictionary you can populate for use in a custom Dockerfile template
20+
# Requires the deploy IAM user to have the managed EC2InstanceProfileForImageBuilderECRContainerBuilds policy attached
21+
aws_ecr:
22+
enabled: false # set to true if using AWS ECR
23+
region: eu-west-1
24+
profile: example
25+
26+
```
27+
28+
<!--ENDROLEVARS-->

roles/deploy_container/README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Deploy container
2+
Step that deploys the codebase in a Docker container image. Requires Docker and the `community.docker` collection for Ansible to be installed on your deploy server. This can be handled by [`ce-provision`](https://github.com/codeenigma/ce-provision) using the `ce_deploy` and `docker_ce` roles.
3+
4+
AWS ECR registries require the AWS CLI user provided for `ce-deploy` to have the managed AWS `EC2InstanceProfileForImageBuilderECRContainerBuilds` policy attached via IAM to allow access to fetch credentials and push containers.
5+
6+
<!--TOC-->
7+
<!--ENDTOC-->
8+
9+
<!--ROLEVARS-->
10+
## Default variables
11+
```yaml
12+
---
13+
deploy_container:
14+
container_name: "example/example"
15+
container_tag: latest # tag will take format container_name:container_tag
16+
docker_registry_url: https://index.docker.io/v1/
17+
docker_registry_user: example
18+
docker_registry_pass: asdf1234
19+
docker_base_command: "docker image build"
20+
docker_build_dir: "{{ _ce_deploy_build_dir }}"
21+
environment_vars: {} # dictionary you can populate for use in a custom Dockerfile template
22+
# Requires the deploy IAM user to have the managed EC2InstanceProfileForImageBuilderECRContainerBuilds policy attached
23+
aws_ecr:
24+
enabled: false # set to true if using AWS ECR
25+
region: eu-west-1
26+
profile: example
27+
28+
```
29+
30+
<!--ENDROLEVARS-->
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
deploy_container:
3+
container_name: "example/example"
4+
container_tag: latest # tag will take format container_name:container_tag
5+
docker_registry_url: https://index.docker.io/v1/
6+
docker_registry_user: example
7+
docker_registry_pass: asdf1234
8+
docker_base_command: "docker image build"
9+
docker_build_dir: "{{ _ce_deploy_build_dir }}"
10+
environment_vars: {} # dictionary you can populate for use in a custom Dockerfile template
11+
# Requires the deploy IAM user to have the managed EC2InstanceProfileForImageBuilderECRContainerBuilds policy attached
12+
aws_ecr:
13+
enabled: false # set to true if using AWS ECR
14+
region: eu-west-1
15+
profile: example
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
# @TODO - for AWS ECR we'll need certain policies attaching to the deploy IAM user
3+
- name: Create Dockerfile from template.
4+
local_action:
5+
module: ansible.builtin.template
6+
src: Dockerfile.j2
7+
dest: "{{ deploy_container.docker_build_dir }}/Dockerfile"
8+
9+
- name: Set Docker registry username and password.
10+
ansible.builtin.set_fact:
11+
_docker_registry_username: "{{ deploy_container.docker_registry_user }}"
12+
_docker_registry_password: "{{ deploy_container.docker_registry_pass }}"
13+
14+
# Token valid for 12 hours
15+
- name: Fetch AWS ECR registry login token.
16+
ansible.builtin.command:
17+
command: "aws ecr get-login-password --region {{ deploy_container.aws_ecr.region }} --profile {{ deploy_container.aws_ecr.profile }}"
18+
when: deploy_container.aws_ecr.enabled
19+
register: _docker_registry_ecr_token
20+
21+
- name: Set AWS ECR registry password.
22+
ansible.builtin.set_fact:
23+
_docker_registry_password: "{{ _docker_registry_ecr_token.stdout }}"
24+
when: deploy_container.aws_ecr.enabled
25+
26+
- name: Set AWS ECR registry username.
27+
ansible.builtin.set_fact:
28+
_docker_registry_username: "AWS"
29+
when: deploy_container.aws_ecr.enabled
30+
31+
- name: Log into Docker registry.
32+
community.docker.docker_login:
33+
registry_url: "{{ deploy_container.docker_registry_url }}"
34+
username: "{{ _docker_registry_username }}"
35+
password: "{{ _docker_registry_password }}"
36+
reauthorize: true
37+
delegate_to: localhost
38+
when: deploy_container.docker_registry_login
39+
40+
- name: Build and push container image.
41+
community.docker.docker_image:
42+
build:
43+
path: "{{ deploy_container.docker_build_dir }}"
44+
repository: "{{ deploy_container.docker_registry_url }}"
45+
name: "{{ deploy_container.container_name }}"
46+
tag: "{{ deploy_container.container_tag | default('latest') }}"
47+
push: true
48+
source: build
49+
delegate_to: localhost
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Basic Dockerfile example
2+
FROM debian:bullseye-slim
3+
MAINTAINER sysadm@codeenigma.com
4+
5+
RUN apt-get update
6+
RUN apt-get install –y nginx
7+
CMD ["echo","Image created"]

0 commit comments

Comments
 (0)