feat: add support for restore dbs (#25)

* feat: add support for restore dbs * fix: remove flag --databases for restore * docs: add description variables * fix: update variables * fix: add clean app role and separate backups and restores * fix: add clean directory role * fix: remove clean_path role * fix: keys and remove int parse * fix: add default device * fix: small adjustment to available storage role * chore: adjustments to delete unnecessary variable from MySQL backup and restore roles * chore: not running a backup inside a restore. Deleting not needed variables * chore: changing MySQL dump parameters Check the topic here: https://dba.stackexchange.com/questions/271981/access-denied-you-need-at-least-one-of-the-process-privileges-for-this-ope to learn more about the tablespaces issue * feat: creating new playbook to implement automatic backup/restore * fix: using include_tasks since block does not support the loop parameter * fix: separate mongobackup from mongorestore * fix: replace variables and conditional for restore mongo databases * fix: remove task delete mongo database * docs: update comment for empty database case * docs: add doc for use ansible restore * docs: include atlas-ansible-inventory in docs * docs: update example * fix: remove unused variables * chore: removed unused files * feat: add numparallelcollections in mongorestore * feat: add mongo parallel variables * feat: add new role database_clone * fix: correct mongo variable * docs: add mongodocs reference and instructions for set variables * fix: move restore to main in restore roles and refactor check available storage * docs: update clone dbs documentation * docs: update readmd * fix: undefinided mysql_backup_file variable * docs: fix yaml block * docs: fix mongodb yaml block * docs: update variable list for restore backups * chore: implementing storage validation based on a path * fix: using proper variables for mongo_restore role * fix: docs links --------- Co-authored-by: jfavellar90 <jhony.avella@edunext.co>
eduNEXT · Mar 22, 2024 · 3bc5fa7 · 3bc5fa7
1 parent 7bd9a7d
commit 3bc5fa7
Show file tree

Hide file tree

Showing 21 changed files with 368 additions and 68 deletions.
diff --git a/README.md b/README.md
@@ -171,3 +171,70 @@ For now this repo supports the playbooks for:
 - Caddy
 
 ---
+
+## Extract and Restore databases:
+
+The procedure involves extracting specific databases, creating backups of each, and subsequently restoring them into a new database. A list of databases to process will be compiled, and for each, a backup will be created followed by a restoration. The outcome will be a new database named after the original with an appended suffix. For example, if the original database is named `edxapp`, the new database will be called `edxapp_clone`, containing the data from the original database.
+
+This process will be carried out using the `admin` user for MySQL and Mongo, which must have the necessary permissions to perform these operations.
+
+### Launching Database Restoration:
+
+1. Install atlas-ansible-utils following [these instructions](https://github.com/eduNEXT/atlas-ansible-utils?tab=readme-ov-file#how-to-use-this-repo).
+2. Configure the variables that the roles mysql_clone and mongo_clone need to work.
+
+  #### How to set variables to clone:
+
+  ##### MySQL
+  The variables that you must configure in [defaults.yml](https://github.com/eduNEXT/atlas-ansible-utils/blob/main/roles/mysql_clone/defaults/main.yml) are:
+
+  - `MYSQL_CLONE_USER`: User with the necessary privileges to perform backups and restore databases.
+  - `MYSQL_CLONE_PASSWORD`: Password of this user.
+  - `MYSQL_CLONE_DB_LIST`: List of databases you wish to clone.
+  - `MYSQL_CLONE_TARGET_PATH`: Path where the database is mounted to check available space.
+
+    The variable configuration should look like this:
+    ```yaml
+      MYSQL_CLONE_USER: admin
+      MYSQL_CLONE_PASSWORD: ABCDefgh12345
+      MYSQL_CLONE_TARGET_PATH: /var/lib/mysql
+      MYSQL_CLONE_DB_LIST:
+      - edxapp
+      - edx_notes_api
+
+      # Other variables...
+    ```
+  ##### MongoDB
+  Similar to the MySQL configuration, the variables that you must configure in [defaults.yml](https://github.com/eduNEXT/atlas-ansible-utils/blob/main/roles/mongo_clone/defaults/main.yml) are:
+
+  - `MONGO_CLONE_USER`: User with the necessary privileges to perform backups and restore databases.
+  - `MONGO_CLONE_PASSWORD`: Password of this user.
+  - `MONGO_CLONE_DB_LIST`: List of databases you wish to clone.
+  - `MONGO_CLONE_TARGET_PATH`: Path where the database is mounted to check available space.
+
+    The variable configuration should look like this:
+      ```yaml
+        MONGO_CLONE_USER: admin
+        MONGO_CLONE_PASSWORD: ABCDefgh12345
+        MONGO_CLONE_TARGET_PATH: /edx/var/mongo/mongodb
+        MONGO_CLONE_DB_LIST:
+        - edxapp
+        - cs_comments_service
+
+        # Other variables...
+      ```
+3. Execute the routine to restore the databases:
+
+  For mongo:
+
+   ```bash
+   ansible-playbook mongo_backup_restore.yml -i /path/inventory/host.ini -v
+   ```
+
+  For mongo:
+
+   ```bash
+   ansible-playbook mysql_backup_restore.yml -i /path/inventory/host.ini -v
+   ```
+
+Upon completion of the execution, you should observe both the existing databases and the new databases suffixed with `_clone` in your database instance, as per the default configuration.
diff --git a/mongo_backup_restore.yml b/mongo_backup_restore.yml
@@ -0,0 +1,10 @@
+# Description: Playbook to clone Mongo databases by extracting backups and restore them with
+# changed database names in the same Mongo instance.
+- name: Launch Mongo backup and restore
+  hosts: mongo_servers
+  become: True
+  gather_facts: True
+  roles:
+    - role: mongo_clone
+      tags:
+        - mongo_clone
diff --git a/mysql_backup_restore.yml b/mysql_backup_restore.yml
@@ -0,0 +1,10 @@
+# Description: Playbook to clone MySQL databases by extracting backups and restore them with
+# changed database names in the same MySQL instance.
+- name: Launch MySQL backup and restore
+  hosts: mysql_servers
+  become: True
+  gather_facts: True
+  roles:
+    - role: mysql_clone
+      tags:
+        - mysql_clone
diff --git a/roles/available_storage/defaults/main.yml b/roles/available_storage/defaults/main.yml
@@ -0,0 +1,9 @@
+# This role helps validating there's enough space in a disk to duplicate its data. This is useful when we
+# want to clone databases stored in a volume.
+# AVAILABLE_STORAGE_PATH_TO_CHECK is checked to make sure we can create a full copy of the data on this path
+# in the disk the path belongs to with no issues
+
+
+AVAILABLE_STORAGE_PATH_TO_CHECK: "/root"
+# Minimum percentage of free disk space expected after restoring a database
+AVAILABLE_STORAGE_THRESHOLD_PERCENTAGE: 20
diff --git a/roles/available_storage/tasks/main.yml b/roles/available_storage/tasks/main.yml
@@ -0,0 +1,24 @@
+- name: Get disk information for the path we are checking
+  shell: df -m "{{ AVAILABLE_STORAGE_PATH_TO_CHECK }}" | awk 'NR==2 {print $2","$3}'
+  register: disk_info
+
+- name: Get the space consumed by the checked path
+  shell: du -sh -m "{{ AVAILABLE_STORAGE_PATH_TO_CHECK }}" | awk '{ print $1 }'
+  register: path_consumed_space
+
+- name: Extract values
+  set_fact:
+    disk_size: disk_info.stdout.split(',')[0] | int
+    disk_used: disk_info.stdout.split(',')[1] | int
+    path_space: path_consumed_space | int
+
+- name: Calculate free storage
+  set_fact:
+    disk_free_after_restore: disk_size - (disk_used + path_space)
+    disk_free_required: disk_size * (AVAILABLE_STORAGE_THRESHOLD_PERCENTAGE / 100)
+    disk_free_percentage: ((disk_size - (disk_used + path_space)) / disk_size ) * 100
+
+- name: Validate free storage
+  fail:
+    msg: "The expected free space on the disk containing the {{ AVAILABLE_STORAGE_PATH_TO_CHECK }} path is less than {{ AVAILABLE_STORAGE_THRESHOLD_PERCENTAGE }}%. Expected free space after restore: {{ disk_free_percentage }}%"
+  when: disk_free_after_restore < disk_free_required
diff --git a/roles/mongo_backup/defaults/main.yml b/roles/mongo_backup/defaults/main.yml
@@ -1,7 +1,8 @@
 ---
-
 MONGO_BACKUP_USER: "admin"
 MONGO_BACKUP_PASSWORD: "mongo_password"
+MONGO_BACKUP_ALL_DATABASES: true
+MONGO_BACKUP_DATABASE: "edxapp"
 MONGO_BACKUP_ROOT: "/var/edunext_tmp/mongo"
 MONGO_BACKUP_DATE: "{{ ansible_date_time.date }}"
 MONGO_BACKUP_LOCATION: "{{ MONGO_BACKUP_ROOT }}/{{ MONGO_BACKUP_DATE }}.d"

diff --git a/roles/mongo_backup/tasks/backup.yml b/roles/mongo_backup/tasks/backup.yml
@@ -0,0 +1,33 @@
+# Description: Playbook to launch Mongo backups
+# It is compatible with the S3 API and Azure Blob Storage.
+---
+- name: Create backup directory
+  file:
+    path: "{{ MONGO_BACKUP_LOCATION }}"
+    state: directory
+
+# To clone the databases, we use the commands recommended by MongoDB:
+# https://www.mongodb.com/docs/database-tools/mongodump/#copy-and-clone-databases
+- name: Create mongo database backup
+  shell: >
+    mongodump
+    --authenticationDatabase admin
+    -u {{ MONGO_BACKUP_USER }} -p '{{ MONGO_BACKUP_PASSWORD }}'
+    --gzip
+    --archive={{ MONGO_BACKUP_LOCATION }}/{{ MONGO_BACKUP_DATE }}_mongo.gz
+    {% if not MONGO_BACKUP_ALL_DATABASES %}
+    --db={{ MONGO_BACKUP_DATABASE }}
+    {% endif %}
+
+- name: Give the server time to recover
+  pause:
+    minutes: 1
+    prompt: Pausing to give the server time to recover
+
+- name: Upload the backup to a remote storage
+  include_role:
+    name: storage_backups
+  vars:
+    STORAGE_BACKUPS_OPTIONS: "{{ MONGO_BACKUP_STORAGE_OPTIONS }}"
+    STORAGE_BACKUPS_FILES_TO_UPLOAD:
+      - "{{ MONGO_BACKUP_LOCATION }}/{{ MONGO_BACKUP_DATE }}_mongo.gz"
diff --git a/roles/mongo_backup/tasks/main.yml b/roles/mongo_backup/tasks/main.yml
@@ -1,39 +1,12 @@
-# Description: Playbook to launch Mongo backups
-# It is compatible with the S3 API and Azure Blob Storage.
----
-
 - name: Clean the backup root path before starting the routine
   file:
     state: absent
     path: "{{ mongo_artifact_path }}/"
   when: MONGO_BACKUP_PRE_CLEAN_ROOT and mongo_artifact_path is defined and mongo_artifact_path != "" and
         MONGO_BACKUP_STORAGE_OPTIONS.EXTERNAL_STORAGE_TYPE != ""
 
-- name: Create backup directory
-  file:
-    path: "{{ MONGO_BACKUP_LOCATION }}"
-    state: directory
-
-- name: Create mongo database backup
-  shell: >
-    mongodump
-    --authenticationDatabase admin
-    -u {{ MONGO_BACKUP_USER }} -p '{{ MONGO_BACKUP_PASSWORD }}'
-    --gzip
-    --archive={{ MONGO_BACKUP_LOCATION }}/{{ MONGO_BACKUP_DATE }}.gz
-
-- name: Give the server time to recover
-  pause:
-    minutes: 1
-    prompt: Pausing to give the server time to recover
-
-- name: Upload the backup to a remote storage
-  include_role:
-    name: storage_backups
-  vars:
-    STORAGE_BACKUPS_OPTIONS: "{{ MONGO_BACKUP_STORAGE_OPTIONS }}"
-    STORAGE_BACKUPS_FILES_TO_UPLOAD:
-      - "{{ MONGO_BACKUP_LOCATION }}/{{ MONGO_BACKUP_DATE }}.gz"
+- name: Launch Mongo Backups
+  include_tasks: backup.yml
 
 - name: Clean artifact path
   file:

diff --git a/roles/mongo_clone/defaults/main.yml b/roles/mongo_clone/defaults/main.yml
@@ -0,0 +1,8 @@
+MONGO_CLONE_USER: admin
+MONGO_CLONE_PASSWORD: admin_pass
+MONGO_CLONE_TARGET_PATH: /edx/var/mongo/mongodb
+MONGO_CLONE_DB_LIST:
+- edxapp
+- cs_comments_service
+MONGO_CLONE_ROOT: /var/edunext_tmp/mongo
+MONGO_CLONE_RESTORE_SUFFIX: "_clone"
diff --git a/roles/mongo_clone/tasks/clone.yml b/roles/mongo_clone/tasks/clone.yml
@@ -0,0 +1,28 @@
+---
+- name: Extract Mongo backup from database {{ item }}
+  include_role:
+    name: mongo_backup
+  vars:
+    MONGO_BACKUP_ALL_DATABASES: false
+    MONGO_BACKUP_PRE_CLEAN_ROOT: true
+    MONGO_BACKUP_USER: "{{ MONGO_CLONE_USER }}"
+    MONGO_BACKUP_PASSWORD: "{{ MONGO_CLONE_PASSWORD }}"
+    MONGO_BACKUP_DATABASE: "{{ item }}"
+    MONGO_BACKUP_ROOT: "{{ MONGO_CLONE_ROOT }}"
+    MONGO_BACKUP_STORAGE_OPTIONS:
+      EXTERNAL_STORAGE_TYPE: ""
+      EXTERNAL_STORAGE_OPTIONS: {}
+
+- name: Generate new database name with suffix
+  set_fact:
+    new_clone_db_name: "{{ item + MONGO_CLONE_RESTORE_SUFFIX }}"
+
+- name: Restore Mongo backup on database {{ new_clone_db_name }}
+  include_role:
+    name: mongo_restore
+  vars:
+    MONGO_RESTORE_BACKUP_ROOT: "{{ MONGO_CLONE_ROOT }}"
+    MONGO_RESTORE_USER: "{{ MONGO_CLONE_USER }}"
+    MONGO_RESTORE_PASSWORD: "{{ MONGO_CLONE_PASSWORD }}"
+    MONGO_RESTORE_ORIGINAL_DATABASE_NAME: "{{ item }}"
+    MONGO_RESTORE_TARGET_DATABASE_NAME: "{{ new_clone_db_name }}"
diff --git a/roles/mongo_clone/tasks/main.yml b/roles/mongo_clone/tasks/main.yml
@@ -0,0 +1,9 @@
+- name: Launch Mongo check available storage
+  include_role:
+    name: available_storage
+  vars:
+    AVAILABLE_STORAGE_PATH_TO_CHECK: "{{ MONGO_CLONE_TARGET_PATH }}"
+
+- name: Backup and restore every single database in the clone list
+  include_tasks: clone.yml
+  loop: "{{ MONGO_CLONE_DB_LIST }}"
diff --git a/roles/mongo_restore/defaults/main.yml b/roles/mongo_restore/defaults/main.yml
@@ -0,0 +1,18 @@
+# Source database configuration
+MONGO_RESTORE_USER: "{{ EDXAPP_MONGO_USER }}"
+MONGO_RESTORE_PASSWORD: "{{ EDXAPP_MONGO_PASSWORD }}"
+
+# Sets the number of collections being restoren in parallel, shich can be adjusted depending on server performance
+MONGO_RESTORE_NUM_PARALLEL_COLLECTIONS: 4
+
+# The variables MONGO_RESTORE_ORIGINAL_DATABASE_NAME and MONGO_RESTORE_TARGET_DATABASE_NAME are used
+# to restore a database backup with a name to new database with a different name. Both variables must be
+# set to enable this clone behavior. This is based on the Mongo documentation that indicates how to clone
+# a database: https://www.mongodb.com/docs/database-tools/mongorestore/#copy-clone-a-database
+MONGO_RESTORE_ORIGINAL_DATABASE_NAME: ""
+MONGO_RESTORE_TARGET_DATABASE_NAME: ""
+
+# Backup folders
+MONGO_RESTORE_BACKUP_ROOT: "/var/edunext_tmp/mongo"
+MONGO_RESTORE_BACKUP_DATE: "{{ ansible_date_time.date }}"
+MONGO_RESTORE_BACKUP_LOCATION: "{{ MONGO_RESTORE_BACKUP_ROOT }}/{{ MONGO_RESTORE_BACKUP_DATE }}.d"
diff --git a/roles/mongo_restore/tasks/main.yml b/roles/mongo_restore/tasks/main.yml
@@ -0,0 +1,19 @@
+# To clone the databases, we use the commands recommended by MongoDB:
+# https://www.mongodb.com/docs/database-tools/mongorestore/#copy-clone-a-database
+- name: Restore database
+  shell: >
+    mongorestore
+    --authenticationDatabase admin
+    -u {{ MONGO_RESTORE_USER  }} -p '{{ MONGO_RESTORE_PASSWORD }}'
+    --gzip
+    --archive={{ MONGO_RESTORE_BACKUP_LOCATION }}/{{ MONGO_RESTORE_BACKUP_DATE }}_mongo.gz
+    --numParallelCollections={{ MONGO_RESTORE_NUM_PARALLEL_COLLECTIONS }}
+    {% if MONGO_RESTORE_ORIGINAL_DATABASE_NAME and MONGO_RESTORE_TARGET_DATABASE_NAME %}
+    --nsFrom={{ MONGO_RESTORE_ORIGINAL_DATABASE_NAME }}.*
+    --nsTo={{ MONGO_RESTORE_TARGET_DATABASE_NAME }}.*
+    {% endif %}
+
+- name: Clean artifact path
+  file:
+    state: absent
+    path: "{{ MONGO_RESTORE_BACKUP_ROOT }}"
diff --git a/roles/mysql_backup/defaults/main.yml b/roles/mysql_backup/defaults/main.yml
@@ -3,7 +3,8 @@
 MYSQL_BACKUP_USER: "root"
 MYSQL_BACKUP_PASSWORD: "mysql_password"
 MYSQL_BACKUP_ALL_DATABASES: true
-MYSQL_BACKUP_DATABASES: "edxapp"
+MYSQL_BACKUP_DATABASES:
+- edxapp
 MYSQL_BACKUP_ROOT: "/var/edunext_tmp/mysql"
 MYSQL_BACKUP_DATE: "{{ ansible_date_time.date }}"
 MYSQL_BACKUP_LOCATION: "{{ MYSQL_BACKUP_ROOT }}/{{ MYSQL_BACKUP_DATE }}.d"

diff --git a/roles/mysql_backup/tasks/backup.yml b/roles/mysql_backup/tasks/backup.yml
@@ -0,0 +1,45 @@
+# Description: Playbook to launch MySQL backups
+# It is compatible with the S3 API and Azure Blob Storage.
+---
+- name: Create backup directory
+  file:
+    path: "{{ MYSQL_BACKUP_LOCATION }}"
+    state: directory
+
+- name: Convert MySQL backups databases to list to provide backwards compatibility
+  set_fact:
+    MYSQL_BACKUP_DATABASES: "{{ MYSQL_BACKUP_DATABASES | split }}"
+  when: MYSQL_BACKUP_DATABASES is string
+
+- name: Create MySQL database backup
+  shell: >
+    mysqldump -u {{ MYSQL_BACKUP_USER }} -p{{ MYSQL_BACKUP_PASSWORD }}
+    --opt --single-transaction --add-drop-database --no-tablespaces
+    {% if MYSQL_BACKUP_ALL_DATABASES %}
+    --all-databases
+    {% elif MYSQL_BACKUP_DATABASES | length == 1 %}
+    {{ MYSQL_BACKUP_DATABASES | join(" ") }}
+    {% else %}
+    --databases {{ MYSQL_BACKUP_DATABASES | join(" ") }}
+    {% endif %}
+    > {{ MYSQL_BACKUP_LOCATION }}/{{ MYSQL_BACKUP_DATE }}_mysql.sql
+  args:
+    chdir: "{{ MYSQL_BACKUP_LOCATION }}"
+
+- name: Compress mysql backup file
+  shell: "gzip {{ MYSQL_BACKUP_LOCATION }}/{{ MYSQL_BACKUP_DATE }}_mysql.sql"
+  args:
+    chdir: "{{ MYSQL_BACKUP_LOCATION}}"
+
+- name: Give the server time to recover
+  pause:
+    minutes: 1
+    prompt: Pausing to give the server time to recover
+
+- name: Upload the backup to a remote storage
+  include_role:
+    name: storage_backups
+  vars:
+    STORAGE_BACKUPS_OPTIONS: "{{ MYSQL_BACKUP_STORAGE_OPTIONS }}"
+    STORAGE_BACKUPS_FILES_TO_UPLOAD:
+      - "{{ MYSQL_BACKUP_LOCATION }}/{{ MYSQL_BACKUP_DATE }}_mysql.sql.gz"