Skip to content

Commit

Permalink
Restructure role, add Systemd timer option. By @conloos (#112)
Browse files Browse the repository at this point in the history
* add full path

* Update Readme.me: reorder optional Arguments, update cron -> systemd timer

* remove ssh_key_file; change cron to timer

* Removed cronie from package installation because systemd timer is used

* docker.sh - Stops all or selected containers to save the persistent data intact. The containers are started in reverse order

* Created arguments_specs.yml

* Role restructured:
  - if needed creation of a service user incl. creation of the ssh-key,
  - add the ssh key to authorized_keys,
  - auto init of the repos,
  - creation and start of systemd timer and services and
  - installation of the Docker helperscript.

* restructure role add import logic

* cleanup: user backup_user

* - "borg_source_directories" is not longer a required Argument
- add "borg_keys_directory" to load key from Service user during starting borgmatic by sudo

* Add borgmatic_initialization_repo (bool) as option to disable init of repo

* cleanup

* fix ansible-lint errors and warnings

* fix letter turner

* add option: borgmatic_timer

* add:
  - borgmatic_timer_systemd: true
readd:
  - borgmatic_cron_name: "borgmatic"

* - renamed borgmatic_cron_name to borgmatic_timer_cron_name to be more convergent.
- Change recommendations implemented by m3nu so that creation of a timer (systemd or cron) is optional and can be selected via borgmatic_timer.

* Add description to borgmatic_timer_cron_name and borgmatic_timer

* Add variable borg_cron_package to install the cron-packages in case of using timer: cron

* reworked timer install logic

* reworked timer install logic

* Add comments for running backup with service account

* add new parameters for tests

* Switch created to perform the backup as root or service account. If a service account is to be used, it will be created.

* Refactored: Check for ssh-key if not present, genereate them.

* Refactored

* Refactored

* renamed tasks/03_configure.yml to tasks/04_create_links_to_borg_and_borgmatic.yml

* Refactored

* Refactored

* add example for service account

* Update Python version for testing

* No auto init

* Add description to install_backup

* Add description to install_backup

* set coverage back to: m3nu.ansible_role_borgbackup

* The initialization of the repository must be activated and does not take place automatically.

* The initialization of the repository must be activated and does not take place automatically.

* Removed install_backup as var (bool) to prevent that this role run

* Rename backup_ssh_command to borg_ssh_command, tis was a double definition

* Renamed backup_repository to borg_repository and add better explanations

* remove copy ssh-keys and cert parts

* Add comments to borg_ssh_key_file and borg_ssh_key_type

* Set allways the borg_ssh_key_file and borg_ssh_command to load the right ssh-key. Add borg_ssh_key_type to select the key type by user

* Add borg_ssh_key_type

* renamed id_rsa to backup

* generate ssh-keys (backup and backup.pub) and add better explanation

* Print out key if borgmatic_initialization_repo is false

* Remove 'su - {{ borgbackup_user }} -c' to execute the borgmatic by the right user

* Add Check frequency, therefore, we no longer need to distinguish between normal and large repos

* Add link to Article

* renamed backup_ssh_command and backup_ssh_key_file to borg_ssh_command and borg_ssh_key_file

* Removed: borgmatic_initialization_repo

* Removed: borgmatic_initialization_repo

* Removed: borgmatic_initialization_repo

* revert changes

* Add Full Automation

* polishing

* rename backup.timer and bakup.service to borgmatic.timer and borgmatic.service

* remove debug

* Try to find services in ansible_facts

* Forgot to install Cron

* change borg_ssh_key_type to ed25519

* remove conditional checks

* - add hint to using a service user
- renamed: borg_ssh_key_file to borg_ssh_key_file_path
- removed advanced example

* add borg_ssh_key_name, renamed borg_ssh_key_file to borg_ssh_key_file_path

* removed static pointing to ~/.ssh/backup SSH private key

* Add README-Advanced-Examples.md for storing more examples

* Fix test idempotence

* Dont symlink when using distro packages

* Remove old test targets, consistent wording, remove tag

* Remove helper scripts, fix absolute path

* Fix cron job, add assert to prevent duplicate timers

* nit-pick

* Create bin links as root, no borg_ssh_command by default.

* Add breaking changes note to README

---------

Co-authored-by: Manu <manu@snapdragon.cc>
  • Loading branch information
conloos and m3nu authored Mar 28, 2023
1 parent 981d4f9 commit dd960dc
Show file tree
Hide file tree
Showing 31 changed files with 820 additions and 229 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v1
uses: actions/setup-python@v4
with:
python-version: 3.8
python-version: "3.10"
- name: Install Molecule
run: |
pip install -U pip setuptools wheel
Expand Down
129 changes: 129 additions & 0 deletions EXAMPLES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Additional Examples

## Custom SSH key for backups only

```
- hosts: webservers
roles:
- role: m3nu.ansible_role_borgbackup
borg_encryption_passphrase: CHANGEME
borg_repository: ssh://m5vz9gp4@m5vz9gp4.repo.borgbase.com/./repo
borgmatic_timer: systemd
borg_ssh_key_name: id_backup
borg_ssh_command: "ssh -i {{ borg_ssh_key_file_path }} -o StrictHostKeyChecking=accept-new"
borg_user: backupuser
borg_group: backupuser
```

## Use service user and copy SSH key to target server

Installs and configures the Borgmatic client and also initializes the repo on the
remote backup server. (not tested)

```
- name: Configure backup
hosts: test.lab
pre_tasks:
- name: Get home of {{ borg_user }}
ansible.builtin.user:
name: "{{ borg_user }}"
state: present
register: user_info
changed_when: false
check_mode: true # Important, otherwise user will be created
- name: Save the user_info, we need them for the home_dir
ansible.builtin.set_fact:
backup_user_info: "{{ user_info }}"
vars_files: []
vars:
borg_encryption_passphrase: "CHANGEME"
borg_repository: "USER@TARGET_SERVER:/PATH/TO/BACKUP"
borg_user: "srv_backup"
borg_group: "srv_backup"
borg_ssh_key_name: id_backup
borg_ssh_command: "ssh -i {{ borg_ssh_key_file_path }} -o StrictHostKeyChecking=accept-new"
borgmatic_timer: systemd
borg_source_directories:
- /srv/www
- /var/lib/automysqlbackup
borg_exclude_patterns:
- /srv/www/old-sites
borg_retention_policy:
keep_hourly: 3
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
borgmatic_hooks:
before_backup:
- echo "`date` - Starting backup."
tasks:
- name: Configure Borg Backup and Backupmatic
tags:
- always
- install_backup
ansible.builtin.include_role:
name: ansible_role_borgbackup
apply:
tags:
- always
- name: Copy SSH-Key to Target {{ borg_repository }} and Init Repo
tags:
- never
- backup_init_repo
block:
- name: Read ssh key
ansible.builtin.slurp:
src: "{{ borg_ssh_key_file_path }}.pub"
register: backup_local_ssh_key
- name: Set authorized key taken from file
ansible.posix.authorized_key:
# example:
# borg_repository: m5vz9gp4@m5vz9gp4.repo.borgbase.com:repo
# have three parts: "username"@"FQDN":"path/to/store/backup", specific:
# a) user: m5vz9gp4
# b) fqdn: m5vz9gp4.repo.borgbase.co
# c) dir: repo
user: "{{ borg_repository | regex_search('(.*)@', '\\1') | first }}" # part a)
state: present
key: "{{ backup_local_ssh_key['content'] | b64decode }}"
delegate_to: "{{ borg_repository | regex_search('@(.*):', '\\1') | first }}" # part b)
- name: Init repository
ansible.builtin.command:
cmd: "su - {{ borg_user }} -c '/usr/local/bin/borgmatic rcreate --encryption keyfile --append-only'"
- name: Activate systemd service and timer
when:
- borgmatic_timer is defined and borgmatic_timer == "systemd"
tags:
- never
- backup_init_repo
block:
- name: Populate service facts
ansible.builtin.service_facts:
- name: Start borgmatic services
ansible.builtin.systemd:
name: "{{ item }}"
state: started
enabled: true
masked: false
daemon_reload: true
when: "item in services"
with_items:
- borgmatic.service
# bug: Need own section without masked else the timer are skipped
- name: Start borgmatic timers
ansible.builtin.systemd:
name: "{{ item }}"
state: started
enabled: true
daemon_reload: true
with_items:
- "borgmatic.timer"
```
95 changes: 61 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,39 +6,58 @@ Set up encrypted, compressed and deduplicated backups using [BorgBackup](https:/

Works great with [BorgBase.com](https://www.borgbase.com) - Simple and Secure Hosting for your Borg Repositories. To manage BorgBase repos via Ansible, also see Andy Hawkins' [BorgBase Collection](https://galaxy.ansible.com/adhawkins/borgbase).

Main features:
- Set up Borg and Borgmatic
- Add cron job at random time
- Provision new remote [BorgBase.com](https://www.borgbase.com) repo for storing backups (optional)
**Main features**
- Install Borg and Borgmatic from PyPi or distro packages
- Set up Borgmatic config
- Schedule regular backups using Cron or Systemd timer

## Breaking changes
- Older versions of this role set up a separate Cron job for creating and checking
backups. With recent Borgmatic version, this feature is now managed in Borgmatic.
As a result the extra Cron job will be removed by this role.
- Older versions of this role only supported Cron for scheduling. If you use
Systemd timers, be sure to remove the Cron job in `/etc/cron.d/borgmatic` first.
The role will also alert you when trying to use both timers.

## Example Playbook
## Example playbook with root as backup user and Cron timer

```
- hosts: webservers
- hosts: all
roles:
- role: m3nu.ansible_role_borgbackup
borg_encryption_passphrase: CHANGEME
borg_repository: m5vz9gp4@m5vz9gp4.repo.borgbase.com:repo
borg_repository: ssh://xxxxxx@xxxxxx.repo.borgbase.com/./repo
borg_source_directories:
- /srv/www
- /var/lib/automysqlbackup
borg_exclude_patterns:
- /srv/www/old-sites
borg_retention_policy:
keep_hourly: 3
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
- /var/www
borgmatic_hooks:
before_backup:
- echo "`date` - Starting backup."
postgresql_databases:
- name: users
hostname: database1.example.org
port: 5433
```

## Example playbook with service user and Systemd timer

```
- hosts: all
roles:
- role: m3nu.ansible_role_borgbackup
borg_encryption_passphrase: CHANGEME
borg_repository: ssh://xxxxxx@xxxxxx.repo.borgbase.com/./repo
borgmatic_timer: systemd
borg_user: "backupuser"
borg_group: "backupuser"
borg_source_directories:
- /var/www
borg_retention_policy:
keep_hourly: 3
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
```



## Installation
Expand All @@ -56,39 +75,47 @@ $ git clone https://github.com/borgbase/ansible-role-borgbackup.git roles/ansibl

## Role Variables

### Required Arguments
- `borg_repository`: Full path to repository. Your own server or [BorgBase.com](https://www.borgbase.com) repo. Not required when using auto creation of repositories. Can be a list if you want to backup to multiple repositories.
- `borg_source_directories`: List of local folders to back up.
### Required Variables
- `borg_repository`: Full path to repository. Your own server or [BorgBase.com](https://www.borgbase.com) repo.
Can be a list if you want to backup to multiple repositories.

### Optional Arguments
### Optional Variables
- `borg_dep_packages`: Dependency Packages to install `borg(backup)` and `borgmatic`.
- `borg_distro_packages`: contains the names of distributions packages for `borg(backup)` and `borgmatic`, only used if `borg_install_method` is set to `package`.
- `borg_encryption_passcommand`: The standard output of this command is used to unlock the encryption key.
- `borg_encryption_passphrase`: Password to use for repokey or keyfile. Empty if repo is unencrypted.
- `borg_exclude_from`: Read exclude patterns from one or more separate named files, one pattern per line.
- `borg_exclude_patterns`: Paths or patterns to exclude from backup. See [official documentation](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns) for more.
- `borg_install_method`: By default `pip` is used to install borgmatic. To install via your distributions package manager set this to `package` and (if needed) overwrite the `borg_distro_packages` variable to contain your distributions package names required to install borgmatic. Note that many distributions ship outdated versions of borgbackup and borgmatic; use at your own risk.
- `borg_lock_wait_time`: Config maximum seconds to wait for acquiring a repository/cache lock. Defaults to 5 seconds.
- `borg_one_file_system`: Don't cross file-system boundaries. Defaults to `true`
- `borg_pip_packages`: Dependancy Packages (pip) to install `borg(backup)` and `borgmatic`.
- `borg_remote_path`: Path to the borg executable on the remote. It will default to `borg`.
- `borg_remote_rate_limit`: Remote network upload rate limit in kiBytes/second.
- `borg_retention_policy`: Retention policy for how many backups to keep in each category (daily, weekly, monthly, etc).
- `borg_ssh_command`: Command to use instead of just "ssh". This can be used to specify ssh options.
- `borg_source_directories`: List of local folders to back up. Default is `/etc/hostname` to prevent an empty backup.
- `borg_ssh_key_name`: Name of the SSH public and pivate key. Default `id_ed25519`
- `borg_ssh_key_file_path`: SSH-key to be used. Default `~/.ssh/{{ borg_ssh_key_name }}`
- `borg_ssh_key_type`: The algorithm used to generate the SSH private key. Choose: `rsa`, `dsa`, `rsa1`, `ecdsa`, `ed25519`. Default: `ed25519`
- `borg_ssh_command`: Command to use instead of just "ssh". This can be used to specify SSH options.
- `borg_version`: Force a specific borg version to be installed
- `borg_venv_path`: Path to store the venv for `borg(backup)` and `borgmatic`

- `borgmatic_check_last`: Number of archives to check. Defaults to `3`
- `borgmatic_checks`: List of consistency checks. Defaults to `['repository']`
- `borgmatic_config_name`: Name to use for the borgmatic config file. Defaults to `config.yaml`
- `borgmatic_cron_checks_day`: Day when cron job for infrequent checks will run. Defaults to `{{ 28 | random }}`
- `borgmatic_cron_checks_hour`: Hour when cron job for infrequent checks will run. Defaults to `{{ range(7, 24) | random }}`
- `borgmatic_cron_checks_minute`: Minute when cron job for infrequent checks will run. Defaults to `{{ 59 | random }}`
- `borgmatic_cron_hour`: Hour when regular create and prune cron job will run. Defaults to `{{ 6 | random }}`
- `borgmatic_cron_minute`: Minute when regular create and prune cron job will run. Defaults to `{{ 59 | random }}`
- `borgmatic_checks`: List of consistency checks. Defaults to monthly checks. See [docs](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#check-frequency) for all options.
- `borgmatic_config_name`: Name to use for the Borgmatic config file. Defaults to `config.yaml`
- `borgmatic_timer_hour`: Hour when regular create and prune cron/systemd-timer job will run. Defaults to `{{ 6 | random }}`
- `borgmatic_timer_minute`: Minute when regular create and prune cron/systemd-timer job will run. Defaults to `{{ 59 | random }}`
- `borgmatic_hooks`: Hooks to monitor your backups e.g. with [Healthchecks](https://healthchecks.io/). See [official documentation](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/) for more.
- `borgmatic_large_repo`: Less frequent, monthly repo checking. Defaults to `true`
- `borgmatic_timer`: If the variable is set, a timer is installed. A choice must be made between `cron` and `systemd`.
- `borgmatic_relocated_repo_access_is_ok`: Bypass Borg error about a repository that has been moved. Defaults to `false`
- `borgmatic_store_atime`: Store atime into archive. Defaults to `true`
- `borgmatic_store_ctime`: Store ctime into archive. Defaults to `true`
- `ssh_key_file`: Path to a private ssh key file (default is `.ssh/id_ed25519`). It generates a ed25519 key if the file doesn't exist yet.
- `borg_version`: Force a specific borg version to be installed
- `borgmatic_version`: Force a specific borgmatic version to be installed
- `borg_install_method`: By default `pip` is used to install borgmatic. To install via your distributions package manager set this to `package` and (if needed) overwrite the `borg_distro_packages` variable to contain your distributions package names required to install borgmatic. Note that many distributions ship outdated versions of borgbackup and borgmatic; use at your own risk.
- `borg_distro_packages`: contains the names of distributions packages for `borg(backup)` and `borgmatic`, only used if `borg_install_method` is set to `package`.

- `borg_user`: Name of the User to create Backups (service account)
- `borg_group`: Name of the Group to create Backups (service account)


## Contributing

Expand All @@ -105,4 +132,4 @@ MIT/BSD

## Author

© 2018-2020 Manuel Riel and contributors.
© 2018-2023 Manuel Riel and contributors.
54 changes: 31 additions & 23 deletions defaults/main.yml
Original file line number Diff line number Diff line change
@@ -1,25 +1,13 @@
---
borg_encryption_passphrase: ''
borg_exclude_patterns: []
borgmatic_config_name: config.yaml
borgmatic_large_repo: true
borgmatic_hooks:
on_error:
- echo "`date` - Error while creating a backup."
before_backup:
- echo "`date` - Starting backup."
after_backup:
- echo "`date` - Finished backup."
borgmatic_checks:
- repository
borgmatic_check_last: 3
borgmatic_store_atime: true
borgmatic_store_ctime: true
borgmatic_relocated_repo_access_is_ok: false
borg_one_file_system: true
borg_exclude_from: []
borg_encryption_passcommand: false
borg_lock_wait_time: 5
borg_ssh_key_type: "ed25519"
borg_ssh_key_name: "id_{{ borg_ssh_key_type }}"
borg_ssh_key_file_path: "{{ backup_user_info.home }}/.ssh/{{ borg_ssh_key_name }}"
borg_ssh_command: false
borg_remote_path: false
borg_remote_rate_limit: 0
Expand All @@ -28,13 +16,33 @@ borg_retention_policy:
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
ssh_key_file: .ssh/id_ed25519
borgmatic_cron_name: borgmatic
borgmatic_cron_hour: "{{ 6 | random(seed=inventory_hostname) }}"
borgmatic_cron_minute: "{{ 59 | random(seed=inventory_hostname) }}"
borgmatic_cron_checks_day: "{{ range(1, 28) | random(seed=inventory_hostname) }}"
borgmatic_cron_checks_hour: "{{ range(9, 24) | random(seed=inventory_hostname) }}"
borgmatic_cron_checks_minute: "{{ 59 | random(seed=inventory_hostname) }}"
borg_version: false
borgmatic_timer_cron_name: "borgmatic"
borgmatic_timer: cron
borgmatic_timer_hour: "{{ range(0, 5) | random(seed=inventory_hostname) }}"
borgmatic_timer_minute: "{{ range(0, 59) | random(seed=inventory_hostname) }}"
borg_install_method: "pip"

borgmatic_config_name: config.yaml
borgmatic_hooks:
on_error:
- echo "`date` - Error while creating a backup."
before_backup:
- echo "`date` - Starting backup."
after_backup:
- echo "`date` - Finished backup."
borgmatic_checks:
- name: repository
frequency: "4 weeks"
- name: archives
frequency: "6 weeks"
borgmatic_check_last: 3
borgmatic_store_atime: true
borgmatic_store_ctime: true
borgmatic_relocated_repo_access_is_ok: false
borgmatic_version: false
borg_install_method: pip

borg_venv_path: "/opt/borgmatic"
borg_user: "root"
borg_group: "root"
...
Loading

0 comments on commit dd960dc

Please sign in to comment.