Qubes OS + Borg + Rsync.net backup strategy scripts
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
host-keys
template
.gitignore
LICENSE.md
README.md
assemble-authorized-keys.sh
new-key.sh
upload-authorized-keys.sh

README.md

Qubes OS, Borg & Rsync.net

This repo contains a few utility scripts to manage Qubes OS backups using Borg Backup into Rsync.net.

Rationale

Backup of data is essential in any workflow. Two top concerns of any backup strategy are the security and the cost of maintaining backups.

Borg Backup offers a reasonably secure, open source backup software solution.

Rsync.net offers special pricing discounts for borg users.

Qubes OS offsers separation of concerns and security domains on your desktop computer.

Let's combine them FTW!

DISCLAIMER

This repo contains template files and scripts to setup your own backup workflow. I am using it for my own workflow, which may or may not be appropriate for your own needs.

Please review all files and scripts and adapt this flow to your preferred backup strategy.

I make no guarantees of backup performance, reliability and integrity if you follow this template.

USE AT YOUR OWN RISK

Usage

This assumes the following:

  • You have setup an account at rsync.net (or any backup provider that allows ssh+borg access);
  • You have installed borg and borgmatic;
  • If using Qubes, you have enabled the crond service on the AppVMs you'll be making backup from.

1. Setup

  • On your personal ~/.ssh/config file, create an entry named "rsyncnet" (see item 3 below) to serve as alias to connect to rsync.net for administration purposes;
  • Create the root repository dir in rsync.net (ssh rsyncnet mkdir -p repos);
  • Copy (or link) the public part of the ssh key you'll use to connect to rsync.net to the file admin-key.openssh.pub;
  • Create the file template/bakserver to serve as template for the AppVMs to connect to the backup server (see item 3 below - use bakserver.template as an example).

2. VM Backup

For each VM you'll want to backup data from, create a new key:

# Replace REPO-NAME. The full path on rsyncnet will be repos/REPO-NAME
$ ./new-key.sh REPO-NAME

# Example
$ ./new-key.sh vm-personal

During key creation you'll need to enter the passphrase to the repo. I suggest generating and storing the passphrase on KeePassX on your VaultVM with high entropy parameters.

The script will also display your passphrase, so only do this when you can be certain nobody is watching your screen.

After generating the key for the new repo, run assemble-authorized-keys.sh to generate an authorized_keys file to be installed on rsync.net. On the first few runs, manually check if the file is correct (specially the admin key section - if that gets messed up you may get locked out of the remote server).

Upload the file to rsync.net with upload-authorized-keys.sh.

The host-keys/REPO-NAME dir should be copied and installed on the target AppVM (see item 4 below). Also, modify the file borgmatic.cfg to your needs (at a minimum you'll need to modify the source_directories section).

3. SSH Config

I usually config ssh connection aliases on ~/.ssh/config.d and then assemble the file ~/.ssh/config by concatenating everything in this dir.

The alias to connect to rsync.net for administration purposes (i.e. anything except backing up from a VM) should look like:

# File ~/.ssh/config.d/rsyncnet
Host rsyncnet 
  HostName <SERVER SENT BY>.rsync.net
  User <USERNAME SENT BY RSYNC.NET>
  IdentityFile ~/.ssh/rsyncnet-admin-key.openssh

The template file template/bakserver should have the HostName and User fields filled with the same information (you can use the bakserver.sample file as an example).

To generate the actual ssh config file run:

$ cat ~/.ssh/config.d/* > ~/.ssh/config

If you don't use this method for configuring aliases, you'll need to manually edit your ~/.ssh/config file on your admin VM and on all the AppVMs used to backup data with the appropriate aliases.

Another alternative is to start using this method by copying your existing ~/.ssh/config into the ~/.ssh/config.d dir and taking from there.

4. AppVM Configuration

The dir generated by the new-key.sh script should have everything needed to configure an AppVM to backup to rsync.net. The files are:

  • bak.crontab: Crontab entry for automated backups;
  • bakserver: The ssh config the VM will use to connect to the backup server (see item 3 above);
  • borgbak: A simple script copied to ~/bin to allow faster usage of borg on rsync.net (seem item 5 below);
  • borgmatic.cfg: The borgmatic config file;
  • borgpass: The password for the backup repo in plain text (see security considerations below);
  • borgrepo: The name of the remote backup repo;
  • install.sh: Install script (read it to see what it does and if it's appropriate to your use case);
  • REPO-NAME.openssh: The private key to backup;
  • REPO-NAME.openssh.pub: The public key to install on the backup server.

This dir is meant to be installed on the target VM as ~/.ssh/backups. Changing this location means changing the location of the key on the bakserver alias and the borgbak utility.

5. The borgbak script

The command line to execute any borg action on rsync.net looks like the following:

$ borg $ACTION --remote-path=borg1 bakserver:repos/$REPO(:$ARCHIVE) $REST
Password: (type password)

It can get tiring typing that all time, so the borgbak script simplifies it by extracting the repo from the file ~/.ssh/backups/borgrepo, the passphrase from the file ~/.ssh/backups/borgpass and the action, archive and rest of the command from the next arguments.

The most commonly used borg commands get simplified to:

$ borgbak list
$ borgbak list personal-2017-08-03T12:56:09.339289
$ borgbak info personal-2017-08-03T12:56:09.339289
$ borgbak extract personal-2017-08-03T12:56:09.339289 home/user/test-file

Complex borg actions may still be issued by using the original borg command and passing the full arguments.

Security Remarks

A few security considerations for this script. They do not represent all potential issues on a given borg installation, only the most relevant ones for this workflow.

Some of the decisions made by this flow can be modified to suit your desired security profile.

Borg Operation Mode

This script currently uses the repokey encryption mode of borg, which means the encryption key file (the actual key bits) is stored on the server. The key file is protected by the key passphrase, which never leaves the client.

If the backup server is compromised, the key file can be extracted. However, that alone should not give access to the actual key bits, as long as the passphrase is strong enough (or if there is a bug or vulnerability on borg's encryption algorithm).

This workflow could be modified to use keyfile mode by generating the key locally (in the new-key.sh script). Using this mode, you must also backup the key file on a different medium or risk losing access to your backups (see remarks about backup accessibility).

SSH Command Restriction

This script assembles an authorized_keys file to be installed on the backup server that restricts the actions of the individual client backup keys to the borg command on a given repo dir.

This is the recommended procedure when multiple sources will write to the same backup server, as it allows each source to have an independent ssh key and borg passphrase, and compromise of one source does not automatically compromise all.

The exception to this is the maintenance key used to connect to the server for full access. Obviously, this key is specially important, should be stored encrypted and only used from a trusted machine.

The current ssh line assembled for each client is the following (minus the newlines):

command="
  cd repos/$HOST;
  borg1 serve --append-only --restrict-to-path repos/$HOST\",
no-port-forwarding,
no-X11-forwarding,
no-pty,
no-agent-forwarding,
no-user-rc
ssh-rsa AAAA...

Append-only Mode

Another security feature of the assembled authorized_keys is to restrict every access of the backup clients to append-only mode.

A compromise of the client VM, the ssh key and/or the borg passphrase does not automatically imply loss of data, although it may imply disclosure of the encrypted data.

All remarks concerning Borg's append-only mode apply.

Plaintext Passphrase

The passphrase used to access a given borg repo is stored as plain text in the target VM and in the VM used to create the keys.

The passphrase must be stored as plaintext in order to allow automated backups.

If the VM (or a given backup client) is compromised and this password is leaked, the backups from this VM are also compromised.

Therefore, each VM (or backup client host) must use a different, strong passphrase such that the compromise of one passphrase does not imply the compromise of all backup repos.

SSH Keys

The same considerations apply to the ssh keys used to connect to the backup server as the ones to the borg repo passphrase.

The keys must be passwordless (in order to allow automated backups) and also unique among the clients such that a compromise of one key does not immediately give access to the others.

Backup Maintenance VM

It is the recommended Qubes philosophy to use a dedicated VM for a given set of activities.

One possible application of this is to use a VM dedicated to backup maintenance (the VM which will have the admin-keys.openssh) and this VM will generate the keys and passphrases for the other VMs.

In this situation, if the backup maintenance VM is compromised, all keys and passphrases may be compromised.

To mitigate against this, after verifying that the new repo is working, you should remove all files (except the *.pub files) from the host-keys dir.

You could also generate the ssh key and passphrase on the client VM and initialize the repository from there.

Backup Accessibility

Losing access to the backed up data in the event of loss of the primary data is as bad as not having a backup in the first place.

To prevent losing access to your backups using this workflow on rsync.net you should have an offline, secondary backup of the relevant keys and passphrases.

At the very least, you should have an offline copy of:

  • The e-mail credentials used to register on rsync.net
  • The passphrases for each borg repo
  • The rsync.net user password

You should also try to maintain copies of:

  • The admin private key used to perform backup maintenance
  • The key file for each borg repo

This should be stored offline (preferably on multiple media) and on a different site. Obviously, you should also store all this information encrypted with a strong master password.

One possible approach to achieve this is to save all this information in a KeePassX file and then print it using PyPaperBak.

Additional Backup Copies

You can also further mitigate risk of data loss by having a copy of all data on rsync.net either locally on another computer/disk or on another provider.

Rsync.net supports the s3cmd utility, so you could clone your data to S3 or Glacier.

Another option is to rsync it to some other backup provider.

Another strategy altogether would be to use Borg to backup to a local computer and then rclone it to rsync.net (or backblaze or S3).