Backups

This is a collection of notes about production backups.

Naming

We assume daily backups. They can be stored in /gemstorage/backups or somewhere (doesn't matter) and we can format the path with something like this. For sanity the directories need to be namespaced by date, but for additional sanity the backup filenames also need to be; and also need the release version encoded.

~$ RELEASE=20180718-2f8dfb75
~$ BACKUP_PATH=`date -u +%Y/%m/%d/%Y-%m-%d-$RELEASE.tgz`
~$ dirname $BACKUP_PATH
2018/07/20
~$ echo $BACKUP_PATH
2018/07/20/2018-07-20-20180718-2f8dfb75.tgz

Mechanics

The application container can include the cron job that triggers backups, which will let us avoid having to maintain a custom Postgres image.

So we need to add a crontab file to the mappings (like we did with the git history) and then we need to update the package stuff to install the crontab and turn on the cron service.

COPY crontab /etc/cron.d/cool-task -- do this via mappings
RUN chmod 0644 /etc/cron.d/cool-task
RUN service cron start

Race Conditions

When we deploy a new version we have to shut down the old version, but we don't want this to happen if it's in the middle of a backup. Note the following:

• docker stop issues a SIGTERM to the entry point process, and if it doesn't terminate within 10 seconds it follows up with SIGKILL. You can control this with --time <seconds>.
• When Postgres gets a SIGTERM it starts an orderly shutdown, which means it will stop accepting new connections, and will wait for all open connections to close before shutting down.

So this means we just need to ensure that we're doing docker stop --time <seconds> with a sufficiently large max to handle backups. Because this value will grow with time we'll need to keep an eye on it.