add borg backup

[etkecc]

Note: the used docker image supports multiple architectures and builds weekly

Another note: we use rsync.net borg and that PR was developed and tested on it, but other vendors (eg: hetzner storage box) work in the same way, so there is no vendor-lock

[HarHarLinks]

In my fork of the playbook I use m3nu.ansible_role_borgbackup. For the DB dump to work I needed to also install a psql-client that matches the container and make the container accessible from the host.

In my experience, trying to back up all of /matrix fails with errors such as /matrix/postgres/data/pg_wal/0000000100000059000000B5: stat: [Errno 2] No such file or directory: '/matrix/postgres/data/pg_wal/0000000100000059000000B5'. Also, my deduplicated archive size shrunk from ~2GB when including /matrix/postgres/data to <200MB after excluding it and instead backing up the DB dump.

I have added in mine the ability to post info about running backups and logs to matrix, and - in case they fail - also to email via the email proxy already included with the playbook via apprise.

      borgmatic_failure_command:
        - "\"{{ apprise_docker_call }} --title '[{{ hostname }}] Borgmatic Backup Error' --body '{% raw %}{{app_title}}{% endraw %}<br>@room: there was an error creating a backup!' --tag=error\""
      borgmatic_before_backup_command:
        - "\"{{ apprise_docker_call }} --title '[{{ hostname }}] Borgmatic Backup' --body '{% raw %}{{app_title}}{% endraw %}<br>starting...' --tag=backup\""
      borgmatic_after_backup_command:
        - "\"{{ apprise_docker_call }} --title '[{{ hostname }}] Borgmatic Backup Complete' --tag=backup\""

My cron jobs are roughly

borgmatic_command: "borgmatic -c /etc/borgmatic/config.yaml --no-color"
borgmatic_log: "--log-file {{ borgmatic_log_file }}"
apprise_send_borgmatic_log: "{{ apprise_docker_call }} -e --body \"<pre><code>$(cat /borgmatic.log)\\n</code></pre>\\n\" --tag=backup"

# backup job
{{ borgmatic_command }} --create --stats > {{ borgmatic_log_file }} 2>&1; {{ apprise_send_borgmatic_log }}
# check job
rm {{ borgmatic_log_file }}; {{ borgmatic_command }} --check --log-file-verbosity 1 {{ borgmatic_log }}; {{ apprise_send_borgmatic_log }}

There are some more paths that are not necessary to include in a backup since they are just cached remote files, here are all I set:

      borg_exclude_patterns:
        - /matrix/synapse/storage/media-store/remote_content
        - /matrix/synapse/storage/media-store/remote_thumbnail
        - /matrix/synapse/storage/media-store/url_cache
        - /matrix/synapse/storage/media-store/url_cache_thumbnails
        - /matrix/postgres/data

(note that one ends in thumbnail while the other uses thumbnails due to an inconsistency in synapse)

[etkecc]

Here is the actual log output of that PR without the #1726 fix:

Apr 03 10:43:00 matrix systemd[1]: Starting Matrix Borg Backup...
Apr 03 10:43:03 matrix matrix-backup-borg[6305]: Remote: Warning: Permanently added 'REDACTED' (ED25519) to the list of known hosts.
Apr 03 10:43:09 matrix matrix-backup-borg[6488]: Remote: Warning: Permanently added 'REDACTED' (ED25519) to the list of known hosts.
Apr 03 10:43:10 matrix matrix-backup-borg[6488]: ------------------------------------------------------------------------------
Apr 03 10:43:10 matrix matrix-backup-borg[6488]:                        Original size      Compressed size    Deduplicated size
Apr 03 10:43:10 matrix matrix-backup-borg[6488]: Deleted data:                    0 B                  0 B                  0 B
Apr 03 10:43:10 matrix matrix-backup-borg[6488]: All archives:               28.23 GB             15.10 GB              8.10 GB
Apr 03 10:43:10 matrix matrix-backup-borg[6488]:                        Unique chunks         Total chunks
Apr 03 10:43:10 matrix matrix-backup-borg[6488]: Chunk index:                   21440                49005
Apr 03 10:43:10 matrix matrix-backup-borg[6488]: ------------------------------------------------------------------------------
Apr 03 10:43:13 matrix matrix-backup-borg[6488]: /matrix/postgres/env-postgres-psql: open: [Errno 13] Permission denied: '/matrix/postgres/env-postgres-psql'
Apr 03 10:43:13 matrix matrix-backup-borg[6488]: /matrix/postgres/env-postgres-server: open: [Errno 13] Permission denied: '/matrix/postgres/env-postgres-server'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/web/env: open: [Errno 13] Permission denied: '/matrix/jitsi/web/env'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/web/config/keys/cert.key: open: [Errno 13] Permission denied: '/matrix/jitsi/web/config/keys/cert.key'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/prosody/env: open: [Errno 13] Permission denied: '/matrix/jitsi/prosody/env'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/prosody/config/data: scandir: [Errno 13] Permission denied: '/matrix/jitsi/prosody/config/data'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/prosody/config/certs/meet.jitsi.key: open: [Errno 13] Permission denied: '/matrix/jitsi/prosody/config/certs/meet.jitsi.key'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/prosody/config/certs/meet.jitsi.crt: open: [Errno 13] Permission denied: '/matrix/jitsi/prosody/config/certs/meet.jitsi.crt'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/prosody/config/certs/auth.meet.jitsi.key: open: [Errno 13] Permission denied: '/matrix/jitsi/prosody/config/certs/auth.meet.jitsi.key'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/prosody/config/certs/auth.meet.jitsi.crt: open: [Errno 13] Permission denied: '/matrix/jitsi/prosody/config/certs/auth.meet.jitsi.crt'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/jicofo/env: open: [Errno 13] Permission denied: '/matrix/jitsi/jicofo/env'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/jitsi/jvb/env: open: [Errno 13] Permission denied: '/matrix/jitsi/jvb/env'
Apr 03 10:45:50 matrix matrix-backup-borg[6488]: /matrix/honoroit/config/env: open: [Errno 13] Permission denied: '/matrix/honoroit/config/env'
Apr 03 10:45:52 matrix matrix-backup-borg[6488]: ------------------------------------------------------------------------------
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: Archive name: matrix-2022-04-03-084311
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: Archive fingerprint: REDACTED
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: Time (start): Sun, 2022-04-03 08:43:13
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: Time (end):   Sun, 2022-04-03 08:45:50
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: Duration: 2 minutes 36.38 seconds
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: Number of files: 24708
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: Utilization of max. archive size: 0%
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: ------------------------------------------------------------------------------
Apr 03 10:45:53 matrix matrix-backup-borg[6488]:                        Original size      Compressed size    Deduplicated size
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: This archive:               14.13 GB              7.56 GB            774.33 MB
Apr 03 10:45:53 matrix matrix-backup-borg[6488]: All archives:               42.36 GB             22.66 GB              8.87 GB
Apr 03 10:47:46 matrix matrix-backup-borg[6488]: Backup created.

As it was mentioned in https://matrix.to/#/!cNSQwPqhHKkIZdBnvt:devture.com/$hXtfV8Y7oYav2VW6tH3jsBVV9hodtUHakssKGq0T75M?via=devture.com&via=matrix.org&via=tu-dresden.de the problem you mentioned may be caused by the user mismatch.

Please, keep in mind that it's "mvp" version and it should be enhanced further.

[HarHarLinks]

may be caused by the user mismatch

From your log it looks like it, the matrix user wouldn't have access to some of these files. In my case the job is simply running as root (idk how much worse this is over your solution, but that isn't relevant to the issue at hand) and should not have any user mismatch trouble, instead I believe it is cased by the flow of

1. borg collects a list of paths and starts working
2. in the meantime postgres keeps running and the files get deleted etc
3. borg fails to find a file that does not exist anymore, and might miss new ones. note that this is prevented by backing up a dump instead as that gets created instantaneously

I should mention that this error did not happen immediately, but only after some months of operation. I only caught it thanks to reading the daily logs (being posted to matrix).

"mvp" version

I understand. Some of my hints were given towards later improvements, but the error I experienced certainly shouldn't happen in an mvp.

My current solution achieves these stats with daily backups, started in October 2021

                       Original size      Compressed size    Deduplicated size
This archive:               30.54 GB             20.45 GB            171.51 MB
All archives:                7.29 TB              3.62 TB            216.02 GB

[etkecc]

I'm prepairing a new commit with exclude patterns

[etkecc]

now you can add anything to the ignore patterns

[spantaleev]

Thank you for adding another backup option! We can never have enough of those ;)

I've submitted some comments - mostly about variables that could be named in a more consistent manner. We can merge this as it is, but if we'd like to rename them later, it'd be more trouble (adding deprecation notices in valdiate_config.yml), so it's better if we do it earlier.

[etkecc]

Will apply the suggestions a bit later today

[spantaleev]

I found a few other potential issues. Sorry for the back and forth!

[spantaleev]

Thanks a lot for adding this and for your patience fixing it up! 🙇

[axiopaladin]

This might be a stupid question, but I couldn't find it obviously explained in any documentation: what timezone is the backup_borg_schedule variable supposed to be in? The default is "04:00", but is that in the matrix server's timezone? The remote server's timezone? UTC?

[etkecc]

@axiopaladin your server's (host OS) timezone