Backup regularly the database

[kelson42]

We need to backup our Zimfarm database. This should:

• happen on a remote location
• be easy to configure
• run periodically
• allow incremental archives
• backups should better be encrypted
• backups secrets should be stored in a secure manner but not at the same place than the backup solution

[kelson42]

We need a solution which could be easily deployed for our other online statefull services. A Docker image which would be easily configurable via environment variables would be the best. We could call it the "borg-companion".

I think we could do that using a custom Docker image:

• using Borg client
• using borgbase.com
• using Bitwarden and its API https://help.bitwarden.com/article/public-api/ to retrieve Borg configuration (with secrets).

To use the borg-companion we would need:

• To mount paths which need to be backuped
• Give Bitwarden secret to download Borg configuration/secrets
• A proper README explaining how it work and (more important) how to retrieve the backups

@m3nu Does that sounds doable? You might have heard about solution already existing which do that?

[m3nu]

Thanks for the shoutout. 👋

The plan sounds reasonable. If you keep the private SSH key in Bitwarden, it would be quite simple really. One would just have to write the config to a Borgmatic template and be done.

The Docker setup steps would be similar to the Ansible role we offer to set up Borgmatic on real servers. The Dockerfile would be like this:

1. Pull some base image (CentOS 8?)
2. Install Borg and Borgmatic
3. Write private SSH key from config
4. Write Borgmatic config from config and template
5. Mount backup paths
6. Run Borgmatic

One thing to consider is Borg's cache in ~/.config/borg. If you have many files, you will want to keep that folder outside the Docker container. Else Borg would re-download it every time you rebuild the container.

Stretch goal: If you have a lot of backups to do, you could use the BorgBase API to create more repos on the fly and assign the correct keys. This would need a small Python client in the Docker image.

[m3nu]

I didn't look into the precise use case yet and don't know much about Bitwarden, but for everything Docker and Borg, I'm happy to help with setup and testing.

[kelson42]

@m3nu Thank you very much for your extended feedback. Hopefully we will be able to do that in the next months ans share the result with the rest of the borg community.

[kelson42]

Someone has done a first step in that direction here https://github.com/rubenbe/borgbase-docker/

[kelson42]

I have started a POC at https://github.com/kiwix/maintenance/tree/master/backup-docker

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

@m3nu We have a first version running of this "backup-companion" at https://github.com/kiwix/borg-backup. Feedbacks welcome.

[m3nu]

Interesting. Especially the use of Bitwarden to keep secrets. Will try to run it in Docker.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

@rgaudin Coukd we close this ticket?

[rgaudin]

sure