Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Automated Backup #117

Merged
merged 4 commits into from Apr 22, 2019

Conversation

@JeffVandrewJr
Copy link
Contributor

commented Mar 25, 2019

My description didn't carry, I'll add it below in a sec.

EDIT: See description and caveats below ⬇️

@JeffVandrewJr JeffVandrewJr changed the title add backup Add Automated Backup Mar 25, 2019

@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Mar 25, 2019

This is a response to #116 @Kukks @NicolasDorier.

Adds the following:

backup.sh: Stops all containers cleanly, creates a tarball of var/lib/docker excluding the bitcoin blockchain, then hands off the tarball to one of many potential cloud uploader scripts.

dropbox-script: This is a cloud upload script for Dropbox that I threw together. The subdirectories are just all of the script dependencies; there are hundreds of files but are minimal in size. I included them directly in the repo so that they'll automatically come along when a user runs btcpay-update.sh. Obviously if you're going to bag this Dropbox script and write your own uploader you can ditch this entire directory and all subdirectories. EDIT: I posted some alternatives to including these files in a later comment below.

Right now Dropbox is the only uploader script that's actually written, but backup.sh is written such that many others could follow.

The following environmental variables are expected to be set:

BACKUP_PROVIDER: Right now the only one that works is Dropbox.
DROPBOX_TOKEN: The Oauth token for Dropbox. The token can easily be obtained at developers.dropbox.com, Create App, give it access only to the special app folder, and then click the button on the dashboard to "generate token." By choosing the option for a special app folder, it keeps BTCPay stuff away from the user's other Dropbox data.

I presume ultimately you guys will set those env variables through the web interface, or you may bag my Dropbox client altogether for something else you're building.

DO NOT USE WITH SENSITIVE DATA: backup.sh right now sends the tarball to Dropbox unencrypted. Obviously this is bad, but I left this open since I didn't know how you wanted to do the encryption (password, key, etc). Easy enough to add a pgp line to the script.

@Kukks

This comment has been minimized.

Copy link
Member

commented Mar 25, 2019

whoa, that is a lot of new files

@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Mar 25, 2019

@Kukks Those 100 files are the Dropbox SDK library and its dependencies. I included them whole since btcpay-update.sh uses git on this repo to pull all the files it needs.

Without those, this is just two files: backup.sh and dropbox-script. But then you'd have to install the rest of the Dropbox dependencies through pip. Or I could compile the Dropbox files all together as one static binary file using Nuitka.

If you're not using my Dropbox script than it wouldn't matter at all and you'd just need backup.sh.

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

image
omfg.

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

Without those, this is just two files: backup.sh and dropbox-script. But then you'd have to install the rest of the Dropbox dependencies through pip

No, just need to use a python docker image.

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

I like this.
Though the best would be that it was encapsulated inside a docker image.

The host would just need to run the docker image and bind the volumes folders. That way, we have repeatable backup solution for any distrib.

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

@JeffVandrewJr Can you dockerify it and put it to https://github.com/btcpayserver/dockerfile-deps ?

Don't need to do the ARM part if you don't have the time, I can do it once you have the amd64 working.

This repo would then just run a command on the backup docker image.

@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Mar 26, 2019

@NicolasDorier

Sure, I can do that.

In my head I didn't run it in Docker to begin with because I was thinking Docker would be shut down for the backup, but I was being stupid because obviously I'm bringing Docker back up before the upload starts anyway.

I'll work on that tomorrow.

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

@JeffVandrewJr I am not even sure that it is worth it to shut down docker containers.

@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Mar 26, 2019

@NicolasDorier

Dockerizing was easy so I made the changes before bed. See the changes thta I made both to this PR and also btcpayserver/dockerfile-deps#4

I have backup.sh stopping the containers briefly because I think there is value in taking a static backup of the lightning folders. I believe I've had conversations with @PierreRochard about his Node Launcher where he takes backups of lightning folders upon shutdown for this reason. [Pierre if you see this and I misremember, correct me!]

Lightning aside, it seems like there would be less chance of getting a backup of corrupt data by stopping everything for a few minutes. But if I'm off base, we can just remove that line from backup.sh and we'd be good to go with no stoppage.

Changes made to file locations:

  • I removed my Dropbox client (and 100k lines!) from the btcpayserver-docker/repo
  • I moved my Dropbox client to the dockerfile-deps repo

What stays the same:

  • backup.sh continues to live in btcpay-docker repo, since it launches the Dropbox container when needed

Changes made to backup.sh:

  • backup.sh now checks to see if a Docker volume exists for the Dropbox client. If it doesn't, it uses docker volume create to make one.
  • After the volume check, it makes a tarball and sends the tarball directly into the Docker volume
  • backup.sh then launches the Dropbox container
  • Post-upload, the Dropbox container automatically shuts itself down gracefully

To do:

  • Choose what encryption scheme we want to use and add the appropriate command to backup.sh
  • backup.sh currently uses the Docker image for the Dropbox container from my personal DockerHub. After you merge btcpayserver/dockerfile-deps#4 you can push an image to the BTCPay DockerHub and use that image instead
  • Maybe have a way to set the STORAGE_PROVIDER and DROPBOX_TOKEN environmental variables through the web UI under Server Settings.
@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

@JeffVandrewJr I think the image for dropbox is perfect. However, you should allow the user to put his own file name in case he wants to keep multiple backups.

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

Ideally here what we should have:

STORAGE_PROVIDER=dropbox DROPBOX_TOKEN=abc backup.sh januarybackup.tar.gz

backup.sh would:

  • Create the januarybackup.tar.gz archive
  • Mount januarybackup.tar.gz to btcpay-dropbox and run (note you do not need to explicitely pass the environment variables if your btcpay-dropbox image declare the variable with ENV DROPBOX_TOKEN)
  • Delete januarybackup.tar.gz if succeed.
@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Mar 26, 2019

@NicolasDorier I have implemented all of your suggestions and updated backup.sh accordingly to allow the user to name his backup file. The docker run command passes any filename argument along as an argument to the container entrypoint ($filename after the image name on that line).

The only exception is that I did continue to pass the DROPBOX_TOKEN from the docker run command rather than through the Dockerfile. If I set ENV DROPBOX_TOKEN in the Dockerfile, it sets it from the CircleCI build environment.

As mentioned above, this is still currently unencrypted.

@astupidmoose

This comment has been minimized.

Copy link

commented Mar 27, 2019

Thinking of having this as an automated cron, Would it make more sense to change the filename="backup.tar.gz" to perhaps filename="backup_$(date "+%b-%d-%Y-%H-%M-%S").tar.gz" or something similar? This would ensure duplicate files are not overwritten, unless thats the goal?

@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Mar 27, 2019

@astupidmoose

The filename is only backup.tar.gz if you don't pass the script an argument. Otherwise the argument is the file name. You could pass what you've got there as an argument when you write your crontab if you want something like that.

At least with the Dropbox client, you'd probably want the files to be constantly overwritten because Dropbox automatically stores prior versions and allows you to restore them. The prior versions don't count against your storage alottment if you do it that way.

@astupidmoose

This comment has been minimized.

Copy link

commented Mar 27, 2019

Yea, your right that probably makes more sense :)

@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Mar 27, 2019

Changed the backup to the /var/lib/docker/volumes level rather than /var/lib/docker. Thanks to @astupidmoose for suggesting based on his speed/size tests.

@Kukks

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

This is cool, once this is merged, I will add a new provider to the bash script "BTCPAY", which will copy the backup file to a drop folder in the btcpay docker volume which in turn will trigger a file upload to any of the configured cloud providers. :D

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2019

Will review and test it later today. It is super cool.

@JeffVandrewJr I think the Restore part should be handled by the same image though. (we can do later though, it should not be that complicated)

@JeffVandrewJr JeffVandrewJr force-pushed the JeffVandrewJr:master branch from d131667 to 2931cb9 Apr 20, 2019

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2019

Merging this, I need to properly document it though. @pavlenex might interest you.

@NicolasDorier NicolasDorier merged commit 8457239 into btcpayserver:master Apr 22, 2019

@JeffVandrewJr

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2019

@NicolasDorier The backup tarballs this produces are uploaded unencrypted right now, as I wasn't sure how we wanted to handle the encryption step (password based encryption? Enforce strength?).

@NicolasDorier

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2019

unsure if it should be encrypted. This would be another piece of data (key, pass) that need to be saved and can be leaked and lost by the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.