New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

suggest remote backup storage options? #2177

Open
ThomasWaldmann opened this Issue Feb 18, 2017 · 40 comments

Comments

Projects
None yet
@ThomasWaldmann
Copy link
Member

ThomasWaldmann commented Feb 18, 2017

@ThomasWaldmann ThomasWaldmann changed the title suggest some backup storage options? suggest remote backup storage options? Feb 18, 2017

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Feb 18, 2017

After we collected some options, this ticket could be closed by moving this to a section on the community repo link list.

Note: this should not get a complete list of server offers, but rather one of the more interesting ones concerning value/price and/or ease-of-use when used with borg.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Feb 18, 2017

@twink0r hmm, some of these don't seem that interesting (looking at price) when comparing to e.g. rsync.net. add some details?

@enkore

This comment has been minimized.

@milkey-mouse

This comment has been minimized.

Copy link
Contributor

milkey-mouse commented Mar 8, 2017

rclone now has a mount option that mounts the backend as a FUSE filesystem. Theoretically one could just mount and backup directly into it, though at least for Amazon Drive, Amazon's "eventual consistency" stuff causes problems like files spontaneously disappearing after upload, only to reappear 30 seconds later (which is acceptable for an online cloud storage website, but not so much for a filesystem).

I was also thinking about the fact that many cloud providers theoretically offer "unlimited" storage, but throttle & have a limit in the fine print. According to an /r/DataHoarder thread it seems relatively safe to store one's backups on Amazon Drive, provided you encrypt first.

And, of course, there are always small VPS instances from Azure, Google Cloud, AWS, Vultr, etc. which can just run borg serve on an SSH connection.

@Vertux

This comment has been minimized.

Copy link

Vertux commented Mar 18, 2017

For me the most appealing offer regarding value/price is Amazon S3 Standard or Glacier for long term backups. And another plus is, that you can choose the location where your data is stored e.g. Frankfurt. I think this is a handy tool to access S3 to make backups with Borg.

@DArtagan

This comment has been minimized.

Copy link

DArtagan commented Mar 28, 2017

I think these people have done an interesting thing: borg does the backups and then they use git-annex to keep an index of the chunks and manage syncing the chunks to remote storage (e.g. B2).

I think the first one has an especially good description, the second is more skimmable:

If we can move away from VPS-type options, that eliminates a common cost. From what I can tell, using this combination opens up inexpensive storage options like B2. How do you think this approach stacks up against rclone?

@willsALMANJ

This comment has been minimized.

Copy link

willsALMANJ commented Jul 1, 2017

It seems like #1795 can be closed in favor of this.

The posts about using borg with git-annex, rclone, etc. are good to know about, but not particularly special. They just describing normal borg use -- backing up to a local disk -- and then using another tool to copy files to cloud storage.

I think rsync.net is definitely worth highlighting. It is awesome that they provide such a customized borg offering. It would be nice to highlight some VPS's that have reasonably priced large storage options, though I don't know how often such a list would need to be curated. I don't see much value in highlighting making a local backup and copying it to the cloud since that can be done with pretty much any service. Maybe one day #102 will be closed and more cloud storage options will work better with borg.

EDIT: something like the rclone option that can mount remote storage locally would be interesting but the rclone mount seems unstable at the moment.

@DArtagan

This comment has been minimized.

Copy link

DArtagan commented Jul 5, 2017

The day #102 closes will be a good day indeed.

Wow, #1795 has the same git-annex links as I included, and here I thought I'd uncovered two previously unknown gems.

However, I do think making a local backup and copying it to the cloud is hugely important: if done right it breaks us from needing a VPS (higher cost/complexity). But doing it right is the catch, just copying the files doesn't prevent something going wrong with the source repo that you're copying from. That's why the borg documentation talks about having redundant, independent repositories. This is where git-annex comes in, as an alternative to the VPS/rsync.net/SSH approach. As crazy as this sounds: it's version control for your backups. Keeping the backups version controlled allows the use of dumb, cloud storage. At least... that's what I'm thinking, but I welcome further consideration.

@markfox1

This comment has been minimized.

Copy link

markfox1 commented Jul 14, 2017

@DArtagan is thinking in the right direction here. Requiring a VPS adds expense and complication.

I'm on the tail end of a four-day deep dive into backup systems. (There were earlier dives, but no so deep.) The best solutions I have been able to find are HashBackup, Tarsnap, and Duplicati.

All are client-side only, in the sense that there is no need to read data back from the remote site when doing a backup. (This is actually not completely true, but very close.) In a nutshell, they do local book-keeping so that they can dedup, encrypt, and send only incremental changes to the remote store. So the first backup is the only painful one.

Each has its flaw too. Neither HashBackup or Tarsnap are open source. HashBackup, although free for the moment, may adopt a license in the $250/year range. Tarsnap is actually quite expensive at $0.25/GB/month stored and $0.25/GB of bandwidth. Duplicati goofed on fixed block offsets vs checksum-based offsets and the security model could be better. (Keep in mind the implications of a compromise of the backed up box.)

But the algorithms each use can make simple backups no more expensive than the cloud-storage and the bandwidth required to get them there. We're talking around $5US/TB/month. That price opens up a lot of doors. It's a big deal.

Borg is well positioned here. It could be the backup tool in fairly short order.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jul 15, 2017

@enkore do you use hetzner storagebox yourself? which protocol / filesystem?

@enkore

This comment has been minimized.

Copy link
Contributor

enkore commented Jul 15, 2017

I haven't rented servers in a while. Someone told me they use it with iirc sshfs or sambda.

@wzyboy

This comment has been minimized.

Copy link
Contributor

wzyboy commented Jul 19, 2017

I am a new user (~4 days of usage) of BorgBackup. After reading this thread I started to using B2 (BackBlaze's S3) as an offsite backup. I have to admit I am really enjoying using B2 -- much lower price than S3/Glacier, reasonable daily free tier usage quota, and rather satisfying uploading speed.

Here's my setup (complies with BackBlaze's 3-2-1 backup strategy):

  • I have a HP Gen8 microserver with 128 SSD and 2 TB + 4 TB HDD. (JBOD, with mergerfs for painless "unionfs-like" access);
  • A systemd timer on my laptop runs borgmatic daily, backing up /home and /etc to the home server via SSH;
  • From the home server, rclone uploads the Borg repo to B2. Because Borg never "modifies" a data file (it just "creates" or "deletes" one), it's rather safe even if a borg create is in progress when a rclone sync job is running.

Note that the current version of rclone uses a new sync method that exhausts your daily free API calls quota quickly if you have a lot of directories. With Borg repo, however, this does not pose an issue, since a Borg repo has very few directories.

@willsALMANJ

This comment has been minimized.

Copy link

willsALMANJ commented Jul 24, 2017

I will experiment with rclone sync and rclone mount. My reservations regarding rclone sync and git annex style options is that for local backup I prefer a plain copy of the files with some kind of snapshot system because that is the easiest to access, so making a local borg backup and then copying it to the cloud means I need to have two local backups if I keep my file system backup, though maybe having an extra backup is a good thing.

@mannp

This comment has been minimized.

Copy link

mannp commented Aug 30, 2017

S3 Compatible would open up a lot of alternatives, including using minio as a backend.

Edit: as well as Wasabi

@mannp

This comment has been minimized.

Copy link

mannp commented Sep 6, 2017

@wzyboy Have you tested restores and how that would work in your case?

I have the same setup and use Backblaze, but not with borg (yet) and I wondered what happens, say if your local repository becomes corrupted, can rclone just restore the broken parts from your local copy?

I am trying to minimise the download costs from B2 if I have a failure of my local repo.

@wzyboy

This comment has been minimized.

Copy link
Contributor

wzyboy commented Sep 6, 2017

@mannp Rclone only copies necessary data whenever possible. Say if 10% of your local backup is corrupted, only 10% data is needed to be downloaded from B2 to repair your local backup.

In desperate situations, you can even rclone mount your B2 bucket and borg mount on top of that. The performance of stacked FUSE might be terrible, though.

@mannp

This comment has been minimized.

Copy link

mannp commented Sep 6, 2017

@wzyboy ok thanks for that; yes I assume it would only copy what was necessary, but wondered if those files were copied back, than borg would be able to recover. I assume it would if the files were restored the same. Not tried though..

Do you try and restrict the B2 transactions, so they remain within your daily allowance, with --fast-list or the like options?

Do you mind sharing your rclone command line?

Good point, I will try the mount, but was always concerned about how many transactions a mount would incur... have you tried it?

Thanks

@wzyboy

This comment has been minimized.

Copy link
Contributor

wzyboy commented Sep 6, 2017

@mannp

ok thanks for that; yes I assume it would only copy what was necessary, but wondered if those files were copied back, than borg would be able to recover. I assume it would if the files were restored the same. Not tried though..

rclone works bi-directionally.

Do you try and restrict the B2 transactions, so they remain within your daily allowance, with --fast-list or the like options?

No. I have never exceeded my daily transaction quota.

Do you mind sharing your rclone command line?

rclone -v sync ${REPO} b2:${BUCKET}

Good point, I will try the mount, but was always concerned about how many transactions a mount would incur... have you tried it?

I suppose in a desperate situation you would not care about how much transaction fee you would pay BackBlaze :-)

@mannp

This comment has been minimized.

Copy link

mannp commented Sep 6, 2017

@wzyboy Ok cool, thanks for taking the time to respond 👍

No command line extras at all, ok fair enough, thanks again.

@Orbixx

This comment has been minimized.

Copy link

Orbixx commented Sep 26, 2017

I've been using https://i-83.net/ which provides NAT'd IPv4 and native IPv6, very cheap storage VPS. Occasionally goes down for what is presumably maintenance, but works well for a daily Borg backup.

Here's a couple of providers I'm also considering but have not tried - be curious to know if anybody has used them:

https://www.wishosting.com/order/main/packages/VPS/?group_id=2 (see the OpenVZ Mini package)

https://backupsy.com/ (see the 40% off for life coupon, also 250GB minimum. 250GB storage for $6/mo)

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Sep 26, 2017

General note: be careful with memory amount, the repo index needs to fit into RAM. So if your repo has a lot of chunks, it might not fit. See the docs for a formula.

@trendco

This comment has been minimized.

Copy link

trendco commented Sep 26, 2017

At the moment i try the 1TB HiDrive from Strato: https://www.strato.de/online-speicher/

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Sep 26, 2017

note: trendco also noted on irc, that he uses the hidrive via sshfs and that strato creates user-accessible backups of the hidrive. will be interesting to hear practical experiences with that, like speed, reliability, etc.

@Cardes

This comment has been minimized.

Copy link

Cardes commented Dec 4, 2017

Hetzners Storageboxes now offer Borg Backup as a service / repository:

Hetzners Wiki Entry

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Dec 15, 2017

https://contabo.de/?show=vps (from @bebehei on irc)

@bebehei

This comment has been minimized.

Copy link
Contributor

bebehei commented Dec 15, 2017

@ThomasWaldmann What about making a proper list out of this? Like a wiki page?

Here in this issue, it's not possible for one to invalidate an old offer and for future readers it's much easier to have a small comprehensive list instead of a thread where half of the stuff doesn't represent the current state.

@milkey-mouse

This comment has been minimized.

Copy link
Contributor

milkey-mouse commented Dec 15, 2017

it's not possible for one to invalidate an old offer

I mean, technically somebody can edit/delete their original comment. I agree that's not a "scalable" strategy, though.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Dec 16, 2017

Important: do PRs against "source" there (as seen in 2nd url in previous comment), not "master".

@enkore

This comment has been minimized.

Copy link
Contributor

enkore commented Dec 16, 2017

You have to explicitly select "master" instead of "source" when creating PRs, btw.

@hi2u

This comment has been minimized.

Copy link

hi2u commented Jan 13, 2018

Is anyone using Borg on a time4vps storage VPS?

The latest version of Debian they offer is Debian 8 with kernel 2.6.32.x - and it's using OpenVZ, so you can't upgrade it yourself.

@ThomasWaldmann Any rough idea on how long you think new versions of Borg will continue to work under Debian 8 with kernel 2.6.32.x?

@jeanregisser

This comment has been minimized.

Copy link

jeanregisser commented Jan 15, 2018

@hi2u I'm using a 1 TB time4vps VPS. No issue sofar.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jan 15, 2018

@hi2u borg is currently tested and built (standalone binary) on Debian 7. When LTS support for D7 ends, it is likely that we will switch to Debian 8 for that (and keep it there as long as D8 receives security updates / LTS support).

No idea about the old kernel.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Dec 17, 2018

There is also https://borgbase.com/ now.

@Vertux

This comment has been minimized.

Copy link

Vertux commented Dec 17, 2018

Just for reference, Hetzner offers a great BorgBackup support too, depending of your rented server you get 100GB for free.

@mannp

This comment has been minimized.

Copy link

mannp commented Dec 17, 2018

Hetzner want 2.90 euro + vat for borgback storage box though, its not free?

borgbase has 100g free while in beta but seemingly no prices for outside of the beta; also based in Hong Kong by the look of it.

Anyone used borgbase?

@Vertux

This comment has been minimized.

Copy link

Vertux commented Dec 18, 2018

@mannp

Hetzner want 2.90 euro + vat for borgback storage box though, its not free?

As I already said, if you rent one of the "bigger" Hetzner Servers e.g. EX41-SSD you get 100GB Backup Storage with Borg Support for free. If you want just a Storage Box, you have to pay for it, they do not offer any storage for free. I personally have no experience with borgbase.

@mannp

This comment has been minimized.

Copy link

mannp commented Dec 20, 2018

@Vertux Sorry to waste your time repeating yourself.

The fact that your links first task is to setup your 'storage box' is what threw me.

Activate Borg and configure your storage box

@Vertux

This comment has been minimized.

Copy link

Vertux commented Dec 20, 2018

No need to apologize @mannp I should have made that clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment