Allow "pulling" backups from remote servers

[fd0]

At the moment, restic only supports creating backups of local files and directories and saving the data to a remote server. This issue is about the other way around: Having a dedicated backup server that logs into the to be backed up systems, gets the data, and saves it locally. For example, this may be implemented by creating an SSH connection to the remote server and starting a restic binary there, which communicates with the local restic binary over stdin/stdout.

Is that a revelant or interesting use case? What do you think?

[ariselseng]

I for one would love this feature. In my case I really trust the backup server as I have cared for that much more than the web servers where my websites is hosted. So for my this issue's use case is much better than the default restic behaviour.

Also It would be awesome if restic didn't need to be installed on the remote server, but could be pushed on the fly ( if same arch as local of course). This way I could upgrade restic on my backup server and all potential compatibility problems would simply go away.

[ar-jan]

+1, that's exactly my use case. I was thinking of using sshfs to accomplish this for now, but what you describe sounds like it'll be faster and more reliable.

[fw42]

👍 I think this would be useful (preferably without needing code execution access on the remote server, just sftp).

[bchapuis]

👍 but I think the deduplication should still be performed on the remote server, before transmitting the data, in order to optimize the bandwidth.

[fd0]

@fw42 without code-access via sftp is possible, but harder to implement (I think).

[fw42]

good point about deduplicating server-side, I guess that doesn't work without real ssh access

[ariselseng]

sftp/sshfs is way too slow for big backups. Getting a file list over sshfs is multiple orders of magnitude slower. We are talking minutes instead of seconds. So even if no files have been changed, it needs a long time to just detect if anything is changed.

[ariselseng]

Some benchmarks with the find command:
I did everything twice to let it cache. The second command obviously too fast, because of disk caching though...

[cowai@cowai-desktop ~]$ sshfs server:/media/raid servermount
[cowai@cowai-desktop ~]$ time(find servermount|wc -l)
76441

real    5m1.678s
user    0m0.760s

[cowai@cowai-desktop ~]$ ssh server 'time(find /media/raid|wc -l)'
76441

real    0m0.156s
user    0m0.032s
sys 0m0.136s

[cowai@cowai-desktop ~]$ time(find servermount|wc -l)
76441

real    4m34.652s
user    0m0.743s
sys 0m2.717s

[cowai@cowai-desktop ~]$ ssh server 'time(find /media/raid|wc -l)'
76441

real    0m0.145s
user    0m0.044s
sys 0m0.112s

[fd0]

Yes, sftp (which is used by sshfs) is a rather old and clunky protocol. Which version of sshfs is that? I'm curious because I've once submitted a patch that speeds up READDIR by several orders of magnitude... does it have the option -o sync_readdir?

[ariselseng]

$ sshfs -V
SSHFS version 2.5
FUSE library version: 2.9.4
fusermount version: 2.9.4
using FUSE kernel interface version 7.19

I am on archlinux connecting a debian 7 server.
I tried with -o sync_readdir, but the command took a minute longer with it.

[fd0]

Ok, then you have a version with my patch included. -o sync_readdir enables the old behavior and is expected to be slower.

[yatesco]

For me, I wouldn't be interested in this and would be concerned about the added complexity. It feels somewhat like re-inventing the wheel given how many excellent tools there are already in this space.

The approach I use is that the central backup server contains a copy of the data to be backed up, runs restic on that backup to a local destination and then rsyncs the repo offsite.

To get the data to the backup server I either use cron/rsync or backuppc (brilliant tool BTW) with a regular dump to the local FS.

Sure - this requires 3 times the amount of disk space (once for the data on the client machine, once for the source copy on the central server and then the local restic repo) but:

• disk space is cheap, really cheap
• built in redundancy
• restorations are really quick due to the local repo

So for me, no, I wouldn't use this. Ironically, one of the reasons I am considering restic instead of borg is because it has no requirements on the remote server (initially I was going to have restic push to an sftp destination rather than local rsync) and yes, I can do the same local rsync with borg.

[dimsua]

+1, that's exactly my use case. We run backup centralised from backup server.