efficiently transfer an archive from one repository to another

[quickslyver]

• I have checked borgbackup docs, FAQ, and open Github issues
• It is a QUESTION

Use case (context)

1. I have a local backup repository where there are multiple archives (dailies, weeklies and monthlies archives).
2. I would like to have a much smaller offsite backup repository where only the most recent daily archive is copied

Problem

Currently it seems there is no command available in borg to copy archives between repository. (true ?)

Workaround

I've found the following workarounds that all have drawbacks:

1. mount the archive i want to copy on the local backup repository (fuse module) and run a borg create on the offsite backup repository. But the fuse module seems not to be designed for that kind of "heavy duty access" (slow)
2. "borg extract" the archive on the local backup server filesystem , "borg create" the archive on the offsite server. But it wastes a lot of storage on the local backup server.
3. backup directly from the source servers (the backed-up servers) to the offsite server: then the backed-up need to have access directly to the offsite server and they need to be backed-up twice (for the local and the offsite server)

The perfect solution would be a direct copy of the archive from one repository to another using a borg backup command

Questions

Two questions:

1. Is there a better workaround for my use case ?
2. Is adding a command to copy an archive between two different borg repository feasible ? (I can help)

[ThomasWaldmann]

Use Option 3. Archive transfer from one repo to other is neither trivial nor especially fast due to security reasons.

…

On June 1, 2018 5:35:51 PM GMT+02:00, quickslyver ***@***.***> wrote: - I have checked borgbackup docs, FAQ, and open Github issues - It is a QUESTION ## Use case (context) 1. I have a local backup repository where there are multiple archives (dailies, weeklies and monthlies archives). 2. I would like to have a much smaller offsite backup repository where only the most recent daily archive is copied ## Problem Currently it seems there is no command available in borg to copy archives between repository. (true ?) ## Workaround I've found the following workarounds that all have drawbacks: 1. mount the archive i want to copy on the local backup repository (fuse module) and run a borg create on the offsite backup repository. But the fuse module seems not to be designed for that can of "heavy duty access" 2. "borg extract" the archive on the local backup server filesystem , "borg create" the archive on the offsite server. But it wastes a lot of storage on the local backup server. 3. backup directly from the source servers (the backed-up servers) to the offsite server: then the backed-up need to have access directly to the offsite server and they need to be backed-up twice (for the local and the offsite server) The perfect solution would be a direct copy of the archive from one repository to another using a borg backup command ## Questions Two questions: 1. Is there a better workaround for my use case ? 2. Is adding a command to copy an archive between two different borg repository feasible ? (I can help) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #3867

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[quickslyver]

Archive transfer from one repo to other is neither trivial nor especially fast due to security reasons.

yes, I understand that the algorithm would be something near that of "borg recreate with re-chunking", rechunking is needed due to different encryption key.

[wolfpackmars2]

Couldn't you just use two backup repositories? The local copy which has multiple backups and the remote backup, which presumably you could auto-prune to the last backup at the end of a backup operation. Yes you would be running borg backup twice, but it's probably ultimately less overhead than trying to keep two separate backups in sync.

[quickslyver]

Yes that's the workaround number 3 !

But with my real use case it's a little cumbersome.

Currently:

• I backup (using rsync) every day tens of servers to a local backup server (currently on ZFS)
• I make snapshots every day a keep a fixed number of daily/weekly/monthly (classic...)
• every week I physically externalize the most recent backup to a 8TB HDD using borg backup (and I can't do this using a network connection)
• I'm trying to switch to borg backup instead of ZFS snapshot for the local backup (which triggers the problem of transferring the most recent borg backup of one repo to another)

I need to make some tests but I think that workaround 1 (fuse mount the source) is a more simple solution. Staying on ZFS snapshots for primary local backup is also viable

---

N.B: advantages of borg backup over ZFS snapshots for the local backup:

• deduplication
• easier to manage
• ability to delete big file/directories backed-up by mistake using borg recreate (this task is very complex with ZFS snapshots)

[hachre]

I have a similar use case for such a potential feature.

In my case there's a central big Borg repo on a server. It holds many backup states of several other servers because I want to make maximum use of the deduplication. Because of the amount of states stored in there, the Borg cache is quite large (around 6 GB).

Now I want to start backing up a bunch of VPSes into that same repo but they can't afford the 6 GB cache due to disk space constraints. The proposed solution of removing chunks.archive.d is not acceptable either in this case because it takes too long to reread all the cache indexes with each backup.

I thought of a solution where I'd create empty Borg repos on the central Borg repo machine (one for each VPS) and always leaving just 1 additional state in there. This way the cache would be small on the clients and backups could still be done "incrementally". I would then need to be able to rotate the oldest backup out into the main repo each time the "small" repo reaches 3 stored backups. This way I'd have a kind of staging area for backups that the clients use and the backups would still ultimately end up in the large repo for deduplication.

I also tried the mount method described by the op (workaround 1) but it was incredibly slow to do it this way, but that's what I'll probably continue to use for now.

[tg-x]

Use Option 3. Archive transfer from one repo to other is neither trivial nor especially fast due to security reasons.

@ThomasWaldmann what about backing up to multiple repositories at once to avoid running backup multiple times? would that be feasible to implement?

[ThomasWaldmann]

@tg-x iirc we have another ticket about that. and it doesn't quite fit into this ticket here.

update: hmm, couldn't find it, maybe it was also on the mailing list or irc.

[ThomasWaldmann]

Guess I need to close this, there is no "efficient way":

• different repos have different AES and hmac keys, thus authentication, decryption, encryption, adding auth tag is required.
• different repos also have a different chunker seed, thus decompression, streaming, chunking, recompression is required.

[ThomasWaldmann]

There are other tickets about archive export and import (as tar).