What's better, many small repos or one monolithic repo?

[pixelrebel]

I want to backup all users' home directories with borg and then sync the repo to S3.
I prefer to have one large repo for this and create archives for each user in same repo. The reason for this is I only want to manage one encryption key. Every time a new user is created, I don't want a new key.
My concern with one large repo is that if it becomes corrupted, all users are affected. Whereas, if I create a repo for each user, corruption can be mitigated to a user.
Is this crazy talk? Hoping someone can tell me, "that's not how borg works, doesn't matter if you use one or 1000 repos."

[RonnyPfannschmidt]

it is doable to just use the same key/key file for all user repos,

when a borg repository becomes indeed corrupt, its usually because a segment file got corrupted
and the affected archives are the the ones using data of that segment

so its mainly depends on if your users share data, and how much of it they share

note that a borg repo that is made aware of corrupt segments can correct them the next time it backs up the same data

that said, having many repos is a standard mitigation technique

[FabioPedretti]

When using only one repo you can also save space, since deduplication can also happens between archives. Just remember to set up a cache TTL (default 20) >= number archives to cycle, or performance will suffer! See here.

For that you may want to use something like this before using borg in your script:

export BORG_FILES_CACHE_TTL=$(expr `ls /home/* | wc -l` \* 4)

[ThomasWaldmann]

If you use same AES key for different repos, borg will manage IVs (NONCEs) for the repos independently and same key+iv combination will be used (not in same repo, but in different repos - likely even N times) and this makes the crypto unsecure. So, don't copy/reuse keys.

[ThomasWaldmann]

Another thing to consider is parallel execution. If you have N repos, you can run N backups in parallel or overlapping.

[pixelrebel]

Thanks for the tips! I think I will run borg unencrypted and let rclone handle the encryption.

@ThomasWaldmann Do you suggest I feed my home directories into parallel borg create...? Would that make for fragmented writes on the disk? Does that even matter for backup data?

[ThomasWaldmann]

Well, if you have a lot of source directories on different hosts, feeding them all into 1 repo might give a timing / serialization problem / total runtime problem. Also, the multiple clients will need to resync their chunks cache often. If you use multiple repos, you can run N in parallel in likely less time.

If all homes are on same host, i guess it is better to just use one repo (except if they are exceptionally huge or contain exceptionally many non-duplicate files).

[alisianoi]

In my case, I would like to have different systemd timers that fire borg at different frequencies for different folders (and also have different pruning strategies). However, this means that borg instances fired by different timers could be asking for repo access at the same time, so some instances get the access and others fail with "could not aquire lock".

I believe this is a use case for multiple "thematic" repo's

[xrysf03]

Thanks for this thread. I may have stumbled upon one other aspect: rotation of the backups. When pruning, you can say --keep-daily, --keep-weekly, --keep-monthly etc. Supposedly, this is a number (count) of days, on which backups have occurred - but, only the latest file (per day) will be kept! Thus, if you'd like to back up "two different assets" every day into a shared repo, you'd end up with only one of the assets backed up every day!
=> if you want to use the built-in pruning, you probably must use a separate repo for each "backup topic".
An alternative is to write your own script to do the "pruning" using the delete command. In that case, I believe you also need an explicit "compact", following your custome "prune by a sequence of deletes"...

I myself have previously written a Perl script to rotate my rsync backups, where I back up several files every day - the script counts unique days from which there are files, and keeps any files (backup archives) for those days that it did count. I thought that with Borg, I wouldn't need to resort to manual scripting. Well... that's correct, but apparently I need to split my repo into two or more :-(

[jdchristensen]

if you want to use the built-in pruning, you probably must use a separate repo for each "backup topic".

No, that is not necessary. You can filter the archives considered for pruning, using for example -P prefix. So you just need to name the archives for each "topic" with a different prefix.

[xrysf03]

@jdchristensen thanks for the tip, sounds spot on - and my respect to you for the zero latency in your response :-)

[ThomasWaldmann]

everybody wanting to prepare a bit already for borg2 can also rather use -a (aka --glob-archives) instead of --prefix (which will go away in borg2 and get superseded by --match-archives, which is similar to --glob-archives, but more powerful).