Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
hardlinked repositories: a bad idea? #4272
Have you checked borgbackup docs, FAQ, and open Github issues?
Is this a BUG / ISSUE report or a QUESTION?
System information. For client/server mode post info for both machines.
Your borg version (borg -V).
Operating system (distribution) and version.
Debian buster 10
Hardware / network configuration, and filesystems used.
ext4, external USB 1.5TB HDD
How much data is handled by borg?
Around two 80GB original size sets, 200GB deduplicated, after a dozen archives.
Full borg commandline that lead to the problem (leave away excludes and passwords)
Describe the problem you're observing.
I am seeing the same error as #3737 after doing some crazy hacks on a repository, but it's probably not due to a memory error and more a user error. Since others might have that trouble (and it might be something we want to avoid in the future), I figured I would report it as a bug, but maybe it could be resolved with a "hell no don't do that" FAQ... (for example here). Here's the story:
I had a repository (
So I figured I would try to split the repository in two. The plan was simple: copy the repository to
That archive was the oldest curie backup, if my memory is exact. Also note that I used hardlinks to avoid using too much space, a technique I took from the originally successful approach taken by
I lost the backtrace here, but once the
Also note that I tried to create a new archive while the delete was happening, but it led to a locking issue (which is fine). I also don't have the backtrace on that, unfortunately.
Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
That's pretty close to what happened, except that in the above the
Include any warning/errors/backtraces from the system logs
After the crash, I have moved the repositories around and tried to fix my mistake by copying the repository in multiple copies I could work on:
Now I have a copy of a copy of the repository in
What's "funny" is that even the
I'm currently running a
referenced this issue
Jan 28, 2019
For what it's worth, here's the result of the repair on the other repository:
... looks like the repair worked fine, which is interesting. i'm running a backup now to see if things work correctly there as well. presumably those
borg-angela: looks like you have 2 zero-length segment files and lost the data that should be in there.
i can not explain why these are zero-length, but as you are not sure that the repo was ok before you did your misc. operations on it, I'ld suggest you try to reproduce the problem starting from a known-valid repo. If you can reproduce, open a new ticket.
This is a "short read". borg tried to read 41 bytes from that offset (length of a PUT header), but obviously it got less than that.
Rather fundamental problems one runs into when copying repos and working on multiple of the copies independently:
Even if one fixes above by tweaking the repoid, one still has:
I guess the unexpected crash when reading the manifest for the repo for borg list comes exactly from that: it used the cached chunks index of the other repo and tried to read data from a location that wasn't valid for this repo.
So the root cause here was not changing the repo id to something unique.
But even when doing that, there is still the counter reuse security issue.
So maybe one just should not do that.
I see. That makes sense! Those repositories are in cleartext, so counter reuse is not a problem. If I understand this right, the repo id is generated with something like:
from binascii import hexlify import os hexlify(os.urandom(32)).decode('ascii')
... is that right?
I have just rerun borg check on the
I'll try to rename the repository and run a check again on both sides.
i tried renaming the
notice the failed mmap on an empty file...
does that mean the files are still missing from the archive? and why does it say "no problems found" even if there are zero'd out files?
sorry to be throwing all of that at you like that, I hope that's useful!