Restore zero'd chunks (`borg check --repair`) by the next `borg create`

[PhrozenByte]

/kind enhancement

Borg currently repairs broken repos (e.g. due to failing hardware) by zero'ing missing or otherwise damaged chunks (borg check --repair). This is fine because it makes an otherwise broken repo usable again at the cost of loosing some data. The data is irrecoverably lost. In practice it means that some files, possibly across a large number of archives, are either completely, or partially zero'd. Borg generally handles this really great by "keeping" the file in the archive, but reporting that some parts of this file (to be more precise: one of the file's chunks) were zero'd when accessing such file (via borg extract or borg mount).

I'd like to draw on the "irrecoverably lost" part of this procedure: some data might indeed be recoverable - not by some obscure data recovery process, but by using the original data. Whenever possible Borg should keep information about the zero'd chunks to check whether a subsequent borg create might allow recreating the lost chunk - and therefore, practically speaking, restoring this previously lost file in older archives, too.

I don't really know much about Borg's data structure and the chunker, so I can't really assess how this might be possible. It might not be possible for any lost data. It might require some imperfections when matching data (like how Borg chooses not to read the contents of a file because it assumes - based on the file's metadata - that the file wasn't changed). It might be so CPU-intensive or error-prone that this "recovery by original data" needs to be requested manually by some CLI option. It might be a lot of work to implement it and thus not worth the effort. But I'm rather confident (like 90% 😆) that it should be possible - and allowing something like this would make Borg even more amazing.

Here are some steps to better illustrate what I mean:

1. Setup test environment

   $ cd some_dir
   $ export BORG_BASE_DIR="$PWD"
   $ borg init -e none repo
   $ mkdir data

2. Create some junk data and create a backup

   $ for i in {1..100}; do s=$(shuf -i 0-10485760 -n 1); dd if=/dev/urandom of=$(mktemp -u -p data XXXXXXXXXX) bs=$s count=1 status=none; done
   $ borg create repo/::test1 data/

3. Delete some repo data (let's choose the largest file), check repo (it will report errors) and repair it (remember one of the affected files)

   $ rm $(find repo/data/ -type f -printf '%s\t%p\n' | sort -n | tail -n1 | cut -f2)
   $ borg check repo
   Index object count mismatch.
   committed index: 307 objects
   rebuilt index:   14 objects
   ID: cc7781ae02aaa25eb2d18b31b40e15376d1f08dfff90eb7f46d4a83dfe7c3e5c rebuilt index: <not found>      committed index: (2, 80919919)   
   …
   $ borg check repo --repair
   borg check repo --repair
   This is a potentially dangerous function.
   check --repair might lead to data loss (for kinds of corruption it is not
   capable of dealing with). BE VERY CAREFUL!
   
   Type 'YES' if you understand this and want to continue: YES
   test1: data/LXFJGLqmgW: New missing file chunk detected (Byte 0-1414617). Replacing with all-zero chunk.
   …

4. Create another backup of the same original data (!)

   $ borg create repo/::test2 data/

5. Try to extract one of the affected files

   $ borg extract repo/::test1 data/LXFJGLqmgW
   data/LXFJGLqmgW: File has damaged (all-zero) chunks. Try running borg check --repair.
   $ borg extract repo/::test2 data/LXFJGLqmgW

As you can see Borg fails to restore the affected file (data/LXFJGLqmgW in this example) from the pre-data-loss test1 archive, but happily restores the exact same file from the post-data-loss test2 archive. Right now this is expected behaviour, but I suggest trying more to detect that both archives contain the same data and that - theoretically - the file can be restored from the test1 archive, too.

By the way: Why is borg extract mentioning borg check --repair? Isn't borg check --repair the only way for a chunk to get zero'd, so that having an all-zero chunk means that borg check --repair was used already?

[ThomasWaldmann]

Borg already can heal lost chunks, all you need is to run borg check --repair again after the borg create that added back the missing chunks.

[PhrozenByte]

Awesome! ❤️ I've just tested this and it indeed works. Borg amazes me again and again. Thanks Thomas! 👍

I just checked the man page again and now that I know it, I even found this information in the docs:

If a previously lost chunk reappears (e.g. via a later backup), in repair mode the all-zero replacement chunk will be replaced by the correct chunk. This requires reading of archive and file metadata, but not data.

However, I've read the docs before and missed it resp. didn't deduce to do something. Therefore I suggest to emphasize this resp. the recovery process in general better in the docs. What do you think is the best place for this? The FAQs? Or the borg check docs?

[bitclick]

Therefore I suggest to emphasize this resp. the recovery process in general better in the docs. What do you think is the best place for this? The FAQs? Or the borg check docs?

I second this! I am affected by a

Remote: Data integrity error: Segment entry checksum mismatch

error on one segement file and from the docs / faq i was not able to learn what this means for my archives. I wondered "What exactly is check --repair doing?". I eventually found a description from @ThomasWaldmann in a mailing-list-thread and now i am trying this out.

After reading the doc again, i can see it is there. I understand that for people doing this all the time it is no big deal but for many people a borg check error will lead to a spike in blood pressure and to keep this as short as possible there should be a separate section "repair procedure", which shows the steps to repair the repo and add the missing chunks again via create, check --repair, and verify check --verify-data (or is this the latter not neccessary?)

possibly add some hints to the borg check warn-output in the error-case.

[PhrozenByte]

@ThomasWaldmann added the documentation label Jul 5, 2023

I took this as a "sure, why not, go ahead and open a PR" and therefore did exactly this: See #7714