How to rebuild index from data files when some data files were accidentally deleted?

[wzyboy]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

Client: borg 1.1.17 (PyInstaller release from GitHub; I know it's old...)

Server: borg 1.2.2 (installed from distro repo)

Operating system (distribution) and version.

Client: Ubuntu 20.04 LTS

Server: Arch Linux

Hardware / network configuration, and filesystems used.

LAN connection between client (laptop) and server (home server). Both filesystems are ext4.

How much data is handled by borg?

The repo is 38 GiB in size. Last archive is 27.42 GB uncompressed and 13.48 GB compressed. The client backs up to the repo daily between 00:00 and 02:00 (random).

Full borg commandline that lead to the problem (leave away excludes and passwords)

The normal daily backup command is:

borg create --verbose --stats --one-file-system --compression lz4 --exclude-caches --exclude-if-present .nobackup --keep-exclude-tags \
  --exclude "/home/*/.cache" \
  --exclude "/home/*/.local/share/Trash" \
  --exclude "/swapfile" \
  "::{hostname}-{now}" \
  "/home" "/root" "/etc" "/var/log" "/var/mail"

However, I foolishly ran find -size 17c -delete on the server to delete all 17-byte small files a few days ago. I discovered the issue today when monitoring shows recent backups failed.

Describe the problem you're observing.

Borg reported Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID) error when doing daily backup. I realized that my action of deleting all the 17c files a few days ago was wrong.

I deleted the ~/.config/borg/security/xxx and ~/.cache/borg/xxx directories and re-run the backup command. Borg reported:

Creating archive at "ssh://borg@[...]/media/borg/[...].borg::[...]-2023-06-24T00:10:44"
Synchronizing chunks cache...
Archives: 38, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 38.
Fetching and building archive index for [...]-2022-01-31T00:10:51 ...
Object with key fa16e06e647429cd1a9560d95994ca4f3f004461a988bc7b6d60d2733e0fdc07 not found in repository ssh://borg@[...]/media/borg/[...].borg::[...]-2023-06-24T00:10:44.
Traceback (most recent call last):
  File "borg/archiver.py", line 4703, in main
  File "borg/archiver.py", line 4635, in run
  File "borg/archiver.py", line 177, in wrapper
  File "borg/archiver.py", line 591, in do_create
  File "borg/cache.py", line 387, in __new__
  File "borg/cache.py", line 381, in local
  File "borg/cache.py", line 474, in __init__
  File "borg/cache.py", line 870, in sync
  File "borg/cache.py", line 824, in create_master_idx
  File "borg/cache.py", line 723, in fetch_and_build_idx
  File "borg/remote.py", line 1080, in get
  File "borg/remote.py", line 1184, in get_many
  File "borg/remote.py", line 937, in get_many
  File "borg/remote.py", line 780, in call_many
  File "borg/remote.py", line 757, in handle_error
borg.repository.Repository.ObjectNotFound: Object with key fa16e06e647429cd1a9560d95994ca4f3f004461a988bc7b6d60d2733e0fdc07 not found in repository ssh://borg@[...]/media/borg/[...].borg::[...]-2023-06-24T00:10:44.

The data file with the largest number is 4976 but the largest hints/index/integrity number is 4964, smaller than the data number. Also the hints/index/integrity files have a mtime of the first failed backup job. Log shows the command took more than 2 minutes to fail so it might be generating the hints file (usual backup + prune took less than 2 minutes):

# last good backup
Jun 20 00:07:08 [...] systemd[1]: Starting borg.sh...
Jun 20 00:07:10 [...] borg.sh[1323635]: Creating archive at "ssh://borg@[...]/media/borg/[...].borg::[...]-2023-06-20T00:07:09"
[...]
Jun 20 00:08:52 [...] systemd[1]: borg.sh.service: Succeeded.
Jun 20 00:08:52 [...] systemd[1]: Finished borg.sh.


# first failed backup
Jun 21 01:15:01 [...] systemd[1]: Starting borg.sh...
Jun 21 01:17:17 [...] borg.sh[1624879]: Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
Jun 21 01:17:17 [...] systemd[1]: borg.sh.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 21 01:17:17 [...] systemd[1]: borg.sh.service: Failed with result 'exit-code'.
Jun 21 01:17:17 [...] systemd[1]: Failed to start borg.sh.


# mtime of 4964 hints is the same as the failed backup
-rw-r-----+ 2 borg borg 1.4K Jun 21 01:17 hints.4964
-rw-r-----+ 2 borg borg  50M Jun 21 01:17 index.4964
-rw-r-----+ 2 borg borg  190 Jun 21 01:17 integrity.4964

Running borg list only shows archives from half a year ago - all archives created in the last few months are gone.

I'm surprised that Borg generated hints/index/integrity files with a smaller number. I thought the number can only increase.

By some wild coincidence I had a backup of hints/index/integrity 4977 files from the last good backup (but sadly not the data 4977 file):

-rw-r-----+ 1 borg borg 4.1M Jun 20 00:08 hints.4977
-rw-r-----+ 1 borg borg  50M Jun 20 00:08 index.4977
-rw-r-----+ 1 borg borg  190 Jun 20 00:08 integrity.4977

I put {hints,index,integrity}.4977 in the repo and did a borg list again. Same results - only archives from half a year ago.

With borg list the hints/index/integrity 4964 files were re-generated.

I deleted the hints/index/integrity files and did borg check --repair --repository-only:

This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: YES
Remote: Starting repository check
Remote: finished segment check at segment 4964
Remote: Starting repository index check
Remote: Finished full repository check, no problems found.

The hints/index/integrity files were re-generated again, number still being 4964. Doing borg list still shows archives from half a year ago.

The data files with number greater than 4964 do exist:

-rw-r-----+ 2 borg borg 337M Jun 16 00:30 4920
-rw-r-----+ 2 borg borg 240M Jun 17 01:47 4932
-rw-r-----+ 2 borg borg 291M Jun 18 01:28 4943
-rw-r-----+ 2 borg borg 314M Jun 18 01:28 4953
-rw-r-----+ 2 borg borg 209M Jun 19 01:02 4955
-rw-r-----+ 2 borg borg 501M Jun 19 01:02 4964
-rw-r-----+ 2 borg borg  71M Jun 19 01:02 4965
-rw-r-----+ 2 borg borg 338M Jun 20 00:08 4967
-rw-r-----+ 2 borg borg 3.3K Jun 20 00:08 4974
-rw-r-----+ 2 borg borg 188M Jun 20 00:08 4976 <- the last file I have

I assume that the objects of archives from the last few months are still there in these files, but the "index" are missing so borg list cannot find them. Is there a way to "scan" the data files and reconstruct the index(?) from the data files?

This is actually the second repo that I ran into problems. I tried to fix another (larger) repo with the same issue. I tried borg check --repair without --repository-only and it did a full repair of all the archives. It took a long time yet it zero'ed all the missing objects instead of discovering them from the data files. I gave up on that repo and accepted the fact that my foolish action made myself lose a few months of history.

With this repo, I'm trying to be more cautious this time by asking the question here to see if there is another method to try to save the history.

[ThomasWaldmann]

The 17b files you deleted are COMMITS. Uncommitted data is considered crap and usually the repo is rolled back to the previous commit.

You could try:

• adding a COMMIT 17b segment file as last file in the repo
• borg delete --cache-only REPO # get rid of client-side cache
• remove index.NNNN, hints.NNNN and other files with .NNNN extension from repo dir
• borg check --repair -v --progress REPO

If it is important, make a backup of what you have first.

[wzyboy]

Thanks. That looks promising. How do I add a 17-byte segment file?

[ThomasWaldmann]

Just pick one from another repo and add it with the highest file number.

[wzyboy]

Wow I did not know that all 17-byte files are identical.

I copied one from another repo and did what you suggested. It worked! All the archives are back! Thanks!

Command output for the record:

# time borg check --repair -vp
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: YES
Remote: Starting repository check
Remote: finished segment check at segment 4977
Remote: Starting repository index check
Remote: Finished full repository check, no problems found.
Starting archive consistency check...
Analyzing archive [...]-2022-05-31T01:07:11 (1/42)
Analyzing archive [...]-2022-06-29T01:40:27 (2/42)
Analyzing archive [...]-2022-07-31T00:58:44 (3/42)
[...]
Analyzing archive [...]-2023-06-18T01:26:44 (40/42)
Analyzing archive [...]-2023-06-19T01:00:51 (41/42)
Analyzing archive [...]-2023-06-20T00:07:09 (42/42)
Writing Manifest.
Committing repo (may take a while, due to compact_segments)...
Remote: compaction freed about 3.35 kB repository space.
Finished committing repo.
Archive consistency check complete, no problems found.

real    11m10.383s
user    6m42.861s
sys     0m3.919s

# `borg list` shows all the archives are back

# the 17-bytes `4977` file was deleted in the process and
# two new data files were generated on the server

-rw-r-----+ 2 borg borg 501M Jun 19 01:02 4964
-rw-r-----+ 2 borg borg  71M Jun 19 01:02 4965
-rw-r-----+ 2 borg borg 338M Jun 20 00:08 4967
-rw-r-----+ 2 borg borg 188M Jun 20 00:08 4976
-rw-r-----+ 1 borg borg 3.3K Jun 24 13:32 4979 <- new
-rw-r-----+ 1 borg borg   17 Jun 24 13:32 4981 <- new

[wzyboy]

I did an experiment (with a backup first) on a third repo that was affected by my stupid action of deleting all 17-byte files.

The highest index number of this repo is N and the highest data number is N-1 so I assume data N was a 17-byte file and was deleted.

1. Copy a 17-byte file from another repo and rename it as N to match the highest index number
2. borg list shows a full list of archives
3. borg create works normally

No brog check needed in this case. 🎉

[ThomasWaldmann]

Clearing all indexes and caches and running borg check was also to get consistence back and have some confidence that everything actually is consistent.