Promote `check --repair --undelete-archives` to a separate and safe `undelete` command

[PhrozenByte]

From an user's perspective (can't say anything code-wise) it would be great to promote the potentially lossy borg check --repair --undelete-archives of Borg 2 to a separate and safe borg undelete command. The reason is that the split of borg delete (resp. borg prune) and borg compact kinda hints for a safe way to undelete an archive before compaction, but borg check --repair is associated with warnings for good reason and could do additional harm.

I'm indecisive about the default action of borg undelete: Either to undelete all archives by default, or to rather just list the archives Borg can undelete - and if not the latter, how to do that otherwise. If not the default action, undeleting all archives could require an --all option. If not the default action, listing the archives Borg can undelete could either require the --dry-run --list options, or something like borg list --consider-deleted (similar to the old borg list --consider-checkpoints we no longer need, i.e. also listing deleted archives with borg list) or borg list --deleted (i.e. then listing deleted archives only). In any case borg undelete should also accept the usual options to match archives, especially including the [NAME] argument and -a/--match-archives options.

Related question: Does borg check --repair --undelete-archives undelete checkpoints? Is there even a difference between a checkpoint and a deleted/pruned archive before compaction? If there's no difference, this could imply the need for such a distinction with borg undelete.

This is also helpful for, but not limited to attack scenarios with --append-only.

Related question: Borg 2.0.0b10+ no longer creates a transactions file in --append-only, because users are expected to use borg check --repair --undelete-archives instead now, correct? If true, the docs are outdated in this regard: https://borgbackup.readthedocs.io/en/master/usage/notes.html#append-only-mode-forbid-compaction

From 682aedb I made the (possibly wrong, so please correct me otherwise) assumption that borg check --repair --undelete-archives might find more archives to undelete if the repo also happens to be corrupted. This should be documented in borg undelete, so that a user might want to run borg check (and possibly borg check --repair if any corruption is found; borg check --verify-data isn't really indicated, right?) before borg undelete.

[ThomasWaldmann]

Good idea, got me thinking...

• borg delete/prune could, rather than killing the entry in archives/, move it to archives-deleted/, so that we don't lose the knowledge about what objects are ("deleted") archive metadata objects.
• or even use the "soft deletion" feature of borgstore.
• borg undelete could then very easily list whatever is a candidate for undeletion. the current implementation in borg check is slow (goes over all objects in the repository) and rather a side-effect if it finds an archive object that has no entry in archives/ directory (we can then either kill the object or make an entry pointing to it).
• borg compact would empty archives-deleted/ (because anything not referenced from archives/ will be gone anyway)

[ThomasWaldmann]

To your questions:

• there are no checkpoint archives anymore (this was a concept needed by borg 1.x due to the way it works, but is not needed anymore because borg2 works very differently, giving the same or even better benefits)
• currently, there is also no implementation of append-only - this only works in a safe-against-attacks way if there is a separate server-side borg process (which is not the case for file:, sftp:, rclone: repos)
• there are no transactions anymore. it is just "first write objects, then write references to these objects".
• borg check undeletion: it would find all archives that still have an archive metadata object in the repo (even if there is no pointer to it in archives/)

[PhrozenByte]

#8500 (comment) sounds excellent 👍

Considering this, I assume that if it's indeed implemented this way, that borg check --repair --undelete-archives could still be useful in corruption scenarios (in case of lost or corrupted archive metadata to be more precise)? We could then have borg undelete as safe "undo command" for borg delete and borg prune, and borg check --repair --undelete-archives for corruption scenarios. Thinking about this, it might then be a good idea to limit borg check --repair --undelete-archives to such corruption scenarios, i.e. explicitly excluding the archives borg undelete would undelete from borg check --repair --undelete-archives (which should then be documented of course). This could then even allow for dropping --undelete-archives and rather make it the default with --repair.

there are no checkpoint archives anymore (this was a concept needed by borg 1.x due to the way it works, but is not needed anymore because borg2 works very differently, giving the same or even better benefits)

Even though borg2 isn't creating checkpoint archives, borg create still is writing data continuously and if it's aborted for whatever reason, the data written remains in the repo, right? It stays there "unreferenced" until it's either picked up by a following borg create, or deleted by borg compact, correct?

That's why I was asking myself whether it might be picked up by borg check --repair --undelete-archives. From your explanation I now assume it won't. Is there any way to pick it up? This could justify to keep --undelete-archives as separate option even beyond the mentioned consolidation of options above. Like "try to safe whatever possible, even if it's just a fraction of the original".

currently, there is also no implementation of append-only - this only works in a safe-against-attacks way if there is a separate server-side borg process (which is not the case for file:, sftp:, rclone: repos)

Oh, okay, didn't know that. But borg serve --append-only is still working and safe, right?

To be honest, borg init --append-only (resp. borg repo-create now) always kinda confused me, because as you said, it never was possible to implement this in a safe-against-attacks way without borg serve, because an attacker could always just modify the data on the filesystem... I thus feel like loosing it for anything except borg serve is no big loss.

Anyways, I mostly noted this in regards to the docs still mentioning transactions. Shall I open a PR to update the docs accordingly? Are there plans to re-add borg repo-create --append-only, or shall I remove it from the docs when I'm at it?

[ThomasWaldmann]

Even though borg2 isn't creating checkpoint archives, borg create still is writing data continuously and if it's aborted for whatever reason, the data written remains in the repo, right? It stays there "unreferenced" until it's either picked up by a following borg create, or deleted by borg compact, correct?

Exactly!

That's why I was asking myself whether it might be picked up by borg check --repair --undelete-archives. From your explanation I now assume it won't.

The main archive metadata object will be written AFTER the archive metadata stream. If it is there already (even if not pointed to by an entry in archives/), it could be found by borg check --repair --undelete-archives. If it is not there, we just have a lot of unreferenced objects (content data as well as archive metadata stream objects) that either a future borg create might reference or borg compact will discard.

But borg serve --append-only is still working and safe, right?

No. There are no transactions anymore and also no segment files that get appended.

But borg serve is at least an existing server-side agent that could be used as a starting point for new "append-only" and quota implementations (or in general: anything that needs to be enforced server-side).

OTOH, I am not too happy with borg serve and that RPC protocol, so not sure how that will be at the end.

[ThomasWaldmann]

About the docs: guess we should update that when we actually reimplemented that stuff. Or when releasing borg2, whatever comes first.

[PhrozenByte]

Got it, thanks! 👍

About the docs: If you've made a decision about how to go forward with --append-only, let me know, I'll happily update the docs accordingly then.

The main archive metadata object will be written AFTER the archive metadata stream. If it is there already (even if not pointed to by an entry in archives/), it could be found by borg check --repair --undelete-archives. If it is not there, we just have a lot of unreferenced objects (content data as well as archive metadata stream objects) that either a future borg create might reference or borg compact will discard.

Just as a scenario and to give possible inspiration: Is it possible to implement this in a way that any continuously written data can be picked up by borg check --repair --undelete-archives no matter what (e.g. by writing the archive metadata object earlier)? I'm thinking about practically getting borg1's "checkpoint archives" back, just different. If it's not possible, not reasonable, or would require noticeable effort, don't even consider this a suggestion, it just popped into my mind and would give rather minor benefits in very limited data recovery scenarios, thus hardly worth any trouble 😄

[ThomasWaldmann]

I'ld prefer to rather not have something like checkpoint archives:

• they are not needed for speeding up a subsequent borg create (that works towards finishing transfer / creating a valid and complete archive). this assumes that borg compact is not running in between, which should be no problem (just run it once a month/week/quarter or so and it usually won't interfere).
• they make cli and code more complex (one always needs to decide whether to consider them or not)
• they make borg check slower (as for a complete check, they also would need to get checked)
• if archive creation tends to be problematic due to long runtime and unstable connection, just running the backup more frequently usually reduces the amount of data that has to get transferred. so the result are more completed archives / more tries to get a completed one.

[ThomasWaldmann]

@PhrozenByte Working on this in PR #8515.

I first implemented borg list --deleted but then noticed that there should be borg undelete --dry-run --list anyway (and realized that users really should first use that anyway, before accidentally undeleting too much or the wrong stuff), so I am considering now whether to remove the --deleted option again from borg list.

[PhrozenByte]

#8515 looks great! ❤️

I'm not sure about what the best solution might be, too. On one hand you're absolutely right, to learn what archives can be undeleted --dry-run --list is sufficient. On the other hand, borg repo-list is more powerful due to --format and --json (which is especially useful for 3rd-party tools). Even though I don't like the --deleted option much (it kinda feels "hacky"), I'd consider it advantageous over just --dry-run --list.

An use case for borg repo-list --deleted that just popped into my mind is to predict on how much space borg compact will free: Unless something else went wrong, adding up all soft-deleted archives should give us a pretty good estimate, right?

[ThomasWaldmann]

Hmm, right, so I'll keep borg list --deleted.

How much space is freed by compact is hard to predict, adding up the "unique chunks sizes" would give the minimum amount, but it could be also more.

Also, quite some stats are reduced in borg2 because they can't be implemented easily due to how it works (and in general, they were a PITA and not always useful).

[ThomasWaldmann]

borg check --repair --undelete-archives can now work a bit differently also:

Usually we either have a normal archives/ directory entry or (for deleted archives) a soft-deleted directory entry.

That repair command will now only create new directory entries if it finds an archive metadata chunk and neither of these directory entries exist. That can only be the case if the entry has been "lost" somehow.

[PhrozenByte]

Hmm, right, so I'll keep borg list --deleted.

👍

How much space is freed by compact is hard to predict, adding up the "unique chunks sizes" would give the minimum amount, but it could be also more.

I see, thanks for the explanation

That repair command will now only create new directory entries if it finds an archive metadata chunk and neither of these directory entries exist. That can only be the case if the entry has been "lost" somehow.

Looks great 👍

Just checked the corresponding docs and considering that we now have soft-deleted archives I'd like to bring up the question whether borg check --repair --undelete-archives should bring the archives it (re-)discovers back as regular ("non-deleted") archives, or as soft-deleted archives. Since users can't choose what to recover and since these archives didn't appear in borg repo-list before and would have been wiped with borg compact, I believe that they should rather be marked as soft-deleted, allowing users to recover them with borg undelete in a second step if they actually want some of the archives back. This needs documentation of course.

By the way, borg check (without --repair and --undelete-archives) would finish with a non-zero exit code if it finds data that could be recovered with borg check --repair --undelete-archives, right? Because right now my scripts would happily run borg compact if borg check succeeds with exit code 0 🙈