different magics, accept/convert attic repos

[ThomasWaldmann]

Just wanted to write something about this change, that makes current borg code incompatible to attic repos (although currently the code likely would still work on them).

Magic strings of the repo files are BORG* now (not ATTIC*) and also the local cache path is .cache/borg/... now (not .cache/attic/...)

See there: 159315e

The reason for this is to avoid interference. At any time, there might be some little change in either attic or borg that makes them incompatible in one or both directions. If both would use the same magic and you would use borg to access attic repos or vice versa (maybe even accidentally), it could just blow up or even permanently damage the repo files. In the worst case, you could not access the repo with either tool any more.

---

💰 there is a bounty for this

[anarcat]

so a few things about this:

1. i agree it's important to avoid blowing up stuff
2. if we're going to change the magic number, i think they should be versioned to cover for possible future changes in the data structures
3. in that spirit, which change actually warranted such a change in the magic numbers? surely there's a change that makes the ondisk format incompatible anymore?

i think this is an important scenario we should deal with properly: we are changing the on-disk format. fine. but what does that entail? do we just do random changes and not look back? i believe we should be really careful when that happen and provide a migration path. migrating from attic to borg isn't much different from migrating between borg 1.x and 2.x, assuming 2.x introduces on-disk changes as well.

so my suggestion would be to not break the on-disk format without a very good reason (i guess that still needs to be defined, but it seems unclear to me right now what the reason for the above change is).

when the on-disk format is changed:

1. it should be clearly documented as a change in internals.html
2. ... but also in a "changelog-style" file (this is currently CHANGES, but this lacks details on how the changes impact older repositories
3. the magic number needs to be incremented
4. a migration layer of some sort needs to be implemented to convert older repository formats

does that make sense?

is it the right place to discuss such issues for the future?

[ThomasWaldmann]

@anarcat the magics are not for versioning, but to tell (detect) whether the files are from this software (borg). That's one of the reasons why they say BORG and not ATTIC. See above for others.

There's already other information in the on-disk files to version the API level. Which is btw. another reason not to tell ATTIC in the magics - who says that attic api v3 will be same as borg api v3?

[anarcat]

@ThomasWaldmann Right okay, that makes sense. Still doesn't invalidate my points i believe. ;)

[ThomasWaldmann]

As borg 0.x.x is now targetted to be attic + conservative changes, a one-time conversion from current(!) attic repos seems reasonable and doable. The converter has to be careful not to convert attic repos (from past or from future) that are incompatible.

The different BORG magics must stay as they are, for reasons outlined above.

[ThomasWaldmann]

Note: I just started a bounty on BountySource for this ticket. The bounty is about implementing the code needed for doing a one-time conversion from attic to borg repositories (so attic users have a migration path without losing their repository or having to begin from scratch).

[anarcat]

i may just do this at some point, but i'm in kind of a squeeze until at least october... so first person to grab this gets the bounty yay! :)

[anarcat]

no one started on this yet?

[ThomasWaldmann]

Not me at least (and I don't know of anybody else working on this).

[anarcat]

so in short, there are a few places that needs to be rewritten:

• key file: s/ATTIC KEY/BORG_KEY/ in get_keys_dir(), that is $ATTIC_KEYS_DIR or $HOME/.(attic|borg)/keys, that is loaded by KeyfileKey.find_key_file() - that finds the keys with the right identifier for the repo, no need to decrypt to convert
• repository segments: s/ATTICSEG/BORG_SEG/ - luckily the segment length didn't change so we can just replace the 8 first bytes of all regular files in $ATTIC_REPO/data/**, the Repository.segment_iterator() can be used here
• hash indexes: s/ATTICIDX/BORG_IDX/ - and this is in a few locations:
  • the files and chunks cache (in $HOME/.cache/attic/<repoid>/), which we could just drop, but if we'd want to convert, we could open it with the Cache.open(), edit in place and then Cache.close() to make sure we have locking right
  • the repository index (in $ATTIC_REPO/index.%d, where %d is the Repository.get_index_transaction_id()), which we should probably update, with a lock, see Repository.open(), which i'm not sure we should use because it may write data on Repository.close()...

i think that about covers it.

[anarcat]

unless the latter wasn't clear, i'm working on this here. :)

[anarcat]

let's move this fun stuff to the PR in #231.

[ThomasWaldmann]

PR #231 closes this. Thanks to @anarcat we have a migration path now! \o/

[anarcat]

you're welcome :)