deduplication does not work after migration from attic

[ThomasWaldmann]

From the mailing list:

I was able to reproduce the problem on a clean repo and small amount of
static data:

attic init repo
attic create --stats repo::backup1 data
borg upgrade repo
rm -rf ~/.cache/borg
borg create -v --stats repo::backup2 data (deduplication doesn't work)
rm -rf ~/.cache/borg
borg create -v --stats repo::backup3 data (deduplication does work)

Also I tried a number of actions with repository to recreate cache
(borg prune for example or borg create -n), but it didn't help.

[enkore]

Chunker params changed.

Quite related: Work on borg rewrite is underway, and that can already change chunker params. I.e. re-chunk old archives without extracting them[1], so that a migrated Attic repo will dedup with newly created Borg archives.

[1] What I basically do is that I have a file-like class around that sources read()s from the chunk-id-list of the source archive; I feed that back into chunkify, which then produces chunks with the current chunker params. When the rewrite is complete the old archive is deleted and so are all old chunks.

[ThomasWaldmann]

Ah, right, the change of default chunker params at borg 1.0.0 explains that.

So, the immediate workaround if the change of chunker params is unwanted is to specify the old chunker params manually, like:

borg create --chunker-params 10,23,16,4095 repo::archive data

But please note: the change to bigger default chunks at borg 1.0.0 was for a reason (lower resource usage [chunk index size, RAM, disk usage]), so think about what you really want.