Corrupted blob

[madroach]

Output of restic version

% restic version
restic 0.16.3 compiled with go1.21.6 on openbsd/amd64

What backend/service did you use to store the repository?

I reproduced this three times on a repository on a directly connected external hard drive and twice on a repository accessed via the rest interface. Always on fresh repositories.

Problem description / Steps to reproduce

% set |grep RESTIC
RESTIC_COMPRESSION=max
RESTIC_PASSWORD_FILE=/home/madroach/.restic/restic-pass
RESTIC_PROGRESS_FPS=2
RESTIC_REPOSITORY=/mnt/test1
% rm -rf /mnt/test1/*
zsh: sure you want to delete all 6 files in /mnt/test1 [yn]? y
% restic init && restic backup --iexclude-file ~/.restic/excludes --exclude-caches /home && restic check --read-data
created restic repository b00a1b13ab at /mnt/test1

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
repository b00a1b13 opened (version 2, compression level max)
created new cache in /home/madroach/.cache/restic
no parent snapshot found, will read all files


Files:       461179 new,     0 changed,     0 unmodified
Dirs:        111648 new,     0 changed,     0 unmodified
Added to the repository: 42.199 GiB (25.732 GiB stored)

processed 461179 files, 45.801 GiB in 15:57
snapshot 966f0404 saved
using temporary cache in /tmp/restic-check-cache-2221875346
repository b00a1b13 opened (version 2, compression level max)
created new cache in /tmp/restic-check-cache-2221875346
create exclusive lock for repository
load indexes
[0:01] 100.00%  3 / 3 index files loaded
check all packs
check snapshots, trees and blobs
[0:02] 100.00%  1 / 1 snapshots
read all data
pack d5efb8999a798c6a35b54ce242c47ae0f049b47fb14b489b65936e0fb411ca9d contains 1 errors: [blob dbcbc33f1df3ed6295f40527080cb035d12539b6b1c35d9f8be22388ba53529c: read blob <data/dbcbc33f> from d5efb899: wrong data returned, hash is 7b440dd5b70dd997e35b9094a4a3eca110f415d96bb8df4b0035111801981077]
[11:09] 100.00%  1582 / 1582 packs

The repository contains pack files with damaged blobs. These blobs must be removed to repair the repository. This can be done using the following commands:

RESTIC_FEATURES=repair-packs-v1 restic repair packs d5efb8999a798c6a35b54ce242c47ae0f049b47fb14b489b65936e0fb411ca9d
restic repair snapshots --forget

Corrupted blobs are either caused by hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

Do you have any idea what may have caused this?

I suspect a software bug since (on 5 tries) always exactly one damaged blob is found.

No messages from the system about bad disk sectors.

This is on an AMD Ryzen 5600 6 core/12 threads CPU, 16GB ram, 512GB nvme running OpenBSD 7.4.

To search for memory problems I ran 12 instances of memtester on a total of 3GB of memory 4 times, heating up the CPU up to 60⁰C. I ran memtest86+ thoroughly half a year ago on this same machine. No errors found.

Did restic help you today? Did it make you happy in any way?

I enjoyed reading the (good) documentation. I'm pretty happy to have found a well designed backup solution.

[madroach]

running the same test with the latest master branch (8b08b52). Results in a very similar error:

read all data
pack 95c70ce87e17760f7f0040630dede6e1bc12f82fbaec911a27138a06b41fdd6a contains 1 errors: [blob f9471ce54b4cc3cec3cd1128adec35a34b25a057f2ee0e7117db4b1e2d3ddeca: <nil>]
[10:40] 100.00%  1592 / 1592 packs
Fatal: repository contains errors

I'd like to try some older version of restic, to see whether this is a regression, but was not able to build any v0.9.6 or v0.9.0 or v0.8.3:

% make restic
go run build.go
Go version go1.21.6 detected, restic requires at least Go 1.7
package cmds/restic is not in std (/usr/local/go/src/cmds/restic)
build failed: exit status 1
exit status 1
*** Error 1 in /home/madroach/src/restic (Makefile:6 'restic')

I made it skip the go version check, but don't know whats wrong with package cmds/restic is not in std. Latest master branch builds fine, latest release not ?!?

[MichaelEischer]

Please try whether the compression levels "auto" or "off" for RESTIC_COMPRESSION also lead to corrupted blobs.

Also please run restic find --blob dbcbc33f1df3ed6295f40527080cb035d12539b6b1c35d9f8be22388ba53529c to determine the file(s) affected by the damaged blobs. If changing the compression setting helps, are the affected files one that you could share?

[madroach]

With compression level off and auto, the backup and check went well without corruption.

restic find told me this:

Found blob dbcbc33f1df3ed6295f40527080cb035d12539b6b1c35d9f8be22388ba53529c
 ... in file /home/madroach/sonstige/Verwaltung/HiWi/HiWi08.pdf

Now the corruption can be triggered much faster:

% mkdir test && restic -r test init && restic -r test backup HiWi08.pdf && restic -r test check --read-data
created restic repository ce6ae60d25 at test

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
repository ce6ae60d opened (version 2, compression level max)
created new cache in /home/madroach/.cache/restic
no parent snapshot found, will read all files
[0:00]          0 index files loaded

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repository: 16.365 MiB (452.771 KiB stored)

processed 1 files, 16.364 MiB in 0:00
snapshot 3f55d40f saved
using temporary cache in /tmp/restic-check-cache-69621831
repository ce6ae60d opened (version 2, compression level max)
created new cache in /tmp/restic-check-cache-69621831
create exclusive lock for repository
load indexes
[0:00] 100.00%  1 / 1 index files loaded
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  1 / 1 snapshots
read all data
pack cbfb26ec9e5154bffd53b640830d002fe234aa078523d09eea565711dbc235e0 contains 1 errors: [blob 63eabeb7df57b492ef50560f8090201e07378aa32bcea692bf3238d35f0f62c1: <nil>]
[0:00] 100.00%  2 / 2 packs
Fatal: repository contains errors

[MichaelEischer]

As noticed in https://forum.restic.net/t/check-reports-errors-wrong-data-returned/7201/5 , this data corruption bug only affects restic 0.16.3.

[MichaelEischer]

Fixed by #4529 and #4681. The fixes are included in the just released restic 0.16.4!