borg extract: check behaviour on integrity (or other) errors

[ThomasWaldmann]

Assuming there is some issue in the repo storage, like partial corruption, unreadable disk sectors or so - how does borg handle them?

It might be the case that borg extract currently aborts when it hits such a file. More helpful would be to log an error for that file and try to continue. At the end, exit with warning status (1). This is similar to what borg does when creating archives and it encounters problems for single files.

[enkore]

I'm moving this to the 1.2 milestone, since this is not possible atm for IntegrityErrors on the Repository layer for remote repos. I.e. it would need an RPC change and would not work with old servers.

I'll leave the code there (feature/extract-logfail) so when 1.2 comes up we can use that as a basis.

How does borg extract do it now

Abort on first error

[ThomasWaldmann]

hmm, thinking about what's worse:
doing a rpc change in 1.1 (we already have some rpc changes) or being unable to (easily) extract an archive due to some defect repo chunks aborting it here and there.

[ThomasWaldmann]

Hmm, what do you mean by "is not possible atm"? The repository code already can raise IntegrityError at a lot of place and borg.remote re-raises them if that error type comes from the remote.

[ThomasWaldmann]

@textshell see above - how is that after your rpc changes?

[textshell]

hmm, not sure about this. The new RPC should not have changed much. But i don‘t see why this is not possible with the 1.0.x or 1.1.0 rpc protocol looking at it briefly.
One change that might affect this though is that IntegrityError now keeps its details string. But the code doesn‘t parse that anyway. (Maybe we should add more machine readable information there...)

@enkore
Do you remember what prevented the code from working with remote repositories?

[enkore]

I'm not a 100 % sure, but I think maybe it was because of the iterators breaking when they raise IntegrityError, and I didn't realize that this was intentional / that they would need to be restarted manually.

[ThomasWaldmann]

Just to be clear about what happens for a repo error:

# local repo
$ borg extract -v ../repo::arch2
Enter passphrase for key /home/tw/w/borg/repo: 
Data integrity error: Segment entry checksum mismatch [segment 4, offset 8]
Traceback (most recent call last):
  File "/home/tw/w/borg/src/borg/archiver.py", line 2762, in main
    exit_code = archiver.run(args)
  File "/home/tw/w/borg/src/borg/archiver.py", line 2699, in run
    return args.func(args)
  File "/home/tw/w/borg/src/borg/archiver.py", line 101, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/home/tw/w/borg/src/borg/archiver.py", line 112, in wrapper
    return method(self, args, repository=repository, manifest=manifest, key=key, archive=archive, **kwargs)
  File "/home/tw/w/borg/src/borg/archiver.py", line 533, in do_extract
    stripped_components=strip_components, original_path=orig_path, pi=pi)
  File "/home/tw/w/borg/src/borg/archive.py", line 499, in extract_item
    for _, data in self.pipeline.fetch_many(ids, is_preloaded=True):
  File "/home/tw/w/borg/src/borg/archive.py", line 180, in fetch_many
    for id_, data in zip(ids, self.repository.get_many(ids, is_preloaded=is_preloaded)):
  File "/home/tw/w/borg/src/borg/repository.py", line 839, in get_many
    yield self.get(id_)
  File "/home/tw/w/borg/src/borg/repository.py", line 833, in get
    return self.io.read(segment, offset, id)
  File "/home/tw/w/borg/src/borg/repository.py", line 1110, in read
    size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data)
  File "/home/tw/w/borg/src/borg/repository.py", line 1146, in _read
    segment, offset))
borg.helpers.IntegrityError: Segment entry checksum mismatch [segment 4, offset 8]

Platform: Linux tux 4.8.8-tw2 #1 SMP Thu Nov 17 01:35:09 CET 2016 x86_64 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.1.0b3.dev241+ng1250b14  Python: CPython 3.5.1+
PID: 28230  CWD: /home/tw/w/borg/ex
sys.argv: ['/home/tw/w/borg-env/bin/borg', 'extract', '-v', '../repo::arch2']
SSH_ORIGINAL_COMMAND: None

# remote repo
$ borg extract -v tw@localhost:repo::arch2
Enter passphrase for key ssh://tw@localhost/./repo: 
Data integrity error: Segment entry checksum mismatch [segment 4, offset 8]
Traceback (most recent call last):
  File "/home/tw/w/borg/src/borg/archiver.py", line 2762, in main
    exit_code = archiver.run(args)
  File "/home/tw/w/borg/src/borg/archiver.py", line 2699, in run
    return args.func(args)
  File "/home/tw/w/borg/src/borg/archiver.py", line 101, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/home/tw/w/borg/src/borg/archiver.py", line 112, in wrapper
    return method(self, args, repository=repository, manifest=manifest, key=key, archive=archive, **kwargs)
  File "/home/tw/w/borg/src/borg/archiver.py", line 533, in do_extract
    stripped_components=strip_components, original_path=orig_path, pi=pi)
  File "/home/tw/w/borg/src/borg/archive.py", line 499, in extract_item
    for _, data in self.pipeline.fetch_many(ids, is_preloaded=True):
  File "/home/tw/w/borg/src/borg/archive.py", line 180, in fetch_many
    for id_, data in zip(ids, self.repository.get_many(ids, is_preloaded=is_preloaded)):
  File "/home/tw/w/borg/src/borg/remote.py", line 766, in get_many
    for resp in self.call_many('get', [{'id': id} for id in ids], is_preloaded=is_preloaded):
  File "/home/tw/w/borg/src/borg/remote.py", line 644, in call_many
    handle_error(unpacked)
  File "/home/tw/w/borg/src/borg/remote.py", line 620, in handle_error
    raise IntegrityError(args[0].decode())
borg.helpers.IntegrityError: Segment entry checksum mismatch [segment 4, offset 8]

Platform: Linux tux 4.8.8-tw2 #1 SMP Thu Nov 17 01:35:09 CET 2016 x86_64 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.1.0b3.dev241+ng1250b14  Python: CPython 3.5.1+
PID: 28763  CWD: /home/tw/w/borg/ex
sys.argv: ['/home/tw/w/borg-env/bin/borg', 'extract', '-v', 'tw@localhost:repo::arch2']
SSH_ORIGINAL_COMMAND: None

[ThomasWaldmann]

As a general note: we have 2 ways to deal with that:

1. keep it like it is. corrupted repos will just blow up extract. the user will then use borg check --repair to fix such repos and retry the extract.

2. try to handle it in extract_item, like logging the repo error and skipping the current file. that would enable a partial extract more quickly for only slightly damaged repos.

[enkore]

2.) -> should probably be more like check --repair would do it, ie. replace the faulty chunks with runs of zeroes?

[textshell]

I don’t like the idea of just going on and printing a warning. That is far to easy to miss. But as on option (or interactive prompt?) i think this would be useful.
I’m not sure replacing with a run of zeros makes the life easier or harder when working with restore problems. I wonder if borg should write a file with a "log" of the data problems. Missing files are somewhat possible to detect without a log, but zeroed sections are almost impossible to detect.

[enkore]

The default in other places, borg-mount, is to fail by default as well.

[ThomasWaldmann]

@enkore not sure about whether we should try to fix-on-the-fly within extract and fuse and ...

This might duplicate / complicate code quite a bit and maybe would never be as good as borg check --repair.

[ThomasWaldmann]

Just put this into 2.0 milestone.

IF we need to change the rpc interface, we should do it with borg2.

I'll have a look at it now, not so sure any more we should change anything. Having only borg check (--repair) deal with corruption is likely easier (and maybe also better).