Nil blob list errors, zeroed files in backup

[zcalusic]

restic 0.2.0 (v0.2.0-202-gfae65eb)
compiled at 2016-09-04 10:30:20 with go1.7

When doing live backups of any system, some files will grow/change during the backup operation. Typical example are various log files that grow all the time. I see that restic already issues warnings for this case, which is perfectly fine. The operator should be warned of files whose backup copy is possibly incomplete or inconsistent.

error for /var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal: file has changed

But, when checking these incomplete files, they're actually zeroed, which is certainly unexpected. I'd expect to have at least some content there:

ls -al */var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal
-rw-r----- 1 root systemd-journal         0 Aug 29 09:51 2016-08-29T09:44:46+02:00/var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal
-rw-r----- 1 root systemd-journal         0 Aug 30 04:08 2016-08-30T04:00:23+02:00/var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal
-rw-r----- 1 root systemd-journal  50331648 Aug 31 04:08 2016-08-31T04:00:20+02:00/var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal
-rw-r----- 1 root systemd-journal         0 Sep  1 04:06 2016-09-01T04:00:20+02:00/var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal
-rw-r----- 1 root systemd-journal         0 Sep  2 04:07 2016-09-02T04:00:20+02:00/var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal
-rw-r----- 1 root systemd-journal  58720256 Sep  3 04:11 2016-09-03T04:00:22+02:00/var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal
-rw-r----- 1 root systemd-journal 100663296 Sep  4 04:07 2016-09-04T04:00:21+02:00/var/log/journal/0c458b75baf807396c69f0dd530cc822/system.journal

To add to this, checking the repository spits out errors that can't be fixed even with rebuild-index:

error for tree b68877d8:
  tree b68877d8: file "system.journal" has nil blob list
error for tree e61601be:
  tree e61601be: file "system.journal" has nil blob list
error for tree cd9904e3:
  tree cd9904e3: file "system.journal" has nil blob list
error for tree 3c0ba72c:
  tree 3c0ba72c: file "system.journal" has nil blob list

On my system these errors happen in most cases to the above mentioned system.journal, pacct (accounting) file and rrd files. And I have "nil blob list" errors for all of them.

[fd0]

Thanks for the report, that clearly shouldn't happen and instead be handled the way you described it. I'll have a look.

[zcalusic]

This bug is actually 2 different, but similar, bugs. Basically, this is a showstopper for backing up live systems. Because, not only some important files don't get backed up at all, but also the repo is left with fatal errors after the backup run. Double trouble.

I'll give it a try, though I'm now in uncharted waters, having trouble following some code there. But, I'll try to send 2 patches that should fix this.

[zcalusic]

I fixed the other bug easily, but this one is really nasty. Or, I'm missing something obvious, dunno.

I have this in debug output:

SaveFile("/var/log/journal/76a8b17c046149e090452b10b92c7108/system.journal"): 129 blobs

Yet, after the backup is done, NO blobs are saved, the file has zero length. :(

I'll keep trying, but atm, I'm out of ideas.

[thanosme]

Hello, I have the same problem but how can I delete the errored snapshots or blobs?

[fd0]

What do you see? The message about a nil blob list?

[thanosme]

Yes.

restic -r /repo check

error for tree da3cf756:
  tree da3cf756: file "groupmgr.db" has nil blob list
Fatal: repository contains errors

[fd0]

Ok, thanks for the report. There's no need for concern, and I'll probably downgrade this to a "hint" only. You should be able to restore your backups and use the fuse mount regardless of this message.

[thanosme]

Thats nice, but am I able to someway identify the file that causes the error in the repository and remove it?

[thanosme]

Well obviously I now which file is since I showed you above but I think you know what I mean.

[fd0]

The only way to remove the file is to delete the snapshot it is in, sorry.