Backup: verification succeeds then fails on the same backup after reboot

[andrewdavidwong]

What I did and what happened:

1. Create a backup with qvm-backup.

2. Verify the backup via Qubes Manager (restore with "verify only" option).

3. Qubes Manager reports verification successful.

4. Upload a copy of the backup tarball to a cloud storage service.

5. A day later, download the backup tarball from the cloud storage service.

6. Use md5sum to check that the hash of the downloaded tarball matches the hash of the original, local backup tarball. They match.

7. Attempt to verify the downloaded backup tarball via Qubes Manager.

8. Verification fails with the following error message:

   ERROR: ERROR: invalid hmac for file /var/tmp/restore_[...]:
   [...]
   Is the passphrase correct?
   Partially restored files left in /var/tmp/restore_*, investigate them and/or clean them up
   

9. Attempt to verify the original backup tarball via Qubes Manager. It also fails with the same error message.

10. Reboot dom0.

11. Re-attempt verification again three times, with the same failure each time.

Given the sequence of events, the most likely explanation seems to be that the original backup was corrupt, and Qubes Manager (or qvm-backup-restore) erroneously reported that verification was successful. If so, this is very serious, since it means users have no reliable way to know whether their backups are corrupt.

Related issue: #1471

[andrewdavidwong]

This problem (and the one in #1471) seem to manifest only after Qubes has been running for a long time (several days) without a host reboot. In fact, I've noticed for a long time now (across several Qubes versions) that Qubes tends to become somewhat unstable after going too long without a host reboot, so perhaps this is one symptom of that (but one of the more serious ones).

[andrewdavidwong]

This problem (and the one in #1471) seem to manifest only after Qubes has been running for a long time (several days) without a host reboot. In fact, I've noticed for a long time now (across several Qubes versions) that Qubes tends to become somewhat unstable after going too long without a host reboot, so perhaps this is one symptom of that (but one of the more serious ones).

[andrewdavidwong]

User cubit also appears to be experiencing this bug:
https://groups.google.com/d/msgid/qubes-users/KnFmLnO--3-0%40tutanota.com

[andrewdavidwong]

User cubit also appears to be experiencing this bug:
https://groups.google.com/d/msgid/qubes-users/KnFmLnO--3-0%40tutanota.com