[BUG] verify action should catch regression error

[desseim]

Bug summary

An unexpected error is thrown when verifying a repository to which the last backup has failed.

Version, Python, Operating System

Using rdiff-backup version 2.0.5
with cpython C:\Program Files (x86)\rdiff-backup\rdiff-backup.exe version 3.7.7
on Windows-10-10.0.19041, fs encoding utf-8

rdiff-backup call

rdiff-backup.exe --verify D:\

What happened and what did you expect?

Here's the output:

rdiff-backup.exe : [10224] Failed to execute script rdiff-backup
At line:1 char:1
+ rdiff-backup.exe --verify D:\
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ([10224] Failed ...pt rdiff-backup:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Warning, two different times for current mirror found
Exception 'Compressed file ended before the end-of-stream marker was reached' raised of class '<class 'EOFError'>':
  File "rdiff_backup\Main.py", line 395, in error_check_Main
  File "rdiff_backup\Main.py", line 417, in Main
  File "rdiff_backup\Main.py", line 377, in take_action
  File "rdiff_backup\Main.py", line 1058, in Verify
  File "rdiff_backup\compare.py", line 84, in Verify
  File "rdiff_backup\metadata.py", line 331, in iterate
  File "rdiff_backup\metadata.py", line 344, in iterate_records
  File "rdiff_backup\metadata.py", line 322, in get_next_pos
  File "gzip.py", line 287, in read
  File "_compression.py", line 68, in readinto
  File "gzip.py", line 493, in read
Traceback (most recent call last):
  File "rdiff-backup", line 32, in <module>
  File "rdiff_backup\Main.py", line 395, in error_check_Main
  File "rdiff_backup\Main.py", line 417, in Main
  File "rdiff_backup\Main.py", line 377, in take_action
  File "rdiff_backup\Main.py", line 1058, in Verify
  File "rdiff_backup\compare.py", line 84, in Verify
  File "rdiff_backup\metadata.py", line 331, in iterate
  File "rdiff_backup\metadata.py", line 344, in iterate_records
  File "rdiff_backup\metadata.py", line 322, in get_next_pos
  File "gzip.py", line 287, in read
  File "_compression.py", line 68, in readinto
  File "gzip.py", line 493, in read
EOFError: Compressed file ended before the end-of-stream marker was reached

I expected the --verify instruction to complete successfully, and possibly return an error about the state of the backup repository needing regression, but not to throw an EOFError which seems like it isn't expected nor supposed to bubble up to the end user.

More information

Adding -v9 brings no additional information about the error, it only outputs a bunch of Verified SHA1 digest of <file> lines before the exception occurs.
The last backup to the repository had failed because it was a whole-system-drive backup which had been ran by mistake with unprivileged credentials (non-administrator user).
So basically:

$ rdiff-backup.exe --backup --exclude-symbolic-links --include-globbing-filelist path\to\file C:\ D:\  # run as admin, terminates successfully
> rdiff-backup.exe --backup --exclude-symbolic-links --include-globbing-filelist path\to\file C:\ D:\  # run as non-admin user, terminates mid-backup unexpectedly because of file permission issues

The verify command was run right after the failed backup (with administrator privileges though).

v2.1.0a1 throws a similar exception:

rdiff-backup.exe : Warning, two different times for current mirror found
At line:1 char:1
+ rdiff-backup.exe verify D:\
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Warning, two di...nt mirror found:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Traceback (most recent call last):
  File "rdiff-backup", line 32, in <module>
  File "rdiff_backup\Main.py", line 38, in main_run_and_exit
  File "rdiff_backup\Main.py", line 95, in _main_run
  File "rdiffbackup\actions\verify.py", line 97, in run
  File "rdiff_backup\compare.py", line 210, in Verify
  File "rdiff_backup\metadata.py", line 102, in iterate
  File "rdiff_backup\metadata.py", line 115, in _iterate_records
  File "rdiff_backup\metadata.py", line 177, in _get_next_pos
  File "gzip.py", line 300, in read
  File "_compression.py", line 68, in readinto
  File "gzip.py", line 506, in read
EOFError: Compressed file ended before the end-of-stream marker was reached
[5812] Failed to execute script rdiff-backup

The backups to the repository have been run with v2.0.5 though.

[ericzolf]

Good catch. It doesn't make sense to verify a "corrupt" repository.

[desseim]

Thanks for editing in the title, sorry I forgot.

A couple follow-up questions related to your answer:

1. Any other / preferred way to check the integrity of the repository? I know list increments reports an error too, but I thought verify would be the best suited for such a task.
2. Any idea of what got the repository so corrupted? As I explained I just did 2 backups, one successful as administrator then another failed one as a normal user. Is this a simple user error or a bug? I would have expected the backup to fail for not being able to access some files, but not to corrupt the repository.

Thank you!

[ericzolf]

1. you want to use the regress action resp. --check-destination-dir, verify only validates checksums
2. I fixed recently similar issues, so it might be gone, but you can provide another bug report with more information and I'll look into it.

[desseim]

1. I wanted to only check the status of the repository without performing any action. The use case being: to know whether a backup ended successfully, or unexpectedly mid-way. Since on Windows there are a lot of non-terminating errors (ListError for system files being "used" and such) even successful backups return a non-zero value and I can't rely on it for result diagnostic (similar to what I reported in [BUG] remove increments --older-than returns an error when there is no increment to remove in v2.1.0a1 #616).
2. I just wanted to know if that was expected and I was unknowingly using the program wrong, I guess not. I'll test again when the next release is published, thanks.

[ericzolf]

I consider it an enhancement but fair enough :-)