Error during garbage collection with subfolders in backup/ folder #50

juliadin · 2015-01-24T14:54:48Z

When storing backups in subfolders under the backup/ directory, garbage collection fails. Let me illustrate as follows. To me - without looking at the code - it looks like the subdirectories are read as backup files. I can reproduce with the latest git that I got to build (see other bug).
This makes deleting backups from the repository quite a lot harder because I can not clean up behind them. I'd appreciate it if someone could have a look at this. :) Thanks.

Some Version information:

  root@usagi:~# zbackup --help
  ZBackup, a versatile deduplicating backup tool, version 1.4
  Copyright (c) 2012-2014 Konstantin Isakov <ikm@zbackup.org> and
  ZBackup contributors
  Comes with no warranty. Licensed under GNU GPLv2 or later + OpenSSL.
  Visit the project's home page at http://zbackup.org/

  Usage: zbackup [flags] <command> [command args]
    Flags: --non-encrypted|--password-file <file>
      password flag should be specified twice if import/export
      command specified
     --silent (default is verbose)
     --threads <number> (default is 2 on your system)
     --cache-size <number> MB (default is 40)
     --exchange [backups|bundles|index] (can be
      specified multiple times)
     --compression <compression> <lzma|lzo> (default is lzma)
     --help|-h show this message
    Commands:
      init <storage path> - initializes new storage;
      backup <backup file name> - performs a backup from stdin;
      restore <backup file name> - restores a backup to stdout;
      export <source storage path> <destination storage path> -
        performs export from source to destination storage;
      import <source storage path> <destination storage path> -
        performs import from source to destination storage;
      gc <storage path> - performs chunk garbage collection.
    For export/import storage path must be valid (initialized) storage.

Create Repositories, encrypted and unencrypted:

  root@usagi:~# zbackup --password-file /root/.passphrase init /tmp/zbackup_test_crypt/
  root@usagi:~# zbackup --non-encrypted init /tmp/zbackup_test_plain/

Put some content in:

  root@usagi:~# dd if=/dev/urandom bs=4M count=10 | zbackup --password-file /root/.passphrase backup /tmp/zbackup_test_crypt/backups/test1
  Loading index...
  Index loaded.
  Using up to 2 thread(s) for compression
  10+0 records in
  10+0 records out
  41943040 bytes (42 MB) copied, 12.6271 s, 3.3 MB/s

  root@usagi:~# dd if=/dev/urandom bs=4M count=10 | zbackup --non-encrypted backup /tmp/zbackup_test_plain/backups/test1
  Loading index...
  Index loaded.
  Using up to 2 thread(s) for compression
  10+0 records in
  10+0 records out
  41943040 bytes (42 MB) copied, 11.9597 s, 3.5 MB/s

Test garbage collection with a backup in, working fine:

  root@usagi:~# zbackup --password-file /root/.passphrase gc /tmp/zbackup_test_crypt/
  Loading index...
  Loading index file 6603d1fa03891cc91b0a80e02f95c11f3db412361653d294...
  Index loaded.
  Using up to 40 MB of RAM as cache
  Using up to 2 thread(s) for compression
  Checking used chunks...
  Checking backup test1...
  Checking bundles...
  Loading index...
  Garbage collection complete

  root@usagi:~# zbackup --non-encrypted gc /tmp/zbackup_test_plain/
  Loading index...
  Loading index file 1fe0e6eeb365b5da3bfdf1c58d27cc3bab030297d2b47f2e...
  Index loaded.
  Using up to 40 MB of RAM as cache
  Using up to 2 thread(s) for compression
  Checking used chunks...
  Checking backup test1...
  Checking bundles...
  Loading index...
  Loading index file 1fe0e6eeb365b5da3bfdf1c58d27cc3bab030297d2b47f2e...
  Index loaded.
  Cleaning up...
  Garbage collection complete

Create folders inside the backup directory:

  root@usagi:~# mkdir /tmp/zbackup_test_plain/backups/test_folder
  root@usagi:~# mkdir /tmp/zbackup_test_crypt/backups/test_folder

Test garbage collection again, not working anymore. It fails while trying to read the folder as file.
The error messages differ for encrypted and unencrypted repositories:

  root@usagi:~# zbackup --password-file /root/.passphrase gc /tmp/zbackup_test_crypt/
  Loading index...
  Loading index file 999403125662263560008c14cf4f02ed77b818d7fd6eb1a4...
  Loading index file 6603d1fa03891cc91b0a80e02f95c11f3db412361653d294...
  Index loaded.
  Using up to 40 MB of RAM as cache
  Using up to 2 thread(s) for compression
  Checking used chunks...
  Checking backup test1...
  Checking backup test_folder...
  size of the encrypted file is incorrect

  root@usagi:~# zbackup --non-encrypted gc /tmp/zbackup_test_plain/
  Loading index...
  Loading index file dd5096abf90eb102dedabc97b52cf07c380448d7840131f4...
  Loading index file 1fe0e6eeb365b5da3bfdf1c58d27cc3bab030297d2b47f2e...
  Index loaded.
  Using up to 40 MB of RAM as cache
  Using up to 2 thread(s) for compression
  Checking used chunks...
  Script started on Sat 24 Jan 2015 03:19:43 PM CET
  Checking backup test1...
  Checking backup test_folder...
  Can't parse message FileHeader

Add a backup in the created folders:

  root@usagi:~# dd if=/dev/urandom bs=4M count=10 | zbackup --non-encrypted backup /tmp/zbackup_test_plain/backups/test_folder/test2
  Loading index...
  Loading index file dd5096abf90eb102dedabc97b52cf07c380448d7840131f4...
  Loading index file 1fe0e6eeb365b5da3bfdf1c58d27cc3bab030297d2b47f2e...
  Index loaded.
  Using up to 2 thread(s) for compression
  10+0 records in
  10+0 records out
  41943040 bytes (42 MB) copied, 12.3289 s, 3.4 MB/s

  root@usagi:~# dd if=/dev/urandom bs=4M count=10 | zbackup --password-file /root/.passphrase backup /tmp/zbackup_test_crypt/backups/test_folder/test2
  Loading index...
  Loading index file 999403125662263560008c14cf4f02ed77b818d7fd6eb1a4...
  Loading index file 6603d1fa03891cc91b0a80e02f95c11f3db412361653d294...
  Index loaded.
  Using up to 2 thread(s) for compression
  10+0 records in
  10+0 records out
  41943040 bytes (42 MB) copied, 13.425 s, 3.1 MB/s

Try again, but still the same error with the folders no longer empty:

  root@usagi:~# zbackup --password-file /root/.passphrase gc /tmp/zbackup_test_crypt/
  Loading index...
  Loading index file 2d0cc4325853e6b35d43d158d1a5bcece0812b205125bdb9...
  Loading index file 999403125662263560008c14cf4f02ed77b818d7fd6eb1a4...
  Loading index file 6603d1fa03891cc91b0a80e02f95c11f3db412361653d294...
  Index loaded.
  Using up to 40 MB of RAM as cache
  Using up to 2 thread(s) for compression
  Checking used chunks...
  Checking backup test1...
  Checking backup test_folder...
  size of the encrypted file is incorrect

  root@usagi:~# zbackup --non-encrypted gc /tmp/zbackup_test_plain/                                                                                                                                Loading index...
  Loading index file dd5096abf90eb102dedabc97b52cf07c380448d7840131f4...
  Loading index file c1504fca297127e0f2f6bd1ae478bbf32d1220e1b56682a9...
  Loading index file 1fe0e6eeb365b5da3bfdf1c58d27cc3bab030297d2b47f2e...
  Index loaded.
  Using up to 40 MB of RAM as cache
  Using up to 2 thread(s) for compression
  Checking used chunks...
  Checking backup test1...
  Checking backup test_folder...
  Can't parse message FileHeader

The text was updated successfully, but these errors were encountered:

Vlad1mir-D · 2015-01-24T15:04:00Z

Ok, I'll get it sorted out ASAP.

Vlad1mir-D · 2015-01-25T16:33:44Z

Fixed.

juliadin · 2015-01-25T21:17:22Z

Many many thanks, mate. Works like a charm.

mumblez · 2015-03-31T09:58:33Z

@am1go
I'm still having this issue:
centos 7 with epel repo
zbackup 1.4.1

steps:
after init and make backup to /backups (tar c )
create a sub directory under /backups
e.g. for daily, weekly, monthly

run gc on repo (just to see what will happen with the subfolder)
zbackup gc --non-encrypted

...
Can't parse message FileHeader

maybe the centos / epel version isn't patched?

mumblez · 2015-03-31T09:59:25Z

replace /backups repo/backups (got stripped out from comments)

Vlad1mir-D · 2015-04-03T09:27:43Z

maybe the centos / epel version isn't patched?

Correct, 1.4.1 from EPEL still contains this bug.
This will be fixed as soon as I'll release new version and push it to EPEL. Until that you could build zbackup from git.

mumblez · 2015-04-03T18:05:48Z

cool, will compile from source....

I don't get the 'Can't parse message FileHeader' but do get:

"size of the encrypted file is incorrect"

when trying to run gc, (using sub-folders)

Guessing the only work around atm is to move all backups out of sub-directories?

bartonfriedland · 2015-04-19T16:07:23Z

Using zbackup version 1.4-71-g9f93eca, when I attempt to gc a zbackup folder with subdirectories in the backups folder, I get the following in the output:

Checking backup ./backups/.DS_Store...
[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse message of type "FileHeader" because it is missing required fields: version
Can't parse message FileHeader

danmbox · 2015-04-27T12:39:45Z

@am1go will there be a new release (1.4.1.1 or 1.4.2)? The latest release (on github at least) is the one from December 2014.

Vlad1mir-D · 2015-07-30T13:06:45Z

Yes, I'll release a new version with fix as soon as I will have time for it.

2015-04-27 15:39 GMT+03:00 Dan Muresan notifications@github.com:

@am1go https://github.com/Am1GO will there be a new release (1.4.1.1 or
1.4.2)? The latest release (on github at least) is the one from December
2014.

—
Reply to this email directly or view it on GitHub
#50 (comment).

Kind regards,
Vladimir.

DrReD · 2015-09-11T11:37:28Z

Hello,
I just want to communicate that this problem still happens with v.1.4.3.
Thanks for the great program. :)

Vlad1mir-D · 2015-09-11T12:32:45Z

Oh, I forgot to fix it in 1.4 branch :(

Backported 4298a0f (fix for #50)

Vlad1mir-D added the bug label Jan 24, 2015

Vlad1mir-D self-assigned this Jan 24, 2015

Vlad1mir-D added a commit that referenced this issue Jan 25, 2015

Fix for #50 and additional workaround for #48

4298a0f

Vlad1mir-D closed this as completed Jan 25, 2015

Vlad1mir-D mentioned this issue Feb 22, 2015

GC failing with 'Can't parse message FileHeader' #65

Closed

Vlad1mir-D reopened this Sep 11, 2015

Vlad1mir-D modified the milestones: 1.4.3, 1.4.4 Sep 11, 2015

txtdawg mentioned this issue Sep 16, 2015

Fix for issue 50 for the 1.4 series. #96

Closed

Vlad1mir-D added a commit to Vlad1mir-D/zbackup that referenced this issue Sep 16, 2015

Backported 4298a0f (fix for zbackup#50)

411f8ee

Vlad1mir-D pushed a commit that referenced this issue Sep 16, 2015

Merge pull request #97 from Am1GO/stable-1.4

d780c37

Backported 4298a0f (fix for #50)

Vlad1mir-D closed this as completed Sep 16, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error during garbage collection with subfolders in backup/ folder #50

Error during garbage collection with subfolders in backup/ folder #50

juliadin commented Jan 24, 2015

Vlad1mir-D commented Jan 24, 2015

Vlad1mir-D commented Jan 25, 2015

juliadin commented Jan 25, 2015

mumblez commented Mar 31, 2015

mumblez commented Mar 31, 2015

Vlad1mir-D commented Apr 3, 2015

mumblez commented Apr 3, 2015

bartonfriedland commented Apr 19, 2015

danmbox commented Apr 27, 2015

Vlad1mir-D commented Jul 30, 2015

DrReD commented Sep 11, 2015

Vlad1mir-D commented Sep 11, 2015

Error during garbage collection with subfolders in backup/ folder #50

Error during garbage collection with subfolders in backup/ folder #50

Comments

juliadin commented Jan 24, 2015

Vlad1mir-D commented Jan 24, 2015

Vlad1mir-D commented Jan 25, 2015

juliadin commented Jan 25, 2015

mumblez commented Mar 31, 2015

mumblez commented Mar 31, 2015

Vlad1mir-D commented Apr 3, 2015

mumblez commented Apr 3, 2015

bartonfriedland commented Apr 19, 2015

danmbox commented Apr 27, 2015

Vlad1mir-D commented Jul 30, 2015

DrReD commented Sep 11, 2015

Vlad1mir-D commented Sep 11, 2015