Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to restore backup: blocksize mismatch in manifest #2323

Closed
ccwoolf opened this issue Feb 19, 2017 · 12 comments

Comments

Projects
None yet
8 participants
@ccwoolf
Copy link

commented Feb 19, 2017

I have:

  • searched open and closed issues for duplicates

Version info

Duplicati Version: 2.0.1.45_experimental_2017-02-03
Operating System: Windows 10 (10.0.14393)
Backend: Local folder or drive

Bug description

Attempting to restore a backup results in the error:

Failed to process file: duplicati-20170219T070000Z.dlist.zip
Duplicati.Library.Main.Volumes.InvalidManifestException: Invalid manifest detected, the field Blocksize has value 1048576 but the value 102400 was expected
   at Duplicati.Library.Main.Volumes.VolumeBase.ManifestData.VerifyManifest(String manifest, Int64 blocksize, String blockhash, String filehash)
   at Duplicati.Library.Main.Volumes.VolumeReaderBase..ctor(ICompression compression, Options options)
   at Duplicati.Library.Main.Volumes.FilesetVolumeReader..ctor(String compressor, String file, Options options)
   at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)

After this error is thrown, no files are listed for restoration on the Restore Files page.

Steps to reproduce

  • Create a backup using the local drive/folder backend
  • Specify the following options:
    • zip-compression-method: LZMA
    • zip-compression-level: 9
    • blocksize: 1 Mbyte
  • Run the backup, and wait for it to complete
  • Attempt to restore files from the backup

Actual result:
No files are listed in the Restore Files dialogue when attempting to restore.

The following messages are logged:
Rebuild database started, downloading 1 filelists

Failed to process file: duplicati-20170219T070000Z.dlist.zip
Duplicati.Library.Main.Volumes.InvalidManifestException: Invalid manifest detected, the field Blocksize has value 1048576 but the value 102400 was expected
   at Duplicati.Library.Main.Volumes.VolumeBase.ManifestData.VerifyManifest(String manifest, Int64 blocksize, String blockhash, String filehash)
   at Duplicati.Library.Main.Volumes.VolumeReaderBase..ctor(ICompression compression, Options options)
   at Duplicati.Library.Main.Volumes.FilesetVolumeReader..ctor(String compressor, String file, Options options)
   at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)

Recreate/path-update completed, not running consistency checks

MainOperation: Repair
ParsedResult: Warning
EndTime: 19/02/2017 09:31:47
BeginTime: 19/02/2017 09:31:45
Duration: 00:00:01.8148919
Messages: [
    Rebuild database started, downloading 1 filelists,
    Recreate/path-update completed, not running consistency checks
]
Warnings: [
    Failed to process file: duplicati-20170219T070000Z.dlist.zip => Invalid manifest detected, the field Blocksize has value 1048576 but the value 102400 was expected
]
Errors: []
BackendStatistics:
    RemoteCalls: 2
    BytesUploaded: 0
    BytesDownloaded: 63949090
    FilesUploaded: 0
    FilesDownloaded: 1
    FilesDeleted: 0
    FoldersCreated: 0
    RetryAttempts: 0
    UnknownFileSize: 0
    UnknownFileCount: 0
    KnownFileCount: 0
    KnownFileSize: 0
    LastBackupDate: 01/01/0001 00:00:00
    BackupListCount: 0
    TotalQuotaSpace: 0
    FreeQuotaSpace: 0
    AssignedQuotaSpace: 0
    ParsedResult: Success

Expected result: Files are listed for restoration in the Restore Files dialogue

Screenshots

chrome_2017-02-19_09-40-30


Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

@kenkendk

This comment has been minimized.

Copy link
Member

commented Feb 19, 2017

It should work without configuration, but for some reason it does not automatically detect the new block size in your setup. You can set the block size in the "Advanced options" on the second page of the restore (the first page also has a section with "Advanced options" but these are for the destination.

@ccwoolf

This comment has been minimized.

Copy link
Author

commented Feb 20, 2017

Hey @kenkendk, thanks for the reply.

I've tried passing the option --blocksize=1048576 into Duplicati's advanced box on the restore page (screenshot - this is where they go, right?) however I get a similar error:

MainOperation: Repair
ParsedResult: Warning
EndTime: 20/02/2017 18:57:40
BeginTime: 20/02/2017 18:57:39
Duration: 00:00:01.2217554
Messages: [
    Rebuild database started, downloading 1 filelists,
    Recreate/path-update completed, not running consistency checks
]
Warnings: [
    Failed to process file: duplicati-20170220T184154Z.dlist.zip => Invalid manifest detected, the field Blocksize has value 1048576 but the value 1073741824 was expected
]
Errors: []
BackendStatistics:
    RemoteCalls: 2
    BytesUploaded: 0
    BytesDownloaded: 152914
    FilesUploaded: 0
    FilesDownloaded: 1
    FilesDeleted: 0
    FoldersCreated: 0
    RetryAttempts: 0
    UnknownFileSize: 0
    UnknownFileCount: 0
    KnownFileCount: 0
    KnownFileSize: 0
    LastBackupDate: 01/01/0001 00:00:00
    BackupListCount: 0
    TotalQuotaSpace: 0
    FreeQuotaSpace: 0
    AssignedQuotaSpace: 0
    ParsedResult: Success

I've tried substituting 1048576 with 1073741824 in my supplied options, but Duplicati tells me this is too big for a block size.
After some Googling I learned of the --dblock-size option, though attempting to use

--blocksize=1048576
--dblock-size=1073741824

as advanced options throws the same error as above.

@kenkendk

This comment has been minimized.

Copy link
Member

commented Feb 21, 2017

Yes, that should be the right place to add the options.
I will need to investigate why it does not work.

@kenkendk kenkendk added this to the mini issues milestone Feb 21, 2017

@tomwaldnz

This comment has been minimized.

Copy link

commented May 22, 2017

I'm having the same problem, I can't run a restore. This has shaken my confidence in Duplicati. I was planning on switching my main backups over, but I'm now wondering if the software is ready. I figured after 2 years in beta it'd be ok, but maybe with backups I need to be more prudent and try CloudBerry or similar.

I have my files backed up to Amazon S3 (plus some others locally, but that's not relevant here). I customised the block size and upload volume size before my initial backup, with a block size of 200kb (204800 bytes) and an upload volume size of 500.

I installed Duplicati on a different machine and did a test restore and got this error message.

Failed to process file: duplicati-20170520T074022Z.dlist.7z.aes => Invalid manifest detected, the field Blocksize has value 204800 but the value 102400 was expected

Based on this bug report I added the "--blocksize=204800" parameter in the advanced section.

I ran the restore again and I got a very similar error message, but with different numbers and referring to a different file

Failed to process file: duplicati-20170522T073218Z.dlist.7z.aes => Invalid manifest detected, the field Blocksize has value 204800 but the value 209715200 was expected

I note that 209715200 / 1024 = 204800, but I don't know how that's relevant.

-- Env info --
Duplicati Version: 2.0.1.53_experimental_2017-03-13
Operating System: Windows 10
Backend: Amazon S3

Error #1

Warning
Recreate/path-update completed, not running consistency checks

Result
MainOperation: Repair

ParsedResult: Warning

EndTime: 5/22/2017 7:42:28 PM

BeginTime: 5/22/2017 7:42:25 PM

Duration: 00:00:02.6559876

Messages: [

Rebuild database started, downloading 1 filelists,

Recreate/path-update completed, not running consistency checks

]

Warnings: [

Failed to process file: duplicati-20170520T074022Z.dlist.7z.aes => Invalid manifest detected, the field Blocksize has value 204800 but the value 102400 was expected

]

Errors: []

BackendStatistics:

RemoteCalls: 2

BytesUploaded: 0

BytesDownloaded: 466221

FilesUploaded: 0

FilesDownloaded: 1

FilesDeleted: 0

FoldersCreated: 0

RetryAttempts: 0

UnknownFileSize: 0

UnknownFileCount: 0

KnownFileCount: 0

KnownFileSize: 0

LastBackupDate: 1/1/0001 12:00:00 AM

BackupListCount: 0

TotalQuotaSpace: 0

FreeQuotaSpace: 0

AssignedQuotaSpace: 0

ParsedResult: Success

Error #2

MainOperation: Repair

ParsedResult: Warning

EndTime: 5/22/2017 7:59:05 PM

BeginTime: 5/22/2017 7:59:00 PM

Duration: 00:00:05.7069964

Messages: [

Rebuild database started, downloading 1 filelists,

Recreate/path-update completed, not running consistency checks

]

Warnings: [

Failed to process file: duplicati-20170522T073218Z.dlist.7z.aes => Invalid manifest detected, the field Blocksize has value 204800 but the value 209715200 was expected

]

Errors: []

BackendStatistics:

RemoteCalls: 2

BytesUploaded: 0

BytesDownloaded: 460525

FilesUploaded: 0

FilesDownloaded: 1

FilesDeleted: 0

FoldersCreated: 0

RetryAttempts: 0

UnknownFileSize: 0

UnknownFileCount: 0

KnownFileCount: 0

KnownFileSize: 0

LastBackupDate: 1/1/0001 12:00:00 AM

BackupListCount: 0

TotalQuotaSpace: 0

FreeQuotaSpace: 0

AssignedQuotaSpace: 0

ParsedResult: Success
@mr-flibble

This comment has been minimized.

Copy link
Contributor

commented Dec 29, 2017

Hello,
I have same problem when I restoring from another computer (without local database via "Direct restore from backup files ...")

Restoring from PC with local database is without problem.
Duplication version is 2.0.2.14_canary_2017-12-22

Error (warning):

MainOperation: Repair
 ParsedResult: Warning
 EndTime: 29. 12. 2017 11:56:21 (1514544981)
 BeginTime: 29. 12. 2017 11:56:14 (1514544974) Duration: 00:00:06.4483265
Messages: [ Rebuild database started,
 downloading 1 filelists, Recreate/path-update completed, not running consistency checks ]
 Warnings: [ Failed to process file: duplicati-20171229T092843Z.dlist.zip.aes =>
 Invalid manifest detected,
 the field Blocksize has value 1048576 but the value 102400 was expected ]
Errors: []
ParsedResult: Success

My backup parameters:
--zip-compression-level=0
--quota-size=960GB
--blocksize=1MB
--snapshot-policy=On
--symlink-policy=Follow

I find workaround:

If I export backup backup job from computer with database and read additional information - (dblock-size), then I can use it for restore.

So if I restoring with parameters:
--blocksize=1MB
--dblock-size=200MB

no error will appear and I can browse versions but this is not very intuitive...

@loesvs

This comment has been minimized.

Copy link

commented Mar 1, 2018

I had the exact same issue as mrflibble, except for one thing: I was too late for the workaround. I set up a backup ages ago and customized block- and dblocksize. As would be expected, I forgot the exact block sizes I chose. Using the Duplicati GUI, I regularly checked whether my backup was maintained well, by opening the "Direct restore" option and trying to restore any random file. Actually, I did so last week and it worked just fine. To restore, I never had to bother about the block size, as the system was nice enough to remember it for me.

Inevitably, the big crash happened. I had to restore my complete system and as import stuff was stored in the cloud anyways, I took the chance to start from a clean OS install. Which meant I had to reinstall Duplicati as well - and although the backupped database was untouched, I could not restore the files anymore as neither I nor the all-fresh Duplicati installation remembered my customized block sizes.

The workaround @mr-flibble suggests only works if one of you systems still has the original setup. Mine didn't. However, it's not too difficult to find the parameters required.

Workaround for @mr-flibble's workaround

If you have a backup with customized block/dblock sizes and you don't remember how big they were
and
you set up Duplicati2 on a new system and the old installation is not available anymore (so you can't export settings):

Find block size

Luckily, the error you get already mentions one of the missing values. In my case:

the field Blocksize has value 409600 but the value 419430400 was expected

so the blocksize is 409600B.

Find dblock size

  • Open the local database containing the backup
  • Sort the files by size
  • The size of the largest files is your dblock size (round up if necessary to get a value that makes sense)
    Note These should be dblocks, the files of which the name ends in .dblock.zip.aes
  • In my case, the largest files were all 149 MB. As I would never have chosen such a weird value, I figured my dblocks were supposed to be max 150 MB

Restore files

  • Using Duplicati on your new system, restore with parameters (obviously customized to your own values):
--blocksize=409600B
--dblock-size=150MB

For those who are as lazy as me and use the Duplicati GUI:

Restore > Direct restore from backup files

  1. Backup location: choose backup location.
  2. Encryption: enter passphrase and below, click 'Advanced options' and copy the code mentioned above.
  3. Wait for a while and see your files appear
  4. Happily restore!
@kenkendk

This comment has been minimized.

Copy link
Member

commented Jun 29, 2018

Adding to this issue, the --dblock-size parameter is the size of the *.dblock.zip files. This limit is only used when creating the files, as a cut-off to ensure we do not have huge files that needs to be uploaded. It is not used during restore, so you do not have to set it for restores. You can also change this value in existing backups if you like, and new uploads will use the new value.

The --blocksize parameter IS required during restore. When restoring from the commandline, Duplicati will read the size from the first volume it downloads, and use that for the restore operation. Something goes wrong when restoring from the GUI, where the parameter --blocksize=100kb is set, causing the auto-detection of the blocksize to be disabled (the logic is that the user has explicitly requested a specific value by setting the option).

@tomwaldnz: You can do as @loesvs suggests, and snag the value from the error message. When setting the size, be aware that the parameter expects KiB sizes. If you want to specify the value in bytes, you need to add b to then end of the number.

You need to change: --blocksize=1048576 to --blocksize=1048576b.

Without testing, it should also be possible to use --blocksize=, meaning "force the block size to empty", which should then enable the automatic blocksize detection (no explicit value specified).

@kenkendk kenkendk closed this in 355ef92 Jun 30, 2018

@tomwaldnz

This comment has been minimized.

Copy link

commented Jun 30, 2018

Awesome, thanks for fixing this Ken. Once that change gets into a build I'll install Duplicati again and give it another run.

@tomwaldnz tomwaldnz referenced this issue Jul 2, 2018

Closed

Unable to restore backups #2469

1 of 1 task complete
@JonMikelV

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2018

@tomwaldnz, I believe it's in the 2.0.3.9 release which must have come out pretty much as you were typing your message. :-)
https://forum.duplicati.com/t/release-2-0-3-9-canary-2018-06-30/3974

@tomwaldnz

This comment has been minimized.

Copy link

commented Jul 2, 2018

@JonMikelV yes the canary came out, but I found another bug that prevents restore. It looks like an old one that was closed, so I commented there, but I'll probably open a new bug later today as there's no way I can see to reopen the existing bug.

It's disappointing to wait a year for a bug fix only to find another bug that prevents restores.

@Onurtag

This comment has been minimized.

Copy link

commented Feb 28, 2019

Sorry if I understood it wrong but on the latest canary (duplicati-2.0.4.15_canary_2019-02-06), I get the same error if I don't add the --blocksize option.
Dblock-size doesn't seem to be required as expected but I'm not sure if blocksize is supposed to be automatically detected from the backup files or is it necessery to point it when I want to restore a backup?

Steps (duplicati-2.0.4.15):

  1. Make a new backup of random files.
  2. Make changes and backup again x3
  3. Direct restore from backup files
  4. Point to the backup location, input the password, DONT add any advanced options
  5. On the select files screen, pick a different backup version and I get a warning with the same message.

On step 4 if I add my --blocksize value as an advanced option it works without a warning.

@ts678

This comment has been minimized.

Copy link

commented Jun 11, 2019

Confirming @Onurtag finding of an incomplete fix while looking at Restores on non standard parameters. Made a test backup on 2.0.4.15, then confirmed failure on 2.0.4.15, 2.0.3.9, and 2.0.4.18 (where I did further debugging using a debugger). The key is step 5. The newest version comes up properly and automatically when advancing to screen 3 to select files, and the code fix is used. For the others, the code fix is bypassed because autoDetectBlockSize is set to false. Viewing Configuration table of temporary folder dup-<GUID> database (fish around -- it's probably the very latest file) finds blocksize with a value of 102400 despite the manifest. Editing it to 102401 causes that number to be reported on the eventual error, so I think issues are 1) default value gets in DB, and 2) it's read/used. Also ran 2.0.3.8, and it failed on the transition to screen 3, so the fix is at least able to get the latest without adding manual --blocksize in screen 2 Advanced Options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.