Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Snapshot uploaded chunk lengths #500
Introduces 'upload-lengths' to snapshots; containing an array of the post compression & encryption chunk lengths. These upload lengths can then be used when checking snapshot integrity. Not only verifying that the chunk names are available, but that the chunks are of the expected size.
Additionally adds a '-check' flag to backup. This will check the unchanged, remote chunk lengths, to ensure that they are of the expected length. Remote chunks of an unexpected size will be re-uploaded (self-healing). Check is currently a flag as we do need to first pull down the chunk file list from the remote. Not sure how lengthy/costly an operation this could be? so made it optional.
Remote length checks will be skipped for older pre-existing snapshots lacking 'upload-lengths'. What this means is that the first snapshot to introduce upload-lengths will need to recompute the uploaded chunk lengths. These chunks do not need to be re-uploaded however, unless the lengths of the remote chunks are different than locally computed.
Primary motivation is to help address an issue where my sftp backup repository contains zero sized chunks. I suspect this happened during a kernel panic while a backup was running. Discovered this issue while trying to copy chunks to cloud storage.
duplicacy 2.1.1 (e8b892):
Remove a chunk:
Replace with a zero sized chunk:
Now with these changes:
So let's try changing the length again: