Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compression option #12

Closed
abbbi opened this issue May 15, 2021 · 2 comments
Closed

Add compression option #12

abbbi opened this issue May 15, 2021 · 2 comments
Labels
enhancement New feature or request help wanted Extra attention is welcome Needs-testing Feature needs testing

Comments

@abbbi
Copy link
Owner

abbbi commented May 15, 2021

Add compression via python lz4 bindings during backup. Ideas:

To support inline compression during backup, using the sparsestream format, the following has to be taken into account:

  1. Enhance sparsestream format for version 2, add information about compression method and wether if compression was enabled to metadata header
  2. After saving all data to backup stream, dump json information about the data blocks original and compressed size as json array at the end of the backup stream (after the last STOP frame)
  3. After the json information has been dumped to the file, add new sparsestream frame type "comp" which is a regular header with offset and size information about the size of the dumped json information as very last entry to the stream file.

During restore:

  1. read metadata header and check if compression is enabled
  2. if compression is enabled, seek to the very end of the file and read the last frame type, which holds the "comp" information
  3. Seek back the size read from the "comp" metadata header
  4. Read the json array with hold the original and compressed data size.
  5. Use this information for reading the data blocks in compressed size, decompress and write to target file, check if decompressed size matches the original size of the data.
@abbbi abbbi added enhancement New feature or request good first issue Good for newcomers labels May 15, 2021
abbbi added a commit that referenced this issue May 17, 2021
abbbi added a commit that referenced this issue May 18, 2021
abbbi added a commit that referenced this issue May 18, 2021
@abbbi abbbi removed the good first issue Good for newcomers label May 19, 2021
@abbbi
Copy link
Owner Author

abbbi commented May 19, 2021

Feature is implemented in master branch, but needs some testing :)

@abbbi abbbi added help wanted Extra attention is welcome Needs-testing Feature needs testing labels May 19, 2021
@abbbi
Copy link
Owner Author

abbbi commented Jun 5, 2021

Version 0.22 includes --compress option.

@abbbi abbbi closed this as completed Jun 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is welcome Needs-testing Feature needs testing
Projects
None yet
Development

No branches or pull requests

1 participant