Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Please add gzip/bzip compression for delta files in rdiff #8
I tried to use flags --gzip/--bzip for rdiff but got error:
rdiff: ERROR: (rdiff_options) sorry, compression is not really implemented yet
For my data (VPS disks) compression provides really excellent compression for delta files:
source size: 4.6 Gb delta size: 2093.0 MB compressed size: 223.0 source size: 14.8 Gb delta size: 2205.0 MB compresses size: 998.7 MB
I tried to compress signatures biut it's really useless:
du -sh /root/rdiff_signatures_25_june/ 20M /root/rdiff_signatures_25_june/ tar -cpzf /root/rdiff_signatures_25_june.tar.gz /root/rdiff_signatures_25_june/ ls -alh /root/rdiff_signatures_25_june.tar.gz -rw-r--r-- 1 root root 19M Авг 1 00:41 /root/rdiff_signatures_25_june.tar.gz tar -cpjf /root/rdiff_signatures_25_june.tar.bz2 /root/rdiff_signatures_25_june/ ls -alh /root/rdiff_signatures_25_june.tar.bz2 -rw-r--r-- 1 root root 19M Авг 1 00:41 /root/rdiff_signatures_25_june.tar.bz2
But compression for deltas is really useful, please add it :)
Note rsync uses a modified zlib for delta compression that uses matching data that is not included in the delta to "prime" the compression data tables and then throws away the "matching" compressed output. This in general gives slightly better compression than just gzipping the resulting delta. For an example of how this can be done with an unmodified zlib you can look a pysync http://minkirri.apana.org.au/~abo/projects/pysync.
I'm considering tackling this next. Either that or Rabin-Karp rollsums... whichever people prefer.
Note that signature files being collections of hash values probably don't compress at all well, unless they have long runs of identical blocks. I'm planning to only add compression to the deltas, with optional "context compression" support (which compresses hits as well as misses to prime the compressor with context from matching blocks).