Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LZIP compression #68

Closed
TimmThaler opened this Issue Aug 28, 2017 · 16 comments

Comments

Projects
None yet
2 participants
@TimmThaler
Copy link

TimmThaler commented Aug 28, 2017

It would be great, if you could add LZIP to support .lz compression.

http://www.nongnu.org/lzip/lzip.html

@aonez aonez self-assigned this Sep 5, 2017

@aonez aonez added the enhancement label Sep 5, 2017

@aonez aonez modified the milestones: Future, 1.1.0 Sep 5, 2017

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 5, 2017

Thought the provided binary was enough to implement support, but it lacks some features needed, so a code implementation or some code changes will be needed:

  • Output file destination in compression and extraction
  • Method selection
  • Split
  • Join multivolumes

Optional:

  • Exclude files (resource forks)
  • Progress percentage

Here a test version. Just for test purposes: Keka-1.1.0-beta.3r2151-lzip.zip

@aonez aonez added the core label Sep 5, 2017

@TimmThaler

This comment has been minimized.

Copy link
Author

TimmThaler commented Sep 6, 2017

This looks great!
I did a bit of reading. Perhaps that helps for coding.

  • The Command Line Version of lzip provides an option for the Output file destination:

    • -o, --output=<file> : if reading standard input, write to
  • The Progress Percentage is shown in Terminal, when you use the option -v twice.

    • -v -v will give you the percents and the MB of the already read data.
  • The Compression has 10 stages via the options -0 (no compression) to -9 (best compression).

  • Keeping a file after compression will be set with the option -k. Otherwise the input file will be deleted after (de)compressing.

  • Excluding files is not possible (as far as I read the man page).

Unfortunately, the command line version of lzip does not accept folders as an input. I tested a compression with your test version from above and I got an error message, that the archive could not be created.
Because lzip is just a compression tool for single files, folders should be packed into a tar archive which then can be compressed without problems.

I hope, I could help :)

  • Help File of the command line lzip: lzip.txt
@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 6, 2017

Thanks a lot for your research! In fact yesterday I got in to it and developed a patch so the binary accepts output folders and output filenames 😁
It's in early stages, so just as reference: output-write.patch.zip

Did not noticed the -v flag (x2) shows percentage. This will do. 👍

Because lzip is just a compression tool for single files, folders should be packed into a tar archive which then can be compressed without problems.

Noticed that. I'm currently using bsdtar for gzip and bzip, with lzip I think I'll need to create a two phase (tar first, lzip then).

Right now I'm battling with the multivolume decompression, I don't understand why it decompresses each part separately.

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 6, 2017

By the way, you use Lzip for backup purposes?

@TimmThaler

This comment has been minimized.

Copy link
Author

TimmThaler commented Sep 6, 2017

Really great!
I think that should work.

Yes, I use Lzip for backups of my nextcloud server to backup all the data and settings. First I put everything into one tar archive and then I compress with Lzip which saves 20% to 30% space. Actually, when I compress only the mysql backup file the result is up to 85% smaller than the input :-)

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 6, 2017

GNU tar has built in support for Lzip, gonna try this next, so there'll be no need to make a two step compression and extraction of tarball.

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 9, 2017

Trying plzip but getting performance issues, seems to saturate all the CPU threads...

@TimmThaler

This comment has been minimized.

Copy link
Author

TimmThaler commented Sep 9, 2017

Yes, I already thought something like this would happen. This compression ist quite a "CPU owner" - demands very much processor power.
When I use lzip in the macOS Terminal, it takes up to 95% of the CPU usage ... But in the end: The files are much smaller. 😄

@TimmThaler

This comment has been minimized.

Copy link
Author

TimmThaler commented Sep 9, 2017

When I install plzip via homebrew I got the problem, that I am not shown the percentual progress, just 0% and 100%, but nothing in between...

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 9, 2017

No performance issues?

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 10, 2017

...0% and 100%...

In fact now I remember I noticed this too, and this shows the CPU saturation.

Here you have more test builds:

The plzip binary does not have the split ability integrated, and I'm yet unable to join an lzip multivolume, so I may just remove this feature altogether.

@TimmThaler

This comment has been minimized.

Copy link
Author

TimmThaler commented Sep 10, 2017

  • With the LZIP version I got about 50% to 60% CPU usage and with the PLZIP version over 90%. That there will be some performance issues for other applications like Firefox is (for me) ok 😆 😄
  • I don't think, LZIP has multivolume support. The help file doesn't contain any information about that.

@aonez aonez modified the milestones: 1.1.0, Future Sep 13, 2017

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 13, 2017

I'm gonna use lzip for now (without split), I can't even get plzip paused on time.

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 13, 2017

Will add an option to use plzip instead:

defaults write com.aone.Keka UseMultithreadLzip <true|false>
@TimmThaler

This comment has been minimized.

Copy link
Author

TimmThaler commented Sep 13, 2017

Great. This way it will not freeze the system or take too much CPU performance and users still can decide if they want this 😄 👍

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Sep 13, 2017

Anyway I'll try to work on this, since plzip is much more faster for big files. Thanks to this thread I'm now using pigz and lbzip2 on the next Keka, for multithreaded gzip and bzip2 tarballs 😁

@aonez aonez closed this Sep 13, 2017

@aonez aonez referenced this issue Sep 13, 2017

Closed

Use plzip instead of lzip #75

1 of 2 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.