Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebalance compression levels. #31

Merged
merged 6 commits into from
Feb 19, 2016
Merged

Conversation

klauspost
Copy link
Owner

This will rebalance the compression levels to have a more even distribution of compression levels. The general trend of this is to make compression faster, which means that a given compression level (except 1 and 9) will be faster.

The existing implementation favors compression, and from level 5 and up, very small compresssion improvements are added for very high computation cost.

Rough guidelines: Previous level 3 speed is now equal to level 6.

New default is level 5, which in most cases offers a quite good compression, without getting diminished returns from additional computation power.

See "rebalance" tabs for benchmarks of rebalanced levels:

https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing

@klauspost klauspost force-pushed the rebalance-compression-levels branch 2 times, most recently from 1243b10 to 2d8c5f7 Compare January 31, 2016 12:10
@willglynn
Copy link

One thing I note about the original values: they seem to correspond to the compression levels in zlib. Even the compress/flate constants mirror the zlib constants. Someone who's used zlib before might assume that Go compression levels 0-9 are similar to zlib (I did!), and prior to this changeset, they'd be right.

I like the idea of having more speed/size tradeoffs available, but I'd also like a note in the docs saying that the compression levels work differently than zlib.

@klauspost
Copy link
Owner Author

@willglynn You are right. It makes good sense to document why these values are different than zlib.

That said, in my tests zlib has always performed ~1 level worse than go deflate in terms of compression. Noting that you shouldn't expect settings to carry over 1:1 makes sense.

This will rebalance the compression levels to have a more even distribution of compression levels. The general trend of this is to make compression faster, which means that a given compression level (except 1 and 9) will be faster.

The existing implementation favors compression, and from level 5 and up, very small compresssion improvements are added for very high computation cost.

Rough guidelines: Previous level 3 speed is now equal to level 6.

New default is level 5, which in most cases offers a quite good compression, without getting diminished returns from additional computation power.

See "rebalance" tabs for benchmarks of rebalanced levels:

https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing

Make level 6 slightly faster, the decreased speed didn't yeild significant improvements.

Create new level 2 encoder.

Speed up level 1 by reusing the table.

Add documentation.

Remove buffer, if we get small block.

Update documentation

Add Level 3 "Snappy" compressor, which tests up to two hashes.
@klauspost klauspost force-pushed the rebalance-compression-levels branch from 9439634 to 5f70209 Compare February 13, 2016 12:49
klauspost added a commit that referenced this pull request Feb 19, 2016
@klauspost klauspost merged commit 0d66558 into master Feb 19, 2016
@klauspost klauspost deleted the rebalance-compression-levels branch January 18, 2018 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants