Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebalance compression levels. #31

Merged
merged 6 commits into from Feb 19, 2016
Merged

Rebalance compression levels. #31

merged 6 commits into from Feb 19, 2016

Conversation

@klauspost
Copy link
Owner

klauspost commented Jan 30, 2016

This will rebalance the compression levels to have a more even distribution of compression levels. The general trend of this is to make compression faster, which means that a given compression level (except 1 and 9) will be faster.

The existing implementation favors compression, and from level 5 and up, very small compresssion improvements are added for very high computation cost.

Rough guidelines: Previous level 3 speed is now equal to level 6.

New default is level 5, which in most cases offers a quite good compression, without getting diminished returns from additional computation power.

See "rebalance" tabs for benchmarks of rebalanced levels:

https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing

@klauspost klauspost force-pushed the rebalance-compression-levels branch 2 times, most recently from 1243b10 to 2d8c5f7 Jan 30, 2016
@willglynn
Copy link

willglynn commented Feb 1, 2016

One thing I note about the original values: they seem to correspond to the compression levels in zlib. Even the compress/flate constants mirror the zlib constants. Someone who's used zlib before might assume that Go compression levels 0-9 are similar to zlib (I did!), and prior to this changeset, they'd be right.

I like the idea of having more speed/size tradeoffs available, but I'd also like a note in the docs saying that the compression levels work differently than zlib.

@klauspost
Copy link
Owner Author

klauspost commented Feb 1, 2016

@willglynn You are right. It makes good sense to document why these values are different than zlib.

That said, in my tests zlib has always performed ~1 level worse than go deflate in terms of compression. Noting that you shouldn't expect settings to carry over 1:1 makes sense.

klauspost added 2 commits Feb 13, 2016
This will rebalance the compression levels to have a more even distribution of compression levels. The general trend of this is to make compression faster, which means that a given compression level (except 1 and 9) will be faster.

The existing implementation favors compression, and from level 5 and up, very small compresssion improvements are added for very high computation cost.

Rough guidelines: Previous level 3 speed is now equal to level 6.

New default is level 5, which in most cases offers a quite good compression, without getting diminished returns from additional computation power.

See "rebalance" tabs for benchmarks of rebalanced levels:

https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing

Make level 6 slightly faster, the decreased speed didn't yeild significant improvements.

Create new level 2 encoder.

Speed up level 1 by reusing the table.

Add documentation.

Remove buffer, if we get small block.

Update documentation

Add Level 3 "Snappy" compressor, which tests up to two hashes.
@klauspost klauspost force-pushed the rebalance-compression-levels branch from 9439634 to 5f70209 Feb 13, 2016
klauspost added a commit that referenced this pull request Feb 19, 2016
Rebalance compression levels.
@klauspost klauspost merged commit 0d66558 into master Feb 19, 2016
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@klauspost klauspost deleted the rebalance-compression-levels branch Jan 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.