Rebalance compression levels. #31

Merged
merged 6 commits into from Feb 19, 2016

Conversation

Projects
None yet
2 participants
@klauspost
Owner

klauspost commented Jan 30, 2016

This will rebalance the compression levels to have a more even distribution of compression levels. The general trend of this is to make compression faster, which means that a given compression level (except 1 and 9) will be faster.

The existing implementation favors compression, and from level 5 and up, very small compresssion improvements are added for very high computation cost.

Rough guidelines: Previous level 3 speed is now equal to level 6.

New default is level 5, which in most cases offers a quite good compression, without getting diminished returns from additional computation power.

See "rebalance" tabs for benchmarks of rebalanced levels:

https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing

@willglynn

This comment has been minimized.

Show comment
Hide comment
@willglynn

willglynn Feb 1, 2016

One thing I note about the original values: they seem to correspond to the compression levels in zlib. Even the compress/flate constants mirror the zlib constants. Someone who's used zlib before might assume that Go compression levels 0-9 are similar to zlib (I did!), and prior to this changeset, they'd be right.

I like the idea of having more speed/size tradeoffs available, but I'd also like a note in the docs saying that the compression levels work differently than zlib.

One thing I note about the original values: they seem to correspond to the compression levels in zlib. Even the compress/flate constants mirror the zlib constants. Someone who's used zlib before might assume that Go compression levels 0-9 are similar to zlib (I did!), and prior to this changeset, they'd be right.

I like the idea of having more speed/size tradeoffs available, but I'd also like a note in the docs saying that the compression levels work differently than zlib.

@klauspost

This comment has been minimized.

Show comment
Hide comment
@klauspost

klauspost Feb 1, 2016

Owner

@willglynn You are right. It makes good sense to document why these values are different than zlib.

That said, in my tests zlib has always performed ~1 level worse than go deflate in terms of compression. Noting that you shouldn't expect settings to carry over 1:1 makes sense.

Owner

klauspost commented Feb 1, 2016

@willglynn You are right. It makes good sense to document why these values are different than zlib.

That said, in my tests zlib has always performed ~1 level worse than go deflate in terms of compression. Noting that you shouldn't expect settings to carry over 1:1 makes sense.

klauspost added some commits Feb 13, 2016

Rebalance compression levels.
This will rebalance the compression levels to have a more even distribution of compression levels. The general trend of this is to make compression faster, which means that a given compression level (except 1 and 9) will be faster.

The existing implementation favors compression, and from level 5 and up, very small compresssion improvements are added for very high computation cost.

Rough guidelines: Previous level 3 speed is now equal to level 6.

New default is level 5, which in most cases offers a quite good compression, without getting diminished returns from additional computation power.

See "rebalance" tabs for benchmarks of rebalanced levels:

https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing

Make level 6 slightly faster, the decreased speed didn't yeild significant improvements.

Create new level 2 encoder.

Speed up level 1 by reusing the table.

Add documentation.

Remove buffer, if we get small block.

Update documentation

Add Level 3 "Snappy" compressor, which tests up to two hashes.

klauspost added a commit that referenced this pull request Feb 19, 2016

@klauspost klauspost merged commit 0d66558 into master Feb 19, 2016

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@klauspost klauspost deleted the rebalance-compression-levels branch Jan 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment