Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd: Skip random data faster #147

Merged
merged 4 commits into from
Aug 14, 2019
Merged

zstd: Skip random data faster #147

merged 4 commits into from
Aug 14, 2019

Conversation

klauspost
Copy link
Owner

If we did not get a single match, assume that the data cannot be entropy compressed.

It is very likely that the data is random, and in the rare case it is not, we can live with it.

Before:

BenchmarkRandomEncodeAllFastest-12           100          10475190 ns/op        1001.01 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           100          11075653 ns/op         946.74 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             200           8126950 ns/op        1290.25 MB/s         139 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             200           8059989 ns/op        1300.96 MB/s          90 B/op          2 allocs/op

After:

BenchmarkRandomEncodeAllFastest-12           300           4425967 ns/op        2369.15 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           300           5108791 ns/op        2052.49 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             300           5085627 ns/op        2061.84 MB/s          89 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             300           5947031 ns/op        1763.19 MB/s          65 B/op          2 allocs/op

If we did not get a single match, assume that the data cannot be entropy compressed.

It is very likely that the data is random, and in the rare case it is not, we can live with it.

Before:
```
BenchmarkRandomEncodeAllFastest-12           100          10475190 ns/op        1001.01 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           100          11075653 ns/op         946.74 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             200           8126950 ns/op        1290.25 MB/s         139 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             200           8059989 ns/op        1300.96 MB/s          90 B/op          2 allocs/op
```

After:
```
BenchmarkRandomEncodeAllFastest-12           300           4425967 ns/op        2369.15 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           300           5108791 ns/op        2052.49 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             300           5085627 ns/op        2061.84 MB/s          89 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             300           5947031 ns/op        1763.19 MB/s          65 B/op          2 allocs/op
```
@klauspost klauspost merged commit f82c96c into master Aug 14, 2019
@klauspost klauspost deleted the zstd-skip-random-faster branch August 14, 2019 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant