Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd: Add experimental encoder dictionaries #281

Merged
merged 9 commits into from
Sep 3, 2020

Commits on Aug 31, 2020

  1. zstd: Add experimental encoder dictionaries

    Add encoder dictionary support.
    
    Currently uses the dictionary content and literal compression table.
    
    The only unused elements are the FSE tables.
    klauspost committed Aug 31, 2020
    Configuration menu
    Copy the full SHA
    f757a49 View commit details
    Browse the repository at this point in the history
  2. Update README.md

    klauspost committed Aug 31, 2020
    Configuration menu
    Copy the full SHA
    04e3c49 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2020

  1. Fix zero value offsets.

    klauspost committed Sep 1, 2020
    Configuration menu
    Copy the full SHA
    63e255f View commit details
    Browse the repository at this point in the history
  2. Tweak tests.

    klauspost committed Sep 1, 2020
    Configuration menu
    Copy the full SHA
    4df4e69 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bf51405 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2020

  1. Optimize some double work and remove accidental allocs

    ```
    BenchmarkEncodeAllDict/length-19-level-fastest-dict-1
    BenchmarkEncodeAllDict/length-19-level-fastest-dict-1-32         	  141180	      8351 ns/op	   2.28 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-19-level-default-dict-1
    BenchmarkEncodeAllDict/length-19-level-default-dict-1-32         	   19639	     60085 ns/op	   0.32 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-19-level-better-dict-1
    BenchmarkEncodeAllDict/length-19-level-better-dict-1-32          	    5714	    196535 ns/op	   0.10 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-12131-level-fastest-dict-1
    BenchmarkEncodeAllDict/length-12131-level-fastest-dict-1-32      	   19077	     59129 ns/op	 205.16 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-12131-level-default-dict-1
    BenchmarkEncodeAllDict/length-12131-level-default-dict-1-32      	    3999	    301825 ns/op	  40.19 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-12131-level-better-dict-1
    BenchmarkEncodeAllDict/length-12131-level-better-dict-1-32       	    2352	    495323 ns/op	  24.49 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-210569-level-fastest-dict-1
    BenchmarkEncodeAllDict/length-210569-level-fastest-dict-1-32     	    1052	   1115971 ns/op	 188.69 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-210569-level-default-dict-1
    BenchmarkEncodeAllDict/length-210569-level-default-dict-1-32     	     673	   1763746 ns/op	 119.39 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-210569-level-better-dict-1
    BenchmarkEncodeAllDict/length-210569-level-better-dict-1-32      	     423	   2820325 ns/op	  74.66 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-102605-level-fastest-dict-1
    BenchmarkEncodeAllDict/length-102605-level-fastest-dict-1-32     	    2352	    494047 ns/op	 207.68 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-102605-level-default-dict-1
    BenchmarkEncodeAllDict/length-102605-level-default-dict-1-32     	    1142	   1049912 ns/op	  97.73 MB/s	       0 B/op	       0 allocs/op
    BenchmarkEncodeAllDict/length-102605-level-better-dict-1
    BenchmarkEncodeAllDict/length-102605-level-better-dict-1-32      	     662	   1791540 ns/op	  57.27 MB/s	       0 B/op	       0 allocs/op
    ```
    klauspost committed Sep 2, 2020
    Configuration menu
    Copy the full SHA
    f44e25b View commit details
    Browse the repository at this point in the history
  2. Load all.

    klauspost committed Sep 2, 2020
    Configuration menu
    Copy the full SHA
    e78ff31 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    05d3af8 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2020

  1. Configuration menu
    Copy the full SHA
    6c64484 View commit details
    Browse the repository at this point in the history