Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
src
 
 
 
 
 
 
 
 
 
 
 
 

fzcat

Fast zcat with mmap and miniz

I coincidentally noticed that using mmap, miniz, and C++11 threads together I can zcat multiple files faster than pigz so I decided to release the code separately for those who may benefit from it.

How is it faster

First, miniz is a blazing fast implementation of zlib. Second, instead of using open and read we use mmap and pipe that directly into miniz. Then we take the output written to a buffer and write it directly to stdout. This reduces the number of times the memory is copied around, increasing the performance.

How fast is it

Running against 189 files of about 20MB each

Warming up by catting all files to /dev/null
zcat 107.95278143882751
parallel_zcat 42.61844491958618
pigz 55.39036321640015
parallel_pigz 42.56399941444397
fzcat 28.799103498458862
Running again but dropping cache between runs
zcat 109.37209343910217
parallel_zcat 43.26405954360962
pigz 58.75359654426575
parallel_pigz 43.09493350982666
fzcat 31.75873327255249

When is it not faster

When decompressing a single file. In that instance, my per-file threading doesn’t apply and pigz wins

Compiling

You will need a C++ compiler that support C++14. I’d highly recommend the latest version of Clang.

premake4 gmake
make config=static

Usage

fzcat [<file1> [<file2> [<file3>...] ] ]

About

Fast zcat with mmap and miniz

Resources

License

Releases

No releases published

Packages

No packages published