Skip to content
Fast zcat with mmap and miniz
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
3rdparty/miniz
src
test
.gitignore
LICENSE
README.org
premake4.lua

README.org

fzcat

Fast zcat with mmap and miniz

I coincidentally noticed that using mmap, miniz, and C++11 threads together I can zcat multiple files faster than pigz so I decided to release the code separately for those who may benefit from it.

How is it faster

First, miniz is a blazing fast implementation of zlib. Second, instead of using open and read we use mmap and pipe that directly into miniz. Then we take the output written to a buffer and write it directly to stdout. This reduces the number of times the memory is copied around, increasing the performance.

How fast is it

Running against 189 files of about 20MB each

Warming up by catting all files to /dev/null
zcat 107.95278143882751
parallel_zcat 42.61844491958618
pigz 55.39036321640015
parallel_pigz 42.56399941444397
fzcat 28.799103498458862
Running again but dropping cache between runs
zcat 109.37209343910217
parallel_zcat 43.26405954360962
pigz 58.75359654426575
parallel_pigz 43.09493350982666
fzcat 31.75873327255249

When is it not faster

When decompressing a single file. In that instance, my per-file threading doesn’t apply and pigz wins

Compiling

You will need a C++ compiler that support C++14. I’d highly recommend the latest version of Clang.

premake4 gmake
make config=static

Usage

fzcat [<file1> [<file2> [<file3>...] ] ]
You can’t perform that action at this time.