Java port(JNI) of Snappy & LZ4 compression codec
C++ C Java Common Lisp Shell
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
prebuild
src
testdata
.gitignore
LICENSE
README.md
make.sh
pom.xml
renew-snappy.sh

README.md

Introduction

JNICompressions includes JNI wrappers for some of the fastest native lightweight compression libraries, which includes:

LZ4:
http://lz4.googlecode.com/svn/trunk
revision 68

Google Snappy
http://code.google.com/p/snappy/
version 1.0.5

LZ4 and Snappy is bundled with this library, so you do not need to install LZ4 and snappy before building this library.

There are 3 interfaces provided:

* DirectBuffer interface;
  The speed of DirectBuffer interface is almost identical to native C++ interface performance;

* Reusable ByteArray interface;
  After using GetPrimitiveArrayCritical instead of malloc & copy, the performance is now the 
  same as DirectBuffer on Oracle JDKs. Thanks Adrien Grand for the advice. 

* Simple ByteArray interface;
  To improve usability, a very simple byte array interface is added too, but it is 
  much slower, because every compression/decompression need to create new byte arrays(
  involving malloc/free, array initialization, cache invalidate, data copy), 
  this limits total throughput especially in high compression/decompression speed case.

Notice: Current build script only support LINUX & MACOSX x86_64.

Build

There are pre-build Mac OS X 64bit binaries available in /prebuild, currently I do not have other build environments, sorry for the inconvenience, I will try to add more platform and integrate JNI libraries into a single jar file like snappy-java in the future.

Make jar & native library using: $mvn package

Then add target/jnicompressions-0.1.0.jar to your classpath

Test

Run performance test in your own environment:
$mvn test

These test results are very sensitive and have very big error ranges, 1-3% error range is very common, and even bigger for large numbers(speed>3000M/s), so don't take small differences too serious.

Here are some test results on my new MacbookPro: (I keep the old test results, to show how hardware, library version impact on performance)

New Test environment: (Lz4 r68 & Snappy 1.0.5)
Processor Name: Intel Core i7
Processor Speed:  2.4 GHz
Number of Processors: 1
Total Number of Cores:  4
L2 Cache (per Core):  256 KB
L3 Cache: 6 MB
Memory: 8 GB
gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00)

Running com.github.decster.jnicompressions.BlockCompressionTest
com.github.decster.jnicompressions.Lz4Compression: block size: 64K, dataSize/file: 67108864
Direct                   Total: Compress:  525M/s Decompress:   766M/s( 1592M/s) ratio: 48.1%
ByteArray                Total: Compress:  525M/s Decompress:   768M/s( 1596M/s) ratio: 48.1%
Simple                   Total: Compress:  495M/s Decompress:   586M/s( 1218M/s) ratio: 48.1%
com.github.decster.jnicompressions.SnappyCompression: block size: 64K, dataSize/file: 67108864
Direct                   Total: Compress:  473M/s Decompress:   424M/s(  873M/s) ratio: 48.6%
ByteArray                Total: Compress:  474M/s Decompress:   426M/s(  877M/s) ratio: 48.6%
Simple                   Total: Compress:  447M/s Decompress:   361M/s(  744M/s) ratio: 48.6%

Old Test environment: (lz4 r51 & Snappy 1.0.4)
Processor Name: Intel Core i5
Processor Speed:  2.3 GHz
Number of Processors: 1
Total Number of Cores:  2
L2 Cache (per Core):  256 KB
L3 Cache: 3 MB
Memory: 4 GB
gcc version 4.2.1 (Apple Inc. build 5659)

Running com.github.decster.jnicompressions.BlockCompressionTest
com.github.decster.jnicompressions.Lz4Compression: block size: 64K, dataSize/file: 67108864
Direct                   Total: Compress:  433M/s Decompress:   651M/s( 1354M/s) ratio: 48.1%
ByteArray                Total: Compress:  417M/s Decompress:   583M/s( 1212M/s) ratio: 48.1%
Simple                   Total: Compress:  401M/s Decompress:   468M/s(  972M/s) ratio: 48.1%
com.github.decster.jnicompressions.SnappyCompression: block size: 64K, dataSize/file: 67108864
Direct                   Total: Compress:  402M/s Decompress:   495M/s( 1019M/s) ratio: 48.6%
ByteArray                Total: Compress:  390M/s Decompress:   450M/s(  927M/s) ratio: 48.6%
Simple                   Total: Compress:  373M/s Decompress:   383M/s(  788M/s) ratio: 48.6%

Changes

v0.1.1

Using GetPrimitiveArrayCritical instead of malloc & copy, thanks to Adrien Grand for the advice. Update LZ4 to r68 Upate Snappy to 1.0.5

v0.1.0 initial version