The snappy-java is a Java port of the snappy http://code.google.com/p/snappy/, a fast C++ compresser/decompresser developed by Google.
- Fast compression/decompression around 200~400MB/sec.
- Less memory usage. SnappyOutputStream uses only 32KB+ in default.
- JNI-based implementation to achieve comparable performance to the native C++ version.
- Although snappy-java uses JNI, it can be used safely with multiple class loaders (e.g. Tomcat, etc.).
- Compression/decompression of Java primitive arrays (
- Portable across various operating systems; Snappy-java contains native libraries built for Window/Mac/Linux (64-bit). snappy-java loads one of these libraries according to your machine environment (It looks system properties,
- Simple usage. Add the snappy-java-(version).jar file to your classpath. Then call compression/decompression methods in
- Framing-format support (Since 1.1.0 version)
- OSGi support
- Apache License Version 2.0. Free for both commercial and non-commercial use.
Snappy's main target is very high-speed compression/decompression with reasonable compression size. So the compression ratio of snappy-java is modest and about the same as
LZF(ranging 20%-100% according to the dataset).
- The benchmark result indicates snappy-java is the fastest compreesor/decompressor in Java: http://ning.github.com/jvm-compressor-benchmark/results/canterbury-roundtrip-2011-07-28/index.html
- The decompression speed is twice as fast as the others: http://ning.github.com/jvm-compressor-benchmark/results/canterbury-uncompress-2011-07-28/index.html
The current stable version is available from here:
- Release version: http://central.maven.org/maven2/org/xerial/snappy/snappy-java/
- Snapshot version (the latest beta version): https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/
Using with Maven
- Snappy-java is available from Maven's central repository: http://central.maven.org/maven2/org/xerial/snappy/snappy-java
Add the following dependency to your pom.xml:
<dependency> <groupId>org.xerial.snappy</groupId> <artifactId>snappy-java</artifactId> <version>1.1.2</version> <type>jar</type> <scope>compile</scope> </dependency>
Using with sbt
libraryDependencies += "org.xerial.snappy" % "snappy-java" % "1.1.2"
org.xerial.snapy.Snappy in your Java code:
String input = "Hello snappy-java! Snappy-java is a JNI-based wrapper of " + "Snappy, a fast compresser/decompresser."; byte compressed = Snappy.compress(input.getBytes("UTF-8")); byte uncompressed = Snappy.uncompress(compressed); String result = new String(uncompressed, "UTF-8"); System.out.println(result);
In addition, high-level methods (
Snappy.compress(float ..) etc. ) and low-level ones (e.g.
Snappy.rawUncompress(..), etc.), which minimize memory copies, can be used.
SnappyInputStream are also available for reading/writing large data sets.
SnappyFramedInputStream can be used for the framing format.
- See also Javadoc API
[magic header:16 bytes]([block size:int32][compressed data:byte array])*format. You can read the result of
SnappyInputStream, but you cannot read the compressed data generated by
Snappy.uncompress. Here is the compatibility matrix of data foramt:
If you have snappy-java-(VERSION).jar in the current directory, use
-classpath option as follows:
$ javac -classpath ".;snappy-java-(VERSION).jar" Sample.java # in Windows or $ javac -classpath ".:snappy-java-(VERSION).jar" Sample.java # in Mac or Linux
Public discussion group
Post bug reports or feature request to the Issue Tracker: https://github.com/xerial/snappy-java/issues
Public discussion forum is here: [Xerial Public Discussion Group)[http://groups.google.com/group/xerial?hl=en]
Building from the source code
See the installation instruction. Building from the source code is an option when your OS platform and CPU architecture is not supported. To build snappy-java, you need Git, JDK (1.6 or higher), g++ compiler (mingw in Windows) etc.
$ git clone https://github.com/xerial/snappy-java.git $ cd snappy-java $ make
When building on Solaris use
target/snappy-java-$(version).jar is the product additionally containing the native library built for your platform.
Building Linux x86_64 binary
snappy-java tries to static link libstdc++ to increase the availability for various Linux versions. However, standard distributions of 64-bit Linux OS rarely provide libstdc++ compiled with
-fPIC option. I currently uses custom g++, compiled as follows:
$ cd work $ wget (gcc-4.8.3 source) $ tar xvfz (gcc-4.8.3.tar.gz) $ cd gcc-4.8.3 $ ./contrib/download_prerequisites $ cd .. $ mkdir objdir $ cd objdir $ ../gcc-4.8.3/configure --prefix=$HOME/local/gcc-4.8.3 CXXFLAGS=-fPIC CFLAGS=-fPIC --enable-languages=c,c++ $ make $ make install
This g++ build enables static linking of libstdc++. For more infomation on building GCC, see GCC's home page.
Building Linux s390/s390x binaries
Older snapshots of snappy contain a buggy config.h.in that does not work properly on some big-endian platforms like Linux on IBM z (s390/s390x). Building snappy-java on s390/s390x requires fetching the snappy source from GitHub, and processing the source with autoconf to obtain a usable config.h. On a RHEL s390x system, these steps produced a working 64-bit snappy-java build (the process should be similar for other distributions):
$ sudo yum install java-1.7.1-ibm-devel libstdc++-static-devel $ export JAVA_HOME=/usr/lib/jvm/java-1.7.1-ibm-18.104.22.168.10-1jpp.3.el7_0.s390x $ make USE_GIT=1 GIT_REPO_URL=https://github.com/google/snappy.git GIT_SNAPPY_BRANCH=master IBM_JDK_7=1
Cross-compiling for other platforms
The Makefile contains rules for cross-compiling the native library for other platforms so that the snappy-java JAR can support multiple platforms. For example, to build the native libraries for x86 Linux, x86 and x86-64 Windows, and soft- and hard-float ARM:
$ make linux32 win32 win64 linux-arm linux-armhf
If you append
snappy to the line above, it will also build the native library for the current platform and then build the snappy-java JAR (containing all native libraries built so far).
Of course, you must first have the necessary cross-compilers and development libraries installed for each target CPU and OS. For example, on Ubuntu 12.04 for x86-64, install the following packages for each target:
sudo apt-get install g++-multilib libc6-dev-i386 lib32stdc++6
sudo apt-get install g++-mingw-w64-i686
sudo apt-get install g++-mingw-w64-x86-64
sudo apt-get install g++-arm-linux-gnueabi
sudo apt-get install g++-arm-linux-gnueabihf
Unfortunately, cross-compiling for Mac OS X is not currently possible; you must compile within OS X.
If you are using Mac and openjdk7 (or higher), use the following option:
$ make native LIBNAME=libsnappyjava.dylib
snappy-java uses sbt (simple build tool for Scala) as a build tool. Here is a simple usage
$ ./sbt # enter sbt console > ~test # run tests upon source code change > ~test-only * # run tests that matches a given name pattern > publishM2 # publish jar to $HOME/.m2/repository > package # create jar file > findbugs # Produce findbugs report in target/findbugs > jacoco:cover # Report the code coverage of tests to target/jacoco folder
If you need to see detailed debug messages, launch sbt with
$ ./sbt -Dloglevel=debug
For the details of sbt usage, see my blog post: Building Java Projects with sbt
Using snappy-java with Tomcat 6 (or higher) Web Server
Simply put the snappy-java's jar to WEB-INF/lib folder of your web application. Usual JNI-library specific problem no longer exists since snappy-java version 1.0.3 or higher can be loaded by multiple class loaders.