Permalink
..
Failed to load latest commit information.
src
.gitignore
README.md
pom.xml

README.md

util-mmap

util-mmap is Indeed's memory-mapping library for Java. It provides an efficient mechanism for accessing large files. Indeed's analytics platform Imhotep uses it for managing data access.

Installation

The latest util-mmap JAR file can be downloaded via Maven (link, link).

<dependencies>
    <dependency>
        <groupId>com.indeed</groupId>
        <artifactId>util-mmap</artifactId>
        <version>LATEST</version>
    </dependency>
    <!-- ... -->
</dependencies>

The library depends on Unix mmap for its underlying functionality, so it uses JNI. The published JAR file contains native builds for Linux i386, amd64, and OS X x86_64. The Java code in the library loads the correct Linux native shared object (.so, .dylib) file from the JAR file at runtime based on os.name and os.arch system properties. If you need to run on an unsupported OS, you'll need to rebuild the native code. See the instructions in Building.

Usage

A good starting point for using util-mmap is the MMapBuffer class. The following example maps in a large file containing an array of longs, written in little-endian order.

final MMapBuffer buffer = new MMapBuffer(
       filePath,
       FileChannel.MapMode.READ_ONLY,
       ByteOrder.LITTLE_ENDIAN);
final LongArray longArray =
    buffer.memory().longArray(0, buffer.memory().length() / 8);
final long firstValue = longArray.get(0);

Building

You can build the native code for util-mmap yourself with the provided Makefile, which are Linux/OS X-specific and depend on GCC:

$ cd util/mmap/src/main/native/

$ make

$ ls *.o *.so.*
com_indeed_util_mmap_MMapBuffer.o     com_indeed_util_mmap_Stat.o
com_indeed_util_mmap_NativeMemoryUtils.o  libindeedmmap.so.1.0.1

$ sudo make install

$ ls -l /usr/lib/libindeedmmap.so
lrwxrwxrwx 1 root root 66 Feb 25 12:52 /usr/lib/libindeedmmap.so -> 
  /home/user/util/mmap/src/main/native/libindeedmmap.so.1.0.1

The installation step assumes that /usr/lib/ is in your java.library.path. You can also repackage the shared library in the JAR file, or request that we distribute a new build binary in the JAR file by pushing it to the GitHub repository under src/main/resources/native/*platform*/.

Known limitations

The native code base has not been verified on other variants of Unix. For example, mremap is unsupported for Darwin (OS X). Contributions welcome.

License

Apache License Version 2.0

Bitdeli Badge