Skip to content

Home

Jump to bottom
rockeet edited this page Sep 4, 2018 · 79 revisions

中文 Chinese

TerarkDB Documentation

TerarkDB = terark rocksdb + terark-zip-rocksdb

terark-zip-rocksdb is for RocksDB's SSTable implementation: TerarkZipTable.

TerarkZipTable use terark's searchable compression technology and achieved all of these:
        (1) much higher random read performance
        (2) much higher compression ratio (disk file is much smaller)
        (3) much lower memory usage (mmap on compressed file, no double caching)

We forked official rocksdb as terark rocksdb and changed for better support TerarkZipTable(in terark-zip-rocksdb).

Now by using TerarkDB, you are able to store more data on disk (3x+ more than snappy, 2x+ more than zstd), and load more data into memory(10x+ more), and improve read performance greatly! All data are accessed at memory speed!

Notes

1. Compile and Download

1.1 Compile terark rocksdb by yourself

  git clone https://github.com/terark/terarkdb.git
  cd  terarkdb
  git submodule init
  git submodule update
  make -C rocksdb shared_lib DEBUG_LEVEL=0

1.2 Download precompiled terark-zip-rocksdb

Remember:

  • You can not compile terark-zip-rocksdb by yourself
  • Without terark-zip-rocksdb, terark rocksdb works exactly the same as official rocksdb

The download file names like:

  • terark-zip-rocksdb-Linux-x86_64-g++-5.4-bmi2-1.tgz
       -- bmi2-1 means this software can only run on intel-haswell or newer CPU
  • terark-zip-rocksdb-Linux-x86_64-g++-5.4-bmi2-0.tgz
       -- bmi2-0 means this software can run on older CPU (but the CPU must support popcnt & SSE4.2)

The downloaded package contains all dependencies you need as below:

  $ cd terark-zip-rocksdb-Linux-x86_64-g++-4.8-bmi2-1/lib
  # just show filename and symbol links

  $ ls -l libterark*-r.so # "-r" for "release version"
  libterark-core-g++-4.8-r.so
  libterark-core-r.so -> libterark-core-g++-4.8-r.so
  libterark-fsa-g++-4.8-r.so
  libterark-fsa-r.so -> libterark-fsa-g++-4.8-r.so
  libterark-zbs-g++-4.8-r.so
  libterark-zbs-r.so -> libterark-zbs-g++-4.8-r.so
  libterark-zip-rocksdb-g++-4.8-r.so
  libterark-zip-rocksdb-r.so -> libterark-zip-rocksdb-g++-4.8-r.so

  $ ls -l libterark*-d.so # "-d" for "debug version"
  libterark-core-d.so -> libterark-core-g++-4.8-d.so
  libterark-core-g++-4.8-d.so
  libterark-fsa-d.so -> libterark-fsa-g++-4.8-d.so
  libterark-fsa-g++-4.8-d.so
  libterark-zbs-d.so -> libterark-zbs-g++-4.8-d.so
  libterark-zbs-g++-4.8-d.so
  libterark-zip-rocksdb-d.so -> libterark-zip-rocksdb-g++-4.8-d.so
  libterark-zip-rocksdb-g++-4.8-d.so

2. Now, you can try TerarkDB

3. Restrictions

  • User Key Comparator is not supported, you should encode your keys in byte lexical order
  • EnvOptions::use_mmap_reads must be true, can be set by DBOptions::allow_mmap_reads

4. Tuning Guide

5. License

This software is open source.

5.1 submodule rocksdb

submodule rocksdb is our fork of rocksdb and can be compiled by yourself.

License of submodule rocksdb is same as offical rocksdb(BSD clause 3).

5.2 submodule terark-zip-rocksdb

submodule terark-zip-rocksdb implements an MemTable and an SSTable for submodule rocksdb, terark-zip-rocksdb license is Apache 2.0, with NOTES:

  • You can read or redistribute or use the source code under Apache 2.0 license
  • You can not compile this software by yourself, since this software depends on our proprietary core algorithms, which requires a commercial license
  • You can download the precompiled binary library of this software

6. Contacts

Terark.com

Table of Contents

Clone this wiki locally