SidePlugin Java Binding

[中文版 (Chinese)]

SidePlugin Java Binding requires minimal effort

SidePlugin's Java Binding only needs to bind SidePluginRepo, and only needs to bind very few methods (only the declarations are listed here):

class SidePluginRepo extends RocksObject {
    public SidePluginRepo(); // constructor
    public void importAutoFile(String fname) throws RocksDBException;
    public RocksDB openDB(String js) throws RocksDBException;
    public RocksDB openDB(String js, List<ColumnFamilyHandle> out_cfhs);
    public RocksDB openDB() throws RocksDBException;
    public RocksDB openDB(List<ColumnFamilyHandle> out_cfhs) throws RocksDBException;
    public void startHttpServer() throws RocksDBException;
    public void closeHttpServer();
    public void closeAllDB(); // conform to C++ native name CloseAllDB
    public void close(); // synonym to closeAllDB
    public void put(String name, String spec, Options opt); // rarely used
    public void put(String name, String spec, DBOptions dbo);  // rarely used
    public void put(String name, String spec, ColumnFamilyOptions cfo); // rarely used
}

With such a small workload, I have never touched java for many years, and I have never used jni. It only took less than a day to realize this Binding, learn and sell now, from beginning to end. Therefore, a good architecture design can greatly reduce the workload. According to the architecture of SidePlugin, the code in rocksdbjava (about 40,000 lines of java code and c++) can be reduced by at least 90%.

Refer to the C++ version 101, the Java version is not much different; then refer to the actual example SideGetBenchmarks, I believe you can easily call ToplingDB using java.

The put method is for compatibility with existing code: if the existing code has made some settings for various options, and these settings cannot be configured through the configuration file (for example, the java version of the CompactionFilter is customized), the option put first to SidePluginRepo, and then call importAutoFile -- importAutoFile will fill in the missing options in the put option, and override the option of the same name!

Run SideGetBenchmarks

Step 1: Compile and install rocksdb

git clone https://github.com/topling/toplingdb.git --depth 1
cd toplingdb
git submodule update --init --recursive
ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts # silent git
make -j`nproc` DEBUG_LEVEL=0 shared_lib
sudo make install-shared PREFIX=/opt

Step 2: Compile rocksdbjava

make rocksdbjava -j`nproc` DEBUG_LEVEL=0

Step 3: Add rocksdbjava to maven repository

You should now be at the root of the toplingdb repository

cd java/target
cp rocksdbjni-7.10.0-linux64.jar rocksdbjni-7.10.0-SNAPSHOT-linux64.jar
mvn install:install-file -Dfile=rocksdbjni-7.10.0-SNAPSHOT-linux64.jar -DgroupId=org.rocksdb -DartifactId=rocksdbjni -Dversion=7.10.0-SNAPSHOT -Dpackaging=jar

Step 4: Compile SideGetBenchmarks

Assuming you are now at the root of the toplingdb repository

cd java/jmh
mvn clean package

Step 5: Run SideGetBenchmarks

Assuming you are now in toplingdb/java/jmh

mkdir -p /dev/shm/db_bench_enterprise
cp ../../sideplugin/rockside/src/topling/web/{style.css,index.html} /dev/shm/db_bench_enterprise
export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH # for libterark-*
export LD_PRELOAD=libterark-zbs-g++-12.1-r.so:libterark-fsa-g++-12.1-r.so:libjemalloc.so
java -jar target/rocksdbjni-jmh-1.0-SNAPSHOT-benchmarks.jar \
     -p keyCount=1000 -p keySize=128 -p valueSize=32768 \
     -p sideConf=../../sideplugin/rockside/sample-conf/db_bench_enterprise.yaml SideGetBenchmarks

After running, you can visit http://127.0.0.1:2011 to observe the db status.

Note 1: Community Edition users please replace db_bench_enterprise.yaml with db_bench_community.yaml, both files are in toplingdb/sideplugin/rockside/sample-conf, sideplugin/rockside is a submodule of ToplingDB.

Explanation 2: LD_PRELOAD is set because toplingdb and jemalloc dynamic libraries use init exec type TLS (thread local storage), which has the fastest access speed, but dynamic libraries using this TLS must be preloaded. Or explicitly linked by the executable. The jni loaded by java is dynamically loaded by the jvm. If LD_PRELOAD is not used, this kind of TLS initialization in the dynamic library will fail.

limitation

Currently, components implemented in Java cannot be registered in SidePlugin’s plug-in library with Java code. Generally, this is not a serious problem. The only problem is that Distributed Compaction cannot be used.

For example, a custom CompactionFilter/MergeOperator is implemented in Java. Because it cannot be registered in the SidePlugin plug-in library, it cannot preload the corresponding dynamic library for dcompact_worker to run distributed compact. Unless (1) implement a special dynamic library, call back the Java code through jni to register the corresponding components in SidePlugin, and compile java into a dynamic library of local code. Or (2) Use C++ to implement a plug-in with the same function and register it in SidePlugin. For example, the data is in protobuf format. It is not difficult to implement the corresponding plug-in in C++, and the performance is even higher.