Intermittent SIGSEGV in RocksDB jnilib

[agarman]

Expected behavior

mvn test completes successfully in about 10-15 minutes.

Actual behavior

The JVM blew up:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000014afb3b80, pid=77149, tid=0x0000000000059b0b
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [librocksdbjni2577203176943896148.jnilib+0x10b80]  Java_org_rocksdb_RocksDB_getLongProperty__JLjava_lang_String_2I+0x50
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/agarman/src/pulsar/pulsar-broker/hs_err_pid77149.log
2017-03-01 22:40:03,065 - INFO  - [main:TestListener@29] - %%%%%%%%%%%% Before configuration - com.yahoo.pulsar.broker.loadbalance.LoadBalancerTest / setup -- attrs: []
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/bin/sh: line 1: 77149 Abort trap: 6           /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/bin/java -Xmx2G -XX:MaxDirectMemorySize=8G -Dio.netty.leakDetectionLevel=advanced -jar /Users/agarman/src/pulsar/pulsar-broker/target/surefire/surefirebooter1269992674620327305.jar /Users/agarman/src/pulsar/pulsar-broker/target/surefire/surefire2614902326644746145tmp /Users/agarman/src/pulsar/pulsar-broker/target/surefire/surefire_7380098213255121419tmp

Steps to reproduce

Intermittent error.

System configuration

Pulsar version: 17630b9

[merlimat]

This looks like a problem in the JNI wrapper of RocksDB.. even if we were using the API in the wrong way it shouldn't blow up the JVM.

I haven't seen this error locally or on Travis yet. Can you try by updating rocksdbjni dependency to latest version (5.1.2)?

We want to upgrade RocksDB anyway to take advantage of some new features introduced lately.

I think that to fix the new version, it should be enough to add the new dependency in the <dependencyManagement> section of the top level pom.xml:

<dependency>
    <groupId>org.rocksdb</groupId>
    <artifactId>rocksdbjni</artifactId>
    <version>5.1.2</version>
</dependency>

[agarman]

The error has only occurred once. If I don't see it again for a few weeks, I'll close this issue (or of course you can).

[sijie]

the rocksdb version has been upgraded to latest rocksdb stable release. so close the issue for now. if it occurs, please open a new issue.