[jni 5.18.3] java.lang.UnsatisfiedLinkError: org.rocksdb.LRUCache.newLRUCache(JIZD)J

[ChengGit]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Hi friends,

I am on MacOS 10.14.6, kafka:2.4.0, which depends on rocksdbjni:5.18.3, and I bumped into this error when kafka streams tried to create a RocksDBConfigSetter instance during bootstrap:

java.lang.UnsatisfiedLinkError: org.rocksdb.LRUCache.newLRUCache(JIZD)J
 	at org.rocksdb.LRUCache.newLRUCache(Native Method)
 	at org.rocksdb.LRUCache.<init>(LRUCache.java:74)

I use this RocksDBConfigSetter from kafka official document: https://kafka.apache.org/24/documentation/streams/developer-guide/memory-mgmt.html#rocksdb

and it throws error on this line:

private static org.rocksdb.Cache cache = new org.rocksdb.LRUCache(TOTAL_OFF_HEAP_MEMORY, -1, false, INDEX_FILTER_BLOCK_RATIO);

Could this be a libc link issue? Could you shed some light on it?

Expected behavior

I would expect LRUCache be initialised and hen kafka-streams application can bootstrap.

Actual behavior

I see a UnsatisfiedLinkError, and it seems failed to link to a native method newLRUCache

[adamretter]

@ChengGit Have you called RocksDB.loadLibrary();?

[ChengGit]

Hi @adamretter , no I didn't explicitly call that method, should I do it for verification?

[adamretter]

@ChengGit that would seem like a good idea.

[ChengGit]

Thanks @adamretter , I tried to add a constructor public ConstomRocksDBConfig() and put RocksDB.loadLibrary(); inside.
But still I have to remove static modifier for both Cache and WriterBufferManager in order to make it run. Otherwise it still fail on this line

private final static org.rocksdb.Cache cache = new org.rocksdb.LRUCache(BLOCK_CACHE_SIZE, -1, false, INDEX_FILTER_BLOCK_RATIO);

here is my config code (with static modifier removed for cache and writeBufferManager)

   public static class CustomRocksDBConfig implements RocksDBConfigSetter {
        private static final double INDEX_FILTER_BLOCK_RATIO = 1.0D; // 1 percent highPrioPoolRatio
        private static final long BLOCK_CACHE_SIZE = 16 * 1024 * 1024L;
        private static final long BLOCK_SIZE = 16 * 1024; // 16 KB;
        private static final int N_MEMTABLES = 2;
        private static final long MEMTABLE_SIZE = 4 * 1024 * 1024; // 4MB
        private final org.rocksdb.Cache cache = new org.rocksdb.LRUCache(BLOCK_CACHE_SIZE, -1, false, INDEX_FILTER_BLOCK_RATIO);
        private final org.rocksdb.WriteBufferManager writeBufferManager = new org.rocksdb.WriteBufferManager(MEMTABLE_SIZE, cache);

        public CustomRocksDBConfig() {
            System.out.println("\n\n\n\n\n\n\n\n\n\n\n");
            RocksDB.loadLibrary();    
        }

        @Override
        public void setConfig(final String storeName, final Options options, final Map<String, Object> configs) { ..... }

I wanted to keep static modifier as I wanted to have a single cache for all rocksdb instances, in order to control overall memory usage.

[adamretter]

@ChengGit Instead you should just put:

static {
  RocksDB.loadLibrary();
}

Inside the class declaration of the first class you use that interacts with RocksDB.

[ChengGit]

@adamretter many thanks for your patience! It runs with no error - I put the static code block inside my main class where CustomRocksDBConfig class was referenced

kafkaStreamsProperties.put(
"rocksdb.config.setter", "package.to.my.MainClass$CustomRocksDBConfig"
)

Does it imply all native libraries are packaged?

[adamretter]

Does it imply all native libraries are packaged?

I am not sure what you mean, and I can't really comment on how Kafka works.

I think we have solved your issue, so I am going to close the ticket. Feel free to re-open if it isn't right

[ChengGit]

Thanks @adamretter, yeah I confirm adding static block to where this config class was firstly referenced did the trick.

I am not sure what you mean, and I can't really comment on how Kafka works.

Please nevermind that - I added information of unrelated context to this thread.

May I have one more question on this though, what does RocksDB.loadLibrary() do in this case? Is it expected to add this static code block to make RocksDBConfigSetter loaded for 5.18.3?

[adamretter]

RocksDB.loadLibrary() do in this case?

It extracts the native library from the jar file and add loads it into memory.