jraft-example benchmark client hang #410

JerryYangSH · 2020-03-30T02:00:50Z

Describe the bug

When I run jraft-example test, it hangs at BenchmarkClient.main#rebalance, and the 'startBenchmark' test never starts.

The hang stack is below:
"com.alipay.sofa.jraft.benchmark.BenchmarkBootstrap.main()" #12 prio=5 os_prio=0 tid=0x00007f4110d24000 nid=0x613d waiting on condition [0x00007f40f8d86000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.alipay.sofa.jraft.rhea.client.pd.AbstractPlacementDriverClient.getLeader(AbstractPlacementDriverClient.java:273)
at com.alipay.sofa.jraft.rhea.client.pd.AbstractPlacementDriverClient.getLeader(AbstractPlacementDriverClient.java:237)
at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.rebalance(BenchmarkClient.java:183)
at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.main(BenchmarkClient.java:84)
at com.alipay.sofa.jraft.benchmark.BenchmarkBootstrap.main(BenchmarkBootstrap.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)

Expected behavior

The client should continue to startBenchmark.

Actual behavior

It blocks at rebalance in main.

Steps to reproduce

Run 3 servers of example on single node, and a client with below maven commands.

mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.server.BenchmarkServer -Dexec.args="server 127.0.0.1:18091,127.0.0.1:18092,127.0.0.1:18093 jraft-example/config/benchmark_server1.yaml"

mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.server.BenchmarkServer -Dexec.args="server 127.0.0.1:18091,127.0.0.1:18092,127.0.0.1:18093 jraft-example/config/benchmark_server2.yaml"

mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.server.BenchmarkServer -Dexec.args="server 127.0.0.1:18091,127.0.0.1:18092,127.0.0.1:18093 jraft-example/config/benchmark_server3.yaml"

mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.client.BenchmarkClient -Dexec.args="client 127.0.0.1:18091,127.0.0.1:18092,127.0.0.1:18093 jraft-example/config/benchmark_client.yaml 1 50 50 1024"

Minimal yet complete reproducer code (or GitHub URL to code)

https://github.com/sofastack/sofa-jraft/blob/f6d2e8053aa0e8e894ef18a988f4ea9812c36f95/jraft-example/src/main/java/com/alipay/sofa/jraft/benchmark/client/BenchmarkClient.java

Environment

SOFAJRaft version: the latest master
JVM version (e.g. java -version): 1.8.0_231

jerry@u19:~/opensources/sofa-jraft$ java -version
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.library.path=/usr/local/lib
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

OS version (e.g. uname -a): Linux u19 5.0.0-38-generic Redis protocol compatibility layer based on RheaKV #41-Ubuntu SMP Tue Dec 3 00:27:35 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Maven version: Apache Maven 3.6.0
IDE version: No IDEA. Linux bash

The text was updated successfully, but these errors were encountered:

fengjiachun · 2020-03-30T03:36:10Z

Whether you have modified the source code？ The stack and source code do not match！

JerryYangSH · 2020-03-30T05:04:02Z

yes, but it doesn't matter. I can revert my changes... I found the reason is "initialServerList" was not added into benchmark_client.yaml, although it's provided in argument list. but still after "initialServerList" was added into yaml conf, the client still hits InterruptedException and exit.

Below is the environment based on the master.

admin@murray-rust:~/opensources/sofa-jraft-OFFICIAL-1.3.0> git status
On branch master
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   jraft-example/config/benchmark_client.yaml
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        jraft-example/config/benchmark_server1.yaml
        jraft-example/config/benchmark_server2.yaml
        jraft-example/config/benchmark_server3.yaml
        rhea_benchmark_raft_data/
================================
admin@murray-rust:~/opensources/sofa-jraft-OFFICIAL-1.3.0> git log | head
commit 94c2ece97ca1e09a05ef19a3ee45ec3a1265a9be
Author: dennis zhuang <killme2008@gmail.com>
Date:   Wed Mar 4 12:20:32 2020 +0800
    Feature/segment storage improvement (#399)
    * (fix) RocksDBSegmentLogStorage can't be read when sync is false
    * (feat) Impl concurrent writing for segment log storage
=======================================
diff --git a/jraft-example/config/benchmark_client.yaml b/jraft-example/config/benchmark_client.yaml
index 1052898..b85b3ab 100644
--- a/jraft-example/config/benchmark_client.yaml
+++ b/jraft-example/config/benchmark_client.yaml
@@ -39,4 +39,4 @@ rpcOptions:
 failoverRetries: 0
 useParallelKVExecutor: false
-
+initialServerList: 127.0.0.1:18091,127.0.0.1:18092,127.0.0.1:18094
====================
admin@murray-rust:~/opensources/sofa-jraft-OFFICIAL-1.3.0> diff jraft-example/config/benchmark_server.yaml jraft-example/config/benchmark_server1.yaml
15a16
>     ip: 127.0.0.1
51c52
<
---
> initialServerList: 127.0.0.1:18091,127.0.0.1:18092,127.0.0.1:18093
==============================================

JerryYangSH · 2020-03-30T05:07:01Z

<cont>
2020-03-30 04:57:59 [com.alipay.sofa.jraft.benchmark.BenchmarkBootstrap.main()] INFO  BenchmarkClient:93 - Start benchmark...
2020-03-30 04:57:59 [get_batching #0] ERROR LogExceptionHandler:65 - Handle get_batching disruptor event error, event is null
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_241]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_241]
        at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) ~[disruptor-3.3.7.jar:?]
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56) ~[disruptor-3.3.7.jar:?]
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) [disruptor-3.3.7.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
2020-03-30 04:57:59 [put_batching #0] ERROR LogExceptionHandler:65 - Handle put_batching disruptor event error, event is com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore$KVEvent@17bc7580
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_241]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_241]
        at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) ~[disruptor-3.3.7.jar:?]
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56) ~[disruptor-3.3.7.jar:?]
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) [disruptor-3.3.7.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
2020-03-30 04:57:59 [get_batching_only_safe #0] ERROR LogExceptionHandler:65 - Handle get_batching_only_safe disruptor event error, event is com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore$KeyEvent@217988f3
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_241]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_241]
        at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) ~[disruptor-3.3.7.jar:?]
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56) ~[disruptor-3.3.7.jar:?]
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) [disruptor-3.3.7.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
2020-03-30 04:57:59 [Thread-28] ERROR BenchmarkClient:121 - Wrong slidingWindow: java.util.concurrent.Semaphore@228f2e87[Permits = 99], java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.doRequest(BenchmarkClient.java:119)
        at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.lambda$startBenchmark$0(BenchmarkClient.java:100)
        at java.lang.Thread.run(Thread.java:748)

Mar 30, 2020 4:57:59 AM com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 9 org.apache.logging.log4j.core.async.AsyncLoggerConfigHelper$Log4jEventWrapper@53da0ce9
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
        at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[WARNING]
java.lang.RuntimeException: java.lang.InterruptedException
    at com.lmax.disruptor.FatalExceptionHandler.handleEventException (FatalExceptionHandler.java:45)
    at com.lmax.disruptor.BatchEventProcessor.run (BatchEventProcessor.java:156)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait (AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await (AbstractQueuedSynchronizer.java:2048)
    at com.lmax.disruptor.BlockingWaitStrategy.waitFor (BlockingWaitStrategy.java:45)
    at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor (ProcessingSequenceBarrier.java:56)
    at com.lmax.disruptor.BatchEventProcessor.run (BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)

fengjiachun · 2020-03-30T05:09:37Z

I found the reason is "initialServerList" was not added into benchmark_client.yaml,

initialServerList need to be added in main method's args[1]

    public static void main(final String[] args) {
        if (args.length < 7) {
            LOG.error("Args: [initialServerList], [configPath], [threads], [writeRatio], [readRatio], [valueSize] are needed.");
            System.exit(-1);
        }

JerryYangSH · 2020-03-30T05:12:29Z

I found the reason is "initialServerList" was not added into benchmark_client.yaml,

initialServerList need to be added in main method's args[1]
    public static void main(final String[] args) {
        if (args.length < 7) {
            LOG.error("Args: [initialServerList], [configPath], [threads], [writeRatio], [readRatio], [valueSize] are needed.");
            System.exit(-1);
        }

yes, I added. see mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.client.BenchmarkClient -Dexec.args="client 127.0.0.1:18091,127.0.0.1:18092,127.0.0.1:18093 jraft-example/config/benchmark_client.yaml 1 50 50 1024"

JerryYangSH · 2020-03-30T05:16:40Z

It looks slidingWindow error

2020-03-30 04:57:59 [Thread-28] ERROR BenchmarkClient:121 - Wrong slidingWindow: java.util.concurrent.Semaphore@228f2e87[Permits = 99], java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.doRequest(BenchmarkClient.java:119)
        at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.lambda$startBenchmark$0(BenchmarkClient.java:100)
        at java.lang.Thread.run(Thread.java:748)

fengjiachun · 2020-03-30T07:48:21Z

It looks slidingWindow error

2020-03-30 04:57:59 [Thread-28] ERROR BenchmarkClient:121 - Wrong slidingWindow: java.util.concurrent.Semaphore@228f2e87[Permits = 99], java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.doRequest(BenchmarkClient.java:119)
        at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.lambda$startBenchmark$0(BenchmarkClient.java:100)
        at java.lang.Thread.run(Thread.java:748)

Maybe need to set benchmark thread daemon false setDaemon(false)

JerryYangSH · 2020-03-30T10:21:26Z

it makes no difference after setting Daemon(false).

[WARNING]
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema
    at java.util.concurrent.CompletableFuture.reportGet (CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get (CompletableFuture.java:1928)
    at com.alipay.sofa.jraft.rhea.client.FutureHelper.get (FutureHelper.java:46)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.bPut (DefaultRheaKVStore.java:1147)
    at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.main (BenchmarkClient.java:86)
    at com.alipay.sofa.jraft.benchmark.BenchmarkBootstrap.main (BenchmarkBootstrap.java:40)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema
    at com.alipay.sofa.jraft.rhea.serialization.impl.protostuff.ProtoStuffSerializer.writeObject (ProtoStuffSerializer.java:85)
    at com.alipay.sofa.jraft.rhea.rpc.ProtostuffSerializer.serialize (ProtostuffSerializer.java:41)
    at com.alipay.remoting.rpc.protocol.RpcRequestCommand.serializeContent (RpcRequestCommand.java:129)
    at com.alipay.remoting.rpc.RpcCommand.serialize (RpcCommand.java:105)
    at com.alipay.remoting.rpc.RpcRemoting.toRemotingCommand (RpcRemoting.java:354)
    at com.alipay.remoting.rpc.RpcRemoting.invokeWithCallback (RpcRemoting.java:305)
    at com.alipay.remoting.rpc.RpcClientRemoting.invokeWithCallback (RpcClientRemoting.java:89)
    at com.alipay.remoting.rpc.RpcRemoting.invokeWithCallback (RpcRemoting.java:269)
    at com.alipay.remoting.rpc.RpcClient.invokeWithCallback (RpcClient.java:327)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVRpcService.internalCallAsyncWithRpc (DefaultRheaKVRpcService.java:152)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVRpcService.callAsyncWithRpc (DefaultRheaKVRpcService.java:102)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVRpcService.callAsyncWithRpc (DefaultRheaKVRpcService.java:93)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.internalPut (DefaultRheaKVStore.java:1188)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.put (DefaultRheaKVStore.java:1166)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.access$1300 (DefaultRheaKVStore.java:147)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore$PutBatchingHandler.onEvent (DefaultRheaKVStore.java:2274)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore$PutBatchingHandler.onEvent (DefaultRheaKVStore.java:2254)
    at com.lmax.disruptor.BatchEventProcessor.run (BatchEventProcessor.java:137)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.protostuff.runtime.RuntimeSchema
    at java.net.URLClassLoader.findClass (URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:351)
    at com.alipay.sofa.jraft.rhea.serialization.impl.protostuff.ProtoStuffSerializer.writeObject (ProtoStuffSerializer.java:85)
    at com.alipay.sofa.jraft.rhea.rpc.ProtostuffSerializer.serialize (ProtostuffSerializer.java:41)
    at com.alipay.remoting.rpc.protocol.RpcRequestCommand.serializeContent (RpcRequestCommand.java:129)
    at com.alipay.remoting.rpc.RpcCommand.serialize (RpcCommand.java:105)
    at com.alipay.remoting.rpc.RpcRemoting.toRemotingCommand (RpcRemoting.java:354)
    at com.alipay.remoting.rpc.RpcRemoting.invokeWithCallback (RpcRemoting.java:305)
    at com.alipay.remoting.rpc.RpcClientRemoting.invokeWithCallback (RpcClientRemoting.java:89)
    at com.alipay.remoting.rpc.RpcRemoting.invokeWithCallback (RpcRemoting.java:269)
    at com.alipay.remoting.rpc.RpcClient.invokeWithCallback (RpcClient.java:327)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVRpcService.internalCallAsyncWithRpc (DefaultRheaKVRpcService.java:152)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVRpcService.callAsyncWithRpc (DefaultRheaKVRpcService.java:102)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVRpcService.callAsyncWithRpc (DefaultRheaKVRpcService.java:93)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.internalPut (DefaultRheaKVStore.java:1188)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.put (DefaultRheaKVStore.java:1166)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.access$1300 (DefaultRheaKVStore.java:147)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore$PutBatchingHandler.onEvent (DefaultRheaKVStore.java:2274)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore$PutBatchingHandler.onEvent (DefaultRheaKVStore.java:2254)
    at com.lmax.disruptor.BatchEventProcessor.run (BatchEventProcessor.java:137)
    at java.lang.Thread.run (Thread.java:748)
2020-03-30 10:15:02 [composite_batching #0] ERROR LogExceptionHandler:65 - Handle composite_batching disruptor event error, event is null
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_241]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_241]
        at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) ~[disruptor-3.3.7.jar:?]
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56) ~[disruptor-3.3.7.jar:?]

fengjiachun · 2020-03-30T10:51:32Z

I tested it and found no problems

server

$ mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.server.BenchmarkServer -Dexec.args="server 10.15.232.112:18091 jraft-example/config/benchmark_server.yaml"

client

$ mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.client.BenchmarkClient -Dexec.args="client 10.15.232.112:18091 jraft-example/config/benchmark_client.yaml 1 50 50 16"

JerryYangSH · 2020-03-30T11:42:55Z

do you know how to fix below NoClassDefFoundError error?

java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema
    at java.util.concurrent.CompletableFuture.reportGet (CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get (CompletableFuture.java:1928)
    at com.alipay.sofa.jraft.rhea.client.FutureHelper.get (FutureHelper.java:46)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.bPut (DefaultRheaKVStore.java:1147)
    at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.main (BenchmarkClient.java:86)
    at com.alipay.sofa.jraft.benchmark.BenchmarkBootstrap.main (BenchmarkBootstrap.java:40)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema
    at com.alipay.sofa.jraft.rhea.serialization.impl.protostuff.ProtoStuffSerializer.writeObject (ProtoStuffSerializer.java:85)
    at com.alipay.sofa.jraft.rhea.rpc.ProtostuffSerializer.serialize (ProtostuffSerializer.java:41)
    at com.alipay.remoting.rpc.protocol.RpcRequestCommand.serializeContent (RpcRequestCommand.java:129)

**The jar is there at compile time, not sure why it can not be found during runtime:**
admin@murray-rust:~/opensources/sofa-jraft-OFFICIAL-1.3.0> ls /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/* -ltr
-rw-r--r-- 1 admin users   2026 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.pom
-rw-r--r-- 1 admin users     40 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.pom.sha1
-rw-r--r-- 1 admin users 433300 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.jar
-rw-r--r-- 1 admin users     40 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.jar.sha1
-rw-r--r-- 1 admin users    216 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/_remote.repositories

JerryYangSH · 2020-03-30T12:48:54Z

I tested it and found no problems

server

$ mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.server.BenchmarkServer -Dexec.args="server 10.15.232.112:18091 jraft-example/config/benchmark_server.yaml"

client

$ mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.client.BenchmarkClient -Dexec.args="client 10.15.232.112:18091 jraft-example/config/benchmark_client.yaml 1 50 50 16"

Are you able to run 3-node test?

fengjiachun · 2020-03-30T13:01:46Z

do you know how to fix below NoClassDefFoundError error?

java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema
    at java.util.concurrent.CompletableFuture.reportGet (CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get (CompletableFuture.java:1928)
    at com.alipay.sofa.jraft.rhea.client.FutureHelper.get (FutureHelper.java:46)
    at com.alipay.sofa.jraft.rhea.client.DefaultRheaKVStore.bPut (DefaultRheaKVStore.java:1147)
    at com.alipay.sofa.jraft.benchmark.client.BenchmarkClient.main (BenchmarkClient.java:86)
    at com.alipay.sofa.jraft.benchmark.BenchmarkBootstrap.main (BenchmarkBootstrap.java:40)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema
    at com.alipay.sofa.jraft.rhea.serialization.impl.protostuff.ProtoStuffSerializer.writeObject (ProtoStuffSerializer.java:85)
    at com.alipay.sofa.jraft.rhea.rpc.ProtostuffSerializer.serialize (ProtostuffSerializer.java:41)
    at com.alipay.remoting.rpc.protocol.RpcRequestCommand.serializeContent (RpcRequestCommand.java:129)

**The jar is there at compile time, not sure why it can not be found during runtime:**
admin@murray-rust:~/opensources/sofa-jraft-OFFICIAL-1.3.0> ls /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/* -ltr
-rw-r--r-- 1 admin users   2026 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.pom
-rw-r--r-- 1 admin users     40 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.pom.sha1
-rw-r--r-- 1 admin users 433300 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.jar
-rw-r--r-- 1 admin users     40 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.jar.sha1
-rw-r--r-- 1 admin users    216 Mar 21 00:51 /home/admin/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/_remote.repositories

 $mvn dependency:tree -Dincludes=*protostuff*

fengjiachun · 2020-03-30T13:04:44Z

I tested it and found no problems
server

$ mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.server.BenchmarkServer -Dexec.args="server 10.15.232.112:18091 jraft-example/config/benchmark_server.yaml"

client

$ mvn -pl jraft-example exec:java -Dexec.mainClass=com.alipay.sofa.jraft.benchmark.client.BenchmarkClient -Dexec.args="client 10.15.232.112:18091 jraft-example/config/benchmark_client.yaml 1 50 50 16"

Are you able to run 3-node test?

This is not directly related to 3 or 1 nodes

JerryYangSH · 2020-03-30T13:42:35Z

After I pull the code from latest master, the problem is gone! Thanks!

JerryYangSH · 2020-03-31T04:21:25Z

This still drives me crazy. When I merge master into my branch, it still fails at "NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema". Not sure what's wrong.

dependency tree show the same result as your above paste.
Caused by: java.lang.NoClassDefFoundError: io/protostuff/runtime/RuntimeSchema
   at com.alipay.sofa.jraft.rhea.serialization.impl.protostuff.ProtoStuffSerializer.writeObject (ProtoStuffSerializer.java:85)
   at com.alipay.sofa.jraft.rhea.rpc.ProtostuffSerializer.serialize (ProtostuffSerializer.java:41)

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ jraft-example ---
[INFO] com.alipay.sofa:jraft-example:jar:1.3.0
[INFO] \- com.alipay.sofa:jraft-rheakv-core:jar:1.3.0:compile
[INFO]    +- io.protostuff:protostuff-core:jar:1.6.0:compile
[INFO]    |  \- io.protostuff:protostuff-api:jar:1.6.0:compile
[INFO]    \- io.protostuff:protostuff-runtime:jar:1.6.0:compile
[INFO]       \- io.protostuff:protostuff-collectionschema:jar:1.6.0:compile
[INFO]

killme2008 · 2020-03-31T04:22:23Z

Maybe you have two or more protobuf jars in your classpath, exclude one please.

JerryYangSH · 2020-03-31T05:17:35Z

you mean protobuf or protostuff? RuntimeSchema is from jraft-example/target/jraft-bin/lib/protostuff-runtime-1.6.0.jar...


jerry@u19:~/opensources/sofa-jraft$ grep -r RuntimeSchema ~/.m2
Binary file /home/jerry/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.jar matches
jerry@u19:~/opensources/sofa-jraft$ md5sum jraft-example/target/jraft-bin/lib/protostuff-runtime-1.6.0.jar
19056311a2c5132afd9442cc02aef2bc  jraft-example/target/jraft-bin/lib/protostuff-runtime-1.6.0.jar
jerry@u19:~/opensources/sofa-jraft$ md5sum /home/jerry/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.jar
19056311a2c5132afd9442cc02aef2bc  /home/jerry/.m2/repository/io/protostuff/protostuff-runtime/1.6.0/protostuff-runtime-1.6.0.jar
-----------------------

jerry@u19:~/opensources/sofa-jraft$ find . -name *protobuf*.jar
./jraft-example/target/jraft-bin/lib/protobuf-java-3.5.1.jar
jerry@u19:~/opensources/sofa-jraft$ find ~ -name *protobuf*.jar
/home/jerry/.m2/repository/com/google/protobuf/protobuf-java/3.5.1/protobuf-java-3.5.1.jar
/home/jerry/.m2/repository/io/grpc/grpc-protobuf/1.17.0/grpc-protobuf-1.17.0.jar
/home/jerry/.m2/repository/io/grpc/grpc-protobuf-lite/1.17.0/grpc-protobuf-lite-1.17.0.jar
/home/jerry/opensources/sofa-jraft/jraft-example/target/jraft-bin/lib/protobuf-java-3.5.1.jar
/home/jerry/opensources/protobuf-2.5.0/java/target/protobuf-java-2.5.0.jar

JerryYangSH · 2020-03-31T05:41:23Z

Finally I explicitely added protobuf dependency into jraft-example/pom.xml, now it works!

<dependency>
            <groupId>io.protostuff</groupId>
            <artifactId>protostuff-core</artifactId>
            <version>${protostuff.version}</version>
        </dependency>
        <dependency>
            <groupId>io.protostuff</groupId>
            <artifactId>protostuff-runtime</artifactId>
            <version>${protostuff.version}</version>
        </dependency>

fengjiachun closed this as completed Apr 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jraft-example benchmark client hang #410

jraft-example benchmark client hang #410

JerryYangSH commented Mar 30, 2020 •

edited

fengjiachun commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020 •

edited

JerryYangSH commented Mar 30, 2020 •

edited

fengjiachun commented Mar 30, 2020 •

edited

JerryYangSH commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020

fengjiachun commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020

fengjiachun commented Mar 30, 2020 •

edited

JerryYangSH commented Mar 30, 2020 •

edited

JerryYangSH commented Mar 30, 2020

fengjiachun commented Mar 30, 2020

fengjiachun commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020

JerryYangSH commented Mar 31, 2020

killme2008 commented Mar 31, 2020

JerryYangSH commented Mar 31, 2020

JerryYangSH commented Mar 31, 2020

jraft-example benchmark client hang #410

jraft-example benchmark client hang #410

Comments

JerryYangSH commented Mar 30, 2020 • edited

Describe the bug

Expected behavior

Actual behavior

Steps to reproduce

Minimal yet complete reproducer code (or GitHub URL to code)

Environment

fengjiachun commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020 • edited

JerryYangSH commented Mar 30, 2020 • edited

fengjiachun commented Mar 30, 2020 • edited

JerryYangSH commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020

fengjiachun commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020

fengjiachun commented Mar 30, 2020 • edited

JerryYangSH commented Mar 30, 2020 • edited

JerryYangSH commented Mar 30, 2020

fengjiachun commented Mar 30, 2020

fengjiachun commented Mar 30, 2020

JerryYangSH commented Mar 30, 2020

JerryYangSH commented Mar 31, 2020

killme2008 commented Mar 31, 2020

JerryYangSH commented Mar 31, 2020

JerryYangSH commented Mar 31, 2020

JerryYangSH commented Mar 30, 2020 •

edited

JerryYangSH commented Mar 30, 2020 •

edited

JerryYangSH commented Mar 30, 2020 •

edited

fengjiachun commented Mar 30, 2020 •

edited

fengjiachun commented Mar 30, 2020 •

edited

JerryYangSH commented Mar 30, 2020 •

edited