Cluster Data Replication Problem v.0.12.0-SNAPSHOT

[proven-mozdemir]

Describe the bug
Cannot replicate data with IoTDB 0.12.0-SNAPSHOT. Lots of "TTransportException: null" exceptions seen on logs.

To Reproduce
Steps to reproduce the behavior:

1. Download source code

git clone https://github.com/apache/iotdb.git
mv iotdb iotdb-github

2. Compile with maven

cd iotdb-github/
./mvnw.sh clean package -DskipTests
cp cluster/target/cluster-0.12.0-SNAPSHOT.zip .

3. Extract zip from target into node1

unzip cluster-0.12.0-SNAPSHOT.zip
mkdir node1
mv lib node1/
mv sbin/ node1/
mv tools/ node1/
mv conf/ node1/

4. Copy node2 and node3 from node1

cp -rf node1/ node2
cp -rf node1/ node3

5. Change ports for node2 and node3

sed -i -e 's/6667/6668/g' node2/conf/iotdb-engine.properties
sed -i -e 's/6667/6669/g' node3/conf/iotdb-engine.properties
sed -i -e 's/31999/32000/g' node2/conf/iotdb-env.sh
sed -i -e 's/31999/32001/g' node3/conf/iotdb-env.sh

6. Start each nodes one by one in seperate terminals

cd node1/sbin
start-server.sh &

cd ../../node2/sbin
start-server.sh -internal_meta_port 9005 -internal_data_port 40012 -cluster_rpc_port 55561  &

cd ../../node3/sbin
start-server.sh -internal_meta_port 9007 -internal_data_port 40014 -cluster_rpc_port 55562  &

7. Download IoTDB Benchmark tool
   git clone https://github.com/thulab/iotdb-benchmark.git
8. Open with IDE (mine is IntelliJ Ultimate)
9. Add new maven module as iotdb-0.12
10. Copy all files from iotdb-0.11 module and replace version in pom file 0.12.0-SNAPSHOT and class package as "cn.edu.tsinghua.iotdb.benchmark.iotdb012"
11. Replace artifactId as iotdb-0.12 and name Benchmark IoTDB 0.12 in pom.xml
12. Replace dependency version as 0.12.0-SNAPSHOT for dependencies; iotdb-jdbc and iotdb-session
13. Add new version class names in cn.edu.tsinghua.iotdb.benchmark.conf.Constants class;

  public static final String IOTDB012_JDBC_CLASS = "cn.edu.tsinghua.iotdb.benchmark.iotdb012.IoTDB";
  public static final String IOTDB012_SESSION_CLASS = "cn.edu.tsinghua.iotdb.benchmark.iotdb012.IoTDBSession";

14. Add version 0.12 support in cn.edu.tsinghua.iotdb.benchmark.tsdb.DBFactory class;

            case "0.12.0":
              if(config.getINSERT_MODE().equals(Constants.INSERT_USE_JDBC)) {
                dbClass = Constants.IOTDB012_JDBC_CLASS;
              } else {
                dbClass = Constants.IOTDB012_SESSION_CLASS;
              }
              break;
            case "0.11.0":
            ....

15. Create dist-dependencies folder in iotdb-0.12 module directory
16. Create jdbc, service-rpc, session, thrift, tsfile folders into dist-dependencies folder
17. copy pom.xml and jar files from iotdb-github folder into jdbc, service-rpc, session, thrift, tsfile folders
18. Manually install SNAPSHOT builds into local maven repository;

cd iotdb-github/dist-dependencies
mvn install:install-file -Dpackaging=pom -Dfile=./pom.xml -DpomFile=./pom.xml
mvn install:install-file -Dfile=./jdbc/target/iotdb-jdbc-0.12.0-SNAPSHOT.jar -Dpackaging=jar -DpomFile=./jdbc/pom.xml
mvn install:install-file -Dfile=./session/target/iotdb-session-0.12.0-SNAPSHOT.jar -Dpackaging=jar -DpomFile=./session/pom.xml
mvn install:install-file -Dfile=./service-rpc/target/service-rpc-0.12.0-SNAPSHOT.jar -Dpackaging=jar -DpomFile=./service-rpc/pom.xml
mvn install:install-file -Dfile=./thrift/target/iotdb-thrift-0.12.0-SNAPSHOT.jar -Dpackaging=jar -DpomFile=./thrift/pom.xml
mvn install:install-file -Dfile=./tsfile/target/tsfile-0.12.0-SNAPSHOT.jar -Dpackaging=jar -DpomFile=./tsfile/pom.xml

19. Change iotdb-benchmark tool config.properties file as given below, other parameters are left as default;

HOST=<lan ip of your server>
PORT=6667
DB_SWITCH=IoTDB
DB_NAME=plant
BENCHMARK_WORK_MODE=testWithDefaultPath
IS_DELETE_DATA=true
#Whether to create the schema before writing
CREATE_SCHEMA=true
#Write data start time
START_TIME=2021-02-01T00:00:00+03:00
VERSION=0.12.0
OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0
INSERT_MODE=session

20. maven clean and install iotdb-benchmark tool
21. Execute test

Expected behavior
I expect data is replicated into all nodes;

cd iotdb-github
cli/sbin/start-cli.sh -p 6667
select s_0 from root.group_1.d_1 LIMIT 3;
<data exists>

cli/sbin/start-cli.sh -p 6668
select s_0 from root.group_1.d_1 LIMIT 3;
<NO data exists>

cli/sbin/start-cli.sh -p 6669
select s_0 from root.group_1.d_1 LIMIT 3;
<NO data exists>

I get some errors which given under "Additional Context" title
I found an FAQ document on this link; #2270
Translated it into English with Google Translate: https://translate.google.com/translate?sl=auto&tl=en&u=https://github.com/apache/iotdb/issues/2270
According to item one i changed port as 55560 in conf/config.properties file under iotdb-benchmark tool and tried to ingest again,
this time i got Connection Refused exception as given below;

11:39:08.999 [main] ERROR cn.edu.tsinghua.iotdb.benchmark.iotdb012.IoTDBSession - Failed to add session
org.apache.iotdb.rpc.IoTDBConnectionException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
	at org.apache.iotdb.session.SessionConnection.init(SessionConnection.java:91)
	at org.apache.iotdb.session.SessionConnection.<init>(SessionConnection.java:79)
	at org.apache.iotdb.session.Session.constructSessionConnection(Session.java:212)
	at org.apache.iotdb.session.Session.open(Session.java:181)
	at org.apache.iotdb.session.Session.open(Session.java:166)
	at cn.edu.tsinghua.iotdb.benchmark.iotdb012.IoTDBSession.<init>(IoTDBSession.java:38)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
	at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:124)
	at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:346)
	at java.base/java.lang.Class.newInstance(Class.java:604)
	at cn.edu.tsinghua.iotdb.benchmark.tsdb.DBFactory.getDatabase(DBFactory.java:74)
	at cn.edu.tsinghua.iotdb.benchmark.tsdb.DBWrapper.<init>(DBWrapper.java:33)
	at cn.edu.tsinghua.iotdb.benchmark.App.testWithDefaultPath(App.java:98)
	at cn.edu.tsinghua.iotdb.benchmark.App.main(App.java:68)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:87)
	at org.apache.iotdb.session.SessionConnection.init(SessionConnection.java:89)
	... 16 common frames omitted

Whenever i try to insert using Command Line interface as given below;

cd iotdb-github
cli/sbin/start-cli.sh -p 55560
insert into root.group_d1.d_1(timestamp,value) values(1612126801000,87.74);
# i get data for;
select * from root.group_d1.d_1;

And also i get data by connecting 55561 and 55562 port numbers.
Interestingly when i ingest more data only node1's data folder size is getting larger than node2 and node3 ones.
This means data is actually stored only under node1 data folder.

Ingest Replicated Data with IotDb benchmark tool
At the end of the day i am looking for the correct way to replicate ingest operation to other nodes of cluster with iotdb-benchmark tool.

Desktop (please complete the following information):

• OS: Centos Linux
• Version: 8 with kernel 4.18.0-240.10.1.el8_3.x86_64

Additional context
Exceptions seen in error log file:

2021-02-15 11:48:51,749 [pool-8-IoTDB-RPC-Client-thread-2] ERROR org.apache.thrift.server.TThreadPoolServer$WorkerProcess:315 - Thrift error occurred during processing of message. 
org.apache.thrift.transport.TTransportException: null
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:108)
	at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:103)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)

[neuyilan]

Thank you very much for your detailed description,

1. The default distributed RPC port is 55560. I see that you have found the answer to this question.
2. For cluster version testing, we generally use the support_ cluster branch[1] for testing, and the master is not used for the cluster version.

Interestingly when I ingest more data only node1's data folder size is getting larger than node2 and node3 ones.
This means data is actually stored only under the node1 data folder.

3. About node1's data folder size is getting bigger than node2 and node3. Because the default copy is 2 copies, and the data is partitioned according to storage group name + time, it is not possible to guarantee the same size of the data on the three nodes. You can use this tool[1] to look at data partitioning.

Lots of "TTransportException: null" exceptions seen on logs.

4. Yes, there are a large number of errors in the currently distributed version, but it does not affect the normal operation and can be ignored for the time being. However, we welcome to find out the essential cause of this problem.

[1] https://github.com/thulab/iotdb-benchmark/tree/support_cluster
[2] http://iotdb.apache.org/UserGuide/Master/System%20Tools/NodeTool.html

[neuyilan]

After today, the RPC ports of the cluster version is the same as those in the stand-alone version, namely 6667.