New features of cluster scalability and multi-raft #3191

fanhualta · 2021-05-14T04:11:14Z

This pr adds two new features:

cluster scalability
multi-raft

Design docs(currently only in Chinese):

multi-raft: https://cwiki.apache.org/confluence/display/IOTDB/Multi-Raft-For-One-DataGroup+Design+Doc
cluster scalability: https://cwiki.apache.org/confluence/display/IOTDB/Cluster+scalability+design+doc

…raft

…remove local data

2. The two-stage relative order problem of double logs is solved.

1. LogPlan serialize and deserialize bug 2. add shell scripts to remove nodes 3. enrich the function of node tool 4. fix some bugs of adding new nodes

1. SlotPartitionTable serialization null pointer 2. AddNodeLog and RemoveNodeLog serialization issue

1. remove meta leader lead to new leader can not commit remove log until next meta log, which is fixed by empty content log 2. reorganized the logic of get member 3. meta leader should notify the target node to exile when removing

…d syncServiceMap when removing a node.

OneSizeFitsQuorum

Stage 3.
BTW, it seems that there are some failed tests

cluster/src/main/java/org/apache/iotdb/cluster/coordinator/Coordinator.java

cluster/src/main/java/org/apache/iotdb/cluster/client/sync/SyncClientAdaptor.java

cluster/src/main/java/org/apache/iotdb/cluster/log/logtypes/AddNodeLog.java

cluster/src/main/java/org/apache/iotdb/cluster/log/snapshot/FileSnapshot.java

cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java

cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java

cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java

1. remove uselss methods. 2. apply log when handling add/remove node in data group

OneSizeFitsQuorum · 2021-06-05T00:29:53Z

My review has almost been completed. Next, please do two things:

Fix CI
Add some user guide in https://github.com/apache/iotdb/tree/master/docs/zh/UserGuide/Cluster &&https://github.com/apache/iotdb/tree/master/docs/UserGuide/Cluster and add some examples for add/remove node operation

fanhualta · 2021-06-06T16:48:06Z

Please add some user guide in https://github.com/apache/iotdb/tree/master/docs/zh/UserGuide/Cluster &&https://github.com/apache/iotdb/tree/master/docs/UserGuide/Cluster and add some examples for add/remove node operation

I have added docs already.

OneSizeFitsQuorum · 2021-06-06T17:19:09Z

I have added docs already.

Nice~Please pay attention to the timeout CI and fix them

cluster/src/assembly/resources/sbin/remove-node.bat

cluster/src/assembly/resources/sbin/remove-node.sh

cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java

cluster/src/main/java/org/apache/iotdb/cluster/coordinator/Coordinator.java

cluster/src/main/java/org/apache/iotdb/cluster/query/ClusterPlanRouter.java

cluster/src/main/java/org/apache/iotdb/cluster/coordinator/Coordinator.java

cluster/src/main/java/org/apache/iotdb/cluster/log/applier/MetaLogApplier.java

…will lead to wrong result of query desc

sonarcloud · 2021-06-27T11:57:59Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
139 Code Smells

50.5% Coverage
3.1% Duplication

OneSizeFitsQuorum and others added 30 commits December 11, 2020 19:49

init

55fc053

add multi-raft except for add/remove node

d5c075d

keep raft leader in one node && balance data to all raft group

7ee34da

merge master

1aea5a7

merge master

2487976

fix sync schema bug

128e191

Merge branch 'fix_sync_schema_bug' into cluster_multi_raft

2a2e1cd

fix a bug of cal MaxDeduplicatedPathNum

5119aa7

Merge branch 'fix_MaxDeduplicatedPathNum_cal_bug' into cluster_multi_…

d3cca11

…raft

Merge branch 'master' into cluster_multi_raft

72d697a

fix a bug of multi-raft

c4a9dcd

merge master

fdc8d32

assign slots

ae1b6f3

fix a bug

25ea701

merge master

c0c5dfb

fix a bug of forward plan

b717e3f

fix some issues of multi-raft

109d432

fix bugs of wrong previous groups, pull snapshot from self and wrong …

325e4f5

…remove local data

Reimplement the function of adding and removing nodes

452cba3

1. fix ut tests

327eb7e

2. The two-stage relative order problem of double logs is solved.

merge master

f342899

fix a bug of TSStatus.OK set redirect

f8a1432

merge master

e7f8e28

merge master

dd64f0e

merge master

e5e4411

merge master

124190c

This pr fix following bugs:

4609bc0

1. LogPlan serialize and deserialize bug 2. add shell scripts to remove nodes 3. enrich the function of node tool 4. fix some bugs of adding new nodes

This commit fix following issues:

d84f0ff

1. SlotPartitionTable serialization null pointer 2. AddNodeLog and RemoveNodeLog serialization issue

This commit fix following issues:

8c7f80b

1. remove meta leader lead to new leader can not commit remove log until next meta log, which is fixed by empty content log 2. reorganized the logic of get member 3. meta leader should notify the target node to exile when removing

This commit fix a serious bug of abnormal cache of asyncServiceMap an…

4d5ca80

…d syncServiceMap when removing a node.

OneSizeFitsQuorum reviewed Jun 4, 2021

View reviewed changes

fix some issues according pr comments:

292cf23

1. remove uselss methods. 2. apply log when handling add/remove node in data group

fanhualta requested a review from OneSizeFitsQuorum June 4, 2021 14:18

fanhualta added 2 commits June 7, 2021 00:08

merge master

a69eab7

add user guide for cluster scalability and multi-raft in cluster doc

ca0228d

fanhualta closed this Jun 6, 2021

fanhualta reopened this Jun 6, 2021

neuyilan requested changes Jun 7, 2021

View reviewed changes

fanhualta added 2 commits June 10, 2021 00:05

merge master and fix conflicts

1729a2c

fix issues according comments

f1e98e4

mychaow reviewed Jun 10, 2021

View reviewed changes

cluster/src/main/java/org/apache/iotdb/cluster/log/applier/MetaLogApplier.java Show resolved Hide resolved

fanhualta added 9 commits June 21, 2021 21:43

merge master

30478a6

merge master

5059827

fix bug of [IOTDB-1438], path not exists exception

7452091

fix a bug of NPE

f815fcc

fix a bug of wrong desc query result

b2b6ec7

modify NodeTool user to root

56f9dda

fix a serve bug of BatchData serialization and deserialization which …

2248ba1

…will lead to wrong result of query desc

merge master

dc7b26d

fix CI

6a38a46

fanhualta force-pushed the cluster_scalability branch from 300169a to 6a38a46 Compare June 27, 2021 10:55

qiaojialin approved these changes Jun 27, 2021

View reviewed changes

qiaojialin merged commit 00f2c30 into master Jun 27, 2021

fanhualta deleted the cluster_scalability branch June 27, 2021 14:30

OneSizeFitsQuorum restored the cluster_scalability branch July 20, 2021 01:16

HTHou deleted the cluster_scalability branch August 15, 2021 13:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New features of cluster scalability and multi-raft #3191

New features of cluster scalability and multi-raft #3191

fanhualta commented May 14, 2021 •

edited

OneSizeFitsQuorum left a comment •

edited

OneSizeFitsQuorum commented Jun 5, 2021

fanhualta commented Jun 6, 2021

OneSizeFitsQuorum commented Jun 6, 2021 •

edited

sonarcloud bot commented Jun 27, 2021

New features of cluster scalability and multi-raft #3191

New features of cluster scalability and multi-raft #3191

Conversation

fanhualta commented May 14, 2021 • edited

OneSizeFitsQuorum left a comment • edited

Choose a reason for hiding this comment

OneSizeFitsQuorum commented Jun 5, 2021

fanhualta commented Jun 6, 2021

OneSizeFitsQuorum commented Jun 6, 2021 • edited

sonarcloud bot commented Jun 27, 2021

fanhualta commented May 14, 2021 •

edited

OneSizeFitsQuorum left a comment •

edited

OneSizeFitsQuorum commented Jun 6, 2021 •

edited