Region migration related work by liyuheng55555 · Pull Request #12293 · apache/iotdb

liyuheng55555 · 2024-04-03T10:22:06Z

No description provided.

…resetPeerList

…yuheng55555/iotdb into Working/region-migration-0401

liyuheng55555

PTAL~ @BUAAserein

liyuheng55555 · 2024-04-07T03:51:43Z

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensus.java

+  @Override
+  public List<ConsensusGroupId> getAllConsensusGroupIdsFromDisk() {
+    List<ConsensusGroupId> consensusGroupIds = new ArrayList<>();
+    try (DirectoryStream<Path> stream = Files.newDirectoryStream(storageDir.toPath())) {
+      for (Path path : stream) {
+        String[] items = path.getFileName().toString().split("_");
+        ConsensusGroupId consensusGroupId =
+            ConsensusGroupId.Factory.create(Integer.parseInt(items[0]), Integer.parseInt(items[1]));
+        consensusGroupIds.add(consensusGroupId);
+      }
+    } catch (IOException e) {
+      logger.error("Failed to get all consensus group ids from disk", e);
+    }
+    return consensusGroupIds;
+  }
+


@BUAAserein Maybe write an UT for this function ? Test the cooperation between getAllConsensusGroupIdsFromDisk and IoTConsensus.buildPeerDir .

This logic is the same as IoTConsensus, so may no need to add UT

This logic is the same as IoTConsensus, so may no need to add UT

@BUAAserein I still believe it is necessary, as UT are an effective tools to ensure the logic consistency between two functions. This can be exemplified by the numerous serialize-deserialize UTs in IoTDB, such as NodeInfoTest.testSnapshot.

Yes, we add parsingAndConstructIDTest

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensus.java

…resetPeerList

…yuheng55555/iotdb into Working/region-migration-0401

liyuheng55555

PTAL~ @BUAAserein

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java

liyuheng55555 · 2024-04-10T03:40:26Z

.../main/java/org/apache/iotdb/db/consensus/statemachine/dataregion/DataRegionStateMachine.java

-            + region.getDatabaseName()
-            + "-"
-            + region.getDataRegionId();
+    String snapshotDir = "null";


@BUAAserein seems not necessary to set to "null"

default is ""? or just use null

iotdb-client/jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBConnection.java

iotdb-client/jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBStatement.java

iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java

...onfignode/src/main/java/org/apache/iotdb/confignode/procedure/env/RegionMaintainHandler.java

.../src/main/java/org/apache/iotdb/confignode/procedure/impl/region/AddRegionPeerProcedure.java

OneSizeFitsQuorum · 2024-04-10T10:45:41Z

.../main/java/org/apache/iotdb/db/consensus/statemachine/dataregion/DataRegionStateMachine.java

-            + region.getDatabaseName()
-            + "-"
-            + region.getDataRegionId();
+    String snapshotDir = "null";


default is ""? or just use null

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/DataNode.java

iotdb-protocol/thrift-consensus/src/main/thrift/iotconsensus.thrift

OneSizeFitsQuorum · 2024-04-10T10:50:33Z

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/IConsensus.java

+   *
+   * @return consensusGroupId list
+   */
+  List<ConsensusGroupId> getAllConsensusGroupIdsFromDisk();


getAllConsensusGroupIdsWithoutStarting()

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/IConsensus.java

…yuheng55555/iotdb into Working/region-migration-0401

HxpSerein · 2024-04-12T03:17:54Z

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java

+                      filePath.getFileName().toString().endsWith(CONFIGURATION_TMP_FILE_NAME))
+              .collect(Collectors.toList());
+      for (Path filePath : paths) {
+        String formalPath =


if formalPath exsists, should delete first

HxpSerein · 2024-04-12T03:18:57Z

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java

-                  }
-                });
-      }
      serializeConfigurationAndFsyncToDisk();


Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>

…yuheng55555/iotdb into Working/region-migration-0401

Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>

OneSizeFitsQuorum

LGTM

Co-authored-by: BUAAserein <18376359@buaa.edu.cn>

liyuheng55555 and others added 11 commits April 1, 2024 15:18

add node.alive() && adjust check logic

61d1311

add IoTDBRegionMigrateOriginalCrashWhenRemoveRemotePeerIT

da8c475

pass some original dn crash IT

2129a97

fix cn crash IT

8ed4be1

pass IoTDBRegionMigrateCoordinatorCrashWhenRemoveRemotePeerIT && use …

1c4ec7e

…resetPeerList

pass datanode crash

ec1cbf5

add all nodes

8433e6a

call reset peer list on every dn

cfe6149

finish TODO

00434fd

Merge branch 'Working/region-migration-0401' of https://github.com/li…

eaf6cd0

…yuheng55555/iotdb into Working/region-migration-0401

enhance resetPeerList

31046bf

liyuheng55555 commented Apr 7, 2024

View reviewed changes

liyuheng55555 and others added 12 commits April 7, 2024 15:44

add node.alive() && adjust check logic

cc29e17

add IoTDBRegionMigrateOriginalCrashWhenRemoveRemotePeerIT

41a59dd

pass some original dn crash IT

1d5cabf

fix cn crash IT

3217cb0

pass IoTDBRegionMigrateCoordinatorCrashWhenRemoveRemotePeerIT && use …

a461881

…resetPeerList

pass datanode crash

513170c

add all nodes

c13063d

finish TODO

fca0749

call reset peer list on every dn

22c8010

enhance resetPeerList

77d8fb1

move file

c8949b4

self review

bf31c3b

liyuheng55555 force-pushed the Working/region-migration-0401 branch from 60aaa61 to bf31c3b Compare April 7, 2024 07:45

liyuheng55555 and others added 5 commits April 7, 2024 16:07

add guava dependency

e0bd88a

Merge branch 'Working/region-migration-0401' of https://github.com/li…

6d818ed

…yuheng55555/iotdb into Working/region-migration-0401

fix UT

82599ad

fix UT

c3bb501

add kill cluster IT, pass && add still writable check, XoX

342c965

fix problem

5e99732

liyuheng55555 force-pushed the Working/region-migration-0401 branch from 9511e26 to 5e99732 Compare April 9, 2024 06:20

liyuheng55555 added 3 commits April 9, 2024 18:14

fix some problems

491e7f3

fix IT

51f1675

self review

da3e067

liyuheng55555 commented Apr 10, 2024

View reviewed changes

wait until flush

64a04cf

OneSizeFitsQuorum reviewed Apr 10, 2024

View reviewed changes

liyuheng55555 and others added 6 commits April 11, 2024 11:46

as Tan reviewed

51493b9

finish ID <=> dir

e3eddf3

rename tmp file to formal file

c826d2d

choose readonly datanode as add peer coordinator

fcd338b

add parsingAndConstructIDTest

4e16345

Merge branch 'Working/region-migration-0401' of https://github.com/li…

e78572d

…yuheng55555/iotdb into Working/region-migration-0401

HxpSerein reviewed Apr 12, 2024

View reviewed changes

liyuheng55555 and others added 6 commits April 12, 2024 11:50

change deletion

51983c5

fix persist

776bdca

add status log

27a6681

Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>

Merge branch 'Working/region-migration-0401' of https://github.com/li…

fb94006

…yuheng55555/iotdb into Working/region-migration-0401

Merge branch 'Working/region-migration-0401' of https://github.com/li…

a69e7ff

…yuheng55555/iotdb into Working/region-migration-0401

spotless

be5b5b3

Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>

OneSizeFitsQuorum approved these changes Apr 12, 2024

View reviewed changes

OneSizeFitsQuorum merged commit f52d752 into apache:master Apr 12, 2024

OneSizeFitsQuorum deleted the Working/region-migration-0401 branch April 12, 2024 11:09

liyuheng55555 added a commit to liyuheng55555/iotdb that referenced this pull request Apr 15, 2024

Region migration related work (apache#12293)

8b6ae78

Co-authored-by: BUAAserein <18376359@buaa.edu.cn>

OneSizeFitsQuorum mentioned this pull request May 20, 2024

Fixed restart error when DataRegion is RatisConsensus #12554

Merged

HTHou pushed a commit that referenced this pull request Jun 26, 2024

Region migration related work (#12293)

e84d041

SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024

Region migration related work (apache#12293)

75996c0

Conversation

liyuheng55555 commented Apr 3, 2024

Uh oh!

liyuheng55555 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liyuheng55555 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants