Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Region migration improvement #12165

Merged
merged 27 commits into from
Mar 26, 2024
Merged

Region migration improvement #12165

merged 27 commits into from
Mar 26, 2024

Conversation

liyuheng55555
Copy link
Contributor

@liyuheng55555 liyuheng55555 commented Mar 13, 2024 •
edited

liyuheng55555 and others added 11 commits March 1, 2024 16:13
@liyuheng55555 @OneSizeFitsQuorum
init IT frame, and select coordinator deterministically. include belo…
15f80ef 
…w commits:

IT pass: select coordinator deterministically

confignode RegionMigrateProcedure stage finished breakpoint

select coordinator deterministically

IT improve

adding stopForcibly

IT file

function improve

confignode RegionMigrateProcedure stage finished breakpoint

add function

IT basic frame done

save
@liyuheng55555 @OneSizeFitsQuorum
split RegionMigrateProcedure into AddRegionPeerProcedure and RemoveRe…
e7ea7bd 
…gionPeerProcedure, include below commits:

split RegionMigrateProcedure: delete useless code

split procedure: RemoveRegionPeer successfully split, IT pass

split procedure: AddRegionPeer successfully split, IT pass

split procedure: ready

split procedure: ready

split procedure: have some problem
@BUAAserein
Split configuration and add resetPeerList for region migration (apach…
e6d0e4d 
…e#12101)
@SzyWilliam
[To region_migration] Refactor resetPeer in IConsensus (apache#12124)
5314136
@liyuheng55555
[To region_migration] cn pull from dn, IT improve, coordinator avoid …
b7be36c 
…submit duplicate (apache#12127)
@liyuheng55555
part done, some functional problem
f0c4d39
@liyuheng55555
cn crash won't cause problem (some IT still cannot pass)
7aa5f5e
@liyuheng55555
remove logBreakPoint
7bc4311
@liyuheng55555
merge master and resolve conflicts
535f860 
# Conflicts:
#	integration-test/src/main/java/org/apache/iotdb/it/env/cluster/env/AbstractEnv.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlan.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ConfigManager.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/IManager.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/executor/ConfigPlanExecutor.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/AddRegionPeerProcedure.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/RemoveRegionPeerProcedure.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/store/ProcedureFactory.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/thrift/ConfigNodeRPCServiceProcessor.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/client/ConfigNodeClient.java
#	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/FileUtils.java
#	iotdb-protocol/thrift-datanode/src/main/thrift/datanode.thrift
@liyuheng55555
Merge branch 'master' into Working/region_migration_clear2
e5c1646 
# Conflicts:
#	integration-test/src/main/java/org/apache/iotdb/it/env/cluster/env/AbstractEnv.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlan.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ConfigManager.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/IManager.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/executor/ConfigPlanExecutor.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/AddRegionPeerProcedure.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/RemoveRegionPeerProcedure.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/store/ProcedureFactory.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/thrift/ConfigNodeRPCServiceProcessor.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/client/ConfigNodeClient.java
#	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/FileUtils.java
#	iotdb-protocol/thrift-datanode/src/main/thrift/datanode.thrift
@liyuheng55555
resolve conflict
6ef5c9e
liyuheng55555
liyuheng55555 commented Mar 13, 2024
View reviewed changes
integration-test/src/main/java/org/apache/iotdb/it/env/cluster/node/AbstractNodeWrapper.java Outdated
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This optimization will merge in another pr

@liyuheng55555
self review
c0c3ca1
@liyuheng55555
Merge branch 'Working/region_migration_clear' into Working/region_mig…
4f269b9 
…ration_clear2

# Conflicts:
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/conf/ConfigNodeConstant.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlan.java
#	iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlanSerDeTest.java
@liyuheng55555
spotless
6e10b6a
@liyuheng55555
self review
4618455
@liyuheng55555
ut file
2769ec1
@liyuheng55555
Merge branch 'master' into Working/region_migration_clear2
a5bcb5d 
# Conflicts:
#	integration-test/src/main/java/org/apache/iotdb/it/env/cluster/node/AbstractNodeWrapper.java
@liyuheng55555
remove
5a4c645 
spotless
@liyuheng55555 liyuheng55555 force-pushed the Working/region_migration_clear2 branch from d89f815 to 5a4c645 Compare March 15, 2024 08:22
OneSizeFitsQuorum
OneSizeFitsQuorum reviewed Mar 15, 2024
View reviewed changes
...ation-test/src/test/java/org/apache/iotdb/confignode/it/IoTDBRegionMigrateReliabilityIT.java Outdated Show resolved Hide resolved
...node/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlanType.java Show resolved Hide resolved
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java Outdated Show resolved Hide resolved
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java Outdated Show resolved Hide resolved
...ignode/src/main/java/org/apache/iotdb/confignode/manager/load/service/StatisticsService.java
regionPriorityEntry.getValue().getRight().getDataNodeLocations().stream()
.map(TDataNodeLocation::getDataNodeId)
.collect(Collectors.toList()));
} catch (Exception e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CRZbulabula PTAL

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why surrounding with try-catch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why surrounding with try-catch?

because NPE @CRZbulabula

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My mistake!!! I should fix this bug these days >_<

iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java Outdated Show resolved Hide resolved
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java Show resolved Hide resolved
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/RegionMigrateService.java Show resolved Hide resolved
iotdb-protocol/thrift-commons/src/main/thrift/common.thrift Outdated Show resolved Hide resolved
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/DataNodeKillPoints.java Show resolved Hide resolved
CRZbulabula
CRZbulabula requested changes Mar 18, 2024
View reviewed changes
Copy link
Contributor

@CRZbulabula CRZbulabula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL~

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java Outdated Show resolved Hide resolved
...ignode/src/main/java/org/apache/iotdb/confignode/manager/load/service/StatisticsService.java
regionPriorityEntry.getValue().getRight().getDataNodeLocations().stream()
.map(TDataNodeLocation::getDataNodeId)
.collect(Collectors.toList()));
} catch (Exception e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why surrounding with try-catch?

.../src/main/java/org/apache/iotdb/confignode/procedure/impl/region/AddRegionPeerProcedure.java Show resolved Hide resolved
integration-test/src/main/java/org/apache/iotdb/it/env/cluster/env/AbstractEnv.java Show resolved Hide resolved
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/DataNodeKillPoints.java Show resolved Hide resolved
@liyuheng55555
as Tan reviewed 1
66205a3
@liyuheng55555
as Tan reviewed 1
33a5294
@liyuheng55555
as Tan reviewed
c605f80
@liyuheng55555
as review
ae357c8
@liyuheng55555
Merge branch 'master' into Working/region_migration_clear2
98c3a30 
# Conflicts:
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java
@liyuheng55555
spotless
a54ae88
@liyuheng55555
Merge branch 'master' into Working/region_migration_clear2
f82f487 
# Conflicts:
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java
CRZbulabula
CRZbulabula approved these changes Mar 25, 2024
View reviewed changes
Copy link
Contributor

@CRZbulabula CRZbulabula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

OneSizeFitsQuorum
OneSizeFitsQuorum approved these changes Mar 26, 2024
View reviewed changes
Copy link
Contributor

@OneSizeFitsQuorum OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@OneSizeFitsQuorum OneSizeFitsQuorum merged commit a5df27c into apache:master Mar 26, 2024
30 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@OneSizeFitsQuorum OneSizeFitsQuorum OneSizeFitsQuorum approved these changes

@CRZbulabula CRZbulabula CRZbulabula approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@liyuheng55555 @OneSizeFitsQuorum @CRZbulabula @BUAAserein @SzyWilliam