Demote local data member #24617

tommyk-gears · 2023-05-22T14:04:18Z

Introduces com.hazelcast.cluster.Cluster#demoteLocalDataMember method which transitions the local member from a data/normal member to a lite member. The most dramatic effect might be that a cluster can now transition from a state where it holds data to a state where it does not hold any data, but the cluster is still "alive" (i.e. when demoting the last data member).

The implementation is mostly a blend of member promotion and member shutdown.

Checklist:

Labels (Team:, Type:, Source:, Module:) and Milestone set
Label Add to Release Notes or Not Release Notes content set
Request reviewers if possible
Send backports/forwardports if fix needs to be applied to past/future releases
New public APIs have @Nonnull/@Nullable annotations
New public APIs have @since tags in Javadoc

Added a new method `Cluster.demoteLocalDataMember()` which demotes local member to a lite member if it's a normal data member.

…from demoted members

…zelcast into demote-local-data-member

…e member

devOpsHazelcast · 2023-05-22T14:04:22Z

All committers have signed the CLA.

devOpsHazelcast · 2023-05-22T14:04:22Z

Can one of the admins verify this patch?

devOpsHazelcast · 2023-05-22T14:04:22Z

Can one of the admins verify this patch?

vbekiaris · 2023-05-22T14:25:13Z

run-lab-run

vbekiaris

@tommyk-gears thanks a lot for the contribution. I am not sure we should allow removing all data members from the cluster after initial partition arrangement is done, but I didn't want to delay posting my comments further, so here they are. Let me know your thoughts around that.

thanks once more for the contribution!

hazelcast/src/main/java/com/hazelcast/cluster/Cluster.java

hazelcast/src/main/java/com/hazelcast/internal/cluster/impl/ClusterServiceImpl.java

hazelcast/src/main/java/com/hazelcast/internal/partition/impl/InternalPartitionServiceImpl.java

hazelcast/src/main/java/com/hazelcast/internal/partition/impl/PartitionStateGeneratorImpl.java

hazelcast/src/main/java/com/hazelcast/internal/cluster/impl/ClusterServiceImpl.java

hazelcast/src/main/java/com/hazelcast/internal/partition/operation/DemoteResponseOperation.java

hazelcast/src/test/java/com/hazelcast/internal/cluster/impl/DemoteDataMemberTest.java

Co-authored-by: Vassilis Bekiaris <vbekiaris@gmail.com>

…usterServiceImpl.java Co-authored-by: Vassilis Bekiaris <vbekiaris@gmail.com>

…InternalPartitionServiceImpl.java Co-authored-by: Vassilis Bekiaris <vbekiaris@gmail.com>

…lied master address

hazelcast/src/test/java/com/hazelcast/internal/cluster/impl/DemoteDataMemberTest.java

srknzl · 2023-09-07T10:55:05Z

run-lab-run

hz-devops-test · 2023-09-07T10:58:08Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder/hazelcast/src/main/java/com/hazelcast/internal/cluster/impl/MembersViewResponse.java:1: Line does not match expected header line of '/*'. [Header]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder/hazelcast/src/main/java/com/hazelcast/internal/partition/impl/InternalPartitionServiceImpl.java:614: Line is longer than 130 characters (found 137). [LineLength]
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.3.0:checkstyle (default) on project hazelcast: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There are 2 errors reported by Checkstyle 9.3 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder/checkstyle/checkstyle.xml ruleset. -> [Help 1]
--------------------------
[ERROR] 
--------------------------
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
--------------------------
[ERROR] 
--------------------------
[ERROR] For more information about the errors and possible solutions, please read the following articles:
--------------------------
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
--------------------------
[ERROR] 
--------------------------
[ERROR] After correcting the problems, you can resume the build with the command
--------------------------
[ERROR]   mvn  -rf :hazelcast
--------------------------

vbekiaris · 2023-09-27T09:12:37Z

run-lab-run

hz-devops-test · 2023-09-27T10:11:35Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

---------ERRORS-----------
--------------------------
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.899 s <<< FAILURE! -- in com.hazelcast.HazelcastCompletableFutureAsyncUsageTest
--------------------------
[ERROR] com.hazelcast.HazelcastCompletableFutureAsyncUsageTest.noClassUsesCompletableFuture -- Time elapsed: 5.898 s <<< FAILURE!
--------------------------
[ERROR] Failures: 
--------------------------
[ERROR]   HazelcastCompletableFutureAsyncUsageTest.noClassUsesCompletableFuture:40 Architecture Violation [Priority: MEDIUM] - Rule 'classes should use only CompletableFuture async methods with explicit executor service' was violated (1 times):
--------------------------
[ERROR] Tests run: 4574, Failures: 1, Errors: 0, Skipped: 17
--------------------------
[ERROR] There are test failures.
--------------------------
[ERROR] Tests run: 17, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 192.6 s <<< FAILURE! -- in com.hazelcast.jet.kinesis.KinesisIntegrationTest
--------------------------
[ERROR] com.hazelcast.jet.kinesis.KinesisIntegrationTest.dynamicStream_1Shard_splitsDuringData -- Time elapsed: 126.5 s <<< FAILURE!
--------------------------
[ERROR] Failures: 
--------------------------
[ERROR]   KinesisIntegrationTest.dynamicStream_1Shard_splitsDuringData:324->dynamicStream_splitsDuringData:354->assertMessages:637->AbstractKinesisTest.assertMessages:163->HazelcastTestSupport.assertTrueEventually:1269->HazelcastTestSupport.assertTrueEventually:1165->AbstractKinesisTest.lambda$assertMessages$3:180 Messages for key 243 differ!
--------------------------
[ERROR] Tests run: 26, Failures: 1, Errors: 0, Skipped: 3
--------------------------
[ERROR] There are test failures.
--------------------------
--------------------------
-------TEST FAILURE-------
--------------------------
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   HazelcastCompletableFutureAsyncUsageTest.noClassUsesCompletableFuture:40 Architecture Violation [Priority: MEDIUM] - Rule 'classes should use only CompletableFuture async methods with explicit executor service' was violated (1 times):
com.hazelcast.internal.partition.impl.InternalPartitionServiceImpl.checkClusterPartitionRuntimeStates():624 calls CompletableFuture.thenCompose
[INFO] 
[ERROR] Tests run: 4574, Failures: 1, Errors: 0, Skipped: 17
[INFO] 
[ERROR] There are test failures.

JamesHazelcast

Thanks for your PR @tommyk-gears, it's very well presented! Just a few minor comments from me, nothing major, although I do share Vassilis' concerns surrounding the demotion of the last member of a cluster to Lite (as discussed here).

Once we've reached some clarity on that (pending Vassilis' decision), I think we'll basically be ready for approval 👍

hazelcast/src/main/java/com/hazelcast/internal/partition/operation/DemoteRequestOperation.java

hazelcast/src/main/java/com/hazelcast/internal/partition/operation/DemoteResponseOperation.java

hazelcast/src/main/java/com/hazelcast/spi/properties/ClusterProperty.java

hazelcast/src/main/java/com/hazelcast/internal/partition/impl/MigrationManager.java

JamesHazelcast · 2023-09-28T09:08:49Z

Hi @tommyk-gears, I've discussed with Vassilis and we both agree that a Hazelcast cluster without any data members does not make much sense, so we think it's best if we throw an IllegalStateException when an attempt is made to demote the last data member of a cluster. Let us know what you think 👍

This PR is currently failing one of our archunit tests that enforces an executor to be defined explicitly for non-async method calls in CompletableFuture - so switching thenCompose(x ->x) to thenComposeAsync(x -> x, internalAsyncExecutor) should make it happy 😄

- Remove Versioned interface from Demote(Request|Response)Operation - Reorder properties in ClusterProperty to have newly added property last

…rying to demote the only data member

tommyk-gears · 2023-09-29T13:47:10Z

Hi @tommyk-gears, I've discussed with Vassilis and we both agree that a Hazelcast cluster without any data members does not make much sense, so we think it's best if we throw an IllegalStateException when an attempt is made to demote the last data member of a cluster. Let us know what you think 👍

This PR is currently failing one of our archunit tests that enforces an executor to be defined explicitly for non-async method calls in CompletableFuture - so switching thenCompose(x ->x) to thenComposeAsync(x -> x, internalAsyncExecutor) should make it happy 😄

Awesome, thanks for good feedback! I changed the implementation to throw IllegalStateException when trying to demote the last data member, and also reverted the code changes that were made to support demotion of last member.
Also fixed that thenCompose-issue.

JamesHazelcast · 2023-10-02T12:43:16Z

run-lab-run

hz-devops-test · 2023-10-02T12:46:15Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/hazelcast/src/main/java/com/hazelcast/internal/partition/operation/DemoteRequestOperation.java:31:8: Unused import - com.hazelcast.nio.serialization.impl.Versioned. [UnusedImports]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/hazelcast/src/main/java/com/hazelcast/internal/partition/operation/DemoteResponseOperation.java:29:8: Unused import - com.hazelcast.nio.serialization.impl.Versioned. [UnusedImports]
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.3.0:checkstyle (default) on project hazelcast: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There are 2 errors reported by Checkstyle 9.3 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/checkstyle/checkstyle.xml ruleset. -> [Help 1]
--------------------------
[ERROR] 
--------------------------
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
--------------------------
[ERROR] 
--------------------------
[ERROR] For more information about the errors and possible solutions, please read the following articles:
--------------------------
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
--------------------------
[ERROR] 
--------------------------
[ERROR] After correcting the problems, you can resume the build with the command
--------------------------
[ERROR]   mvn  -rf :hazelcast
--------------------------

JamesHazelcast · 2023-10-02T13:50:10Z

run-lab-run

JamesHazelcast

LGTM, thanks for the contribution @tommyk-gears!

JamesHazelcast · 2023-10-03T13:09:22Z

Hi @hazelcast/apis, could someone please review? TIA!

vbekiaris

thanks for contributing this enhancement @tommyk-gears!

vbekiaris · 2023-10-19T14:27:35Z

thank you @tommyk-gears for your contribution 🏅

Introduces Cluster#demoteLocalDataMember method which transitions the local member from a data/normal member to a lite member. The implementation is mostly a blend of member promotion and member shutdown. Co-authored-by: Vassilis Bekiaris <vbekiaris@gmail.com> Co-authored-by: James Holgate <130981049+JamesHazelcast@users.noreply.github.com>

tommyk-gears and others added 9 commits April 20, 2023 21:40

Add capability to demote a data member to a lite member

578feba

Added a new method `Cluster.demoteLocalDataMember()` which demotes local member to a lite member if it's a normal data member.

Merge branch 'hazelcast:master' into demote-local-data-member

1e5a5ea

Merge branch 'hazelcast:master' into demote-local-data-member

0e04e9f

Minimalistic fix to make sure that partitions are eventually removed …

08b5960

…from demoted members

Merge branch 'hazelcast:master' into demote-local-data-member

5a1b674

Merge branch 'demote-local-data-member' of github.com:tommyk-gears/ha…

da621a4

…zelcast into demote-local-data-member

Make sure the member dis-owns all replicas before it is marked as lit…

038925b

…e member

Merge branch 'hazelcast:master' into demote-local-data-member

c1a93ba

fix checkstyle issue

2789c8d

tommyk-gears requested a review from a team as a code owner May 22, 2023 14:04

hz-devops-test added the Source: Community PR or issue was opened by a community user label May 22, 2023

vbekiaris requested review from vbekiaris and ramizdundar May 22, 2023 14:25

vbekiaris reviewed Jun 15, 2023

View reviewed changes

vbekiaris added Type: Enhancement Team: Core Module: Cluster labels Jun 16, 2023

tommyk-gears and others added 8 commits June 16, 2023 10:42

Update hazelcast/src/main/java/com/hazelcast/cluster/Cluster.java

e38fef6

Co-authored-by: Vassilis Bekiaris <vbekiaris@gmail.com>

Update hazelcast/src/main/java/com/hazelcast/internal/cluster/impl/Cl…

30bbaa0

…usterServiceImpl.java Co-authored-by: Vassilis Bekiaris <vbekiaris@gmail.com>

Update hazelcast/src/main/java/com/hazelcast/internal/partition/impl/…

73a865e

…InternalPartitionServiceImpl.java Co-authored-by: Vassilis Bekiaris <vbekiaris@gmail.com>

Use OperationsService#invokeOnMaster rather than invoking on the supp…

56018d6

…lied master address

Merge branch 'hazelcast:master' into demote-local-data-member

7b29ffb

ClusterService#demoteLocalDataMember requires cluster version 5.4+

f6b2b78

Clarify where DemoteResponseOperation is sent

4ce2adc

DemoteDataMemberTest cleanup to align with changes from master

d9d5777

ramizdundar reviewed Jun 16, 2023

View reviewed changes

tommyk-gears added 2 commits September 7, 2023 14:21

fix checkstyle issues

291dda5

Merge branch 'master' into demote-local-data-member

c27bd2a

vbekiaris requested review from vbekiaris and JamesHazelcast September 27, 2023 09:11

JamesHazelcast reviewed Sep 27, 2023

View reviewed changes

tommyk-gears added 4 commits September 29, 2023 10:24

Fix a few trivial review comments

c50accb

- Remove Versioned interface from Demote(Request|Response)Operation - Reorder properties in ClusterProperty to have newly added property last

Use explicit executor in thenCompose

7d51f5c

Cluster#demoteLocalDataMember should throw IllegalStateException if t…

5372222

…rying to demote the only data member

Make DemoteResponseOperation idempotent

0199af3

JamesHazelcast added 2 commits October 2, 2023 13:48

Update DemoteRequestOperation.java

a7d1349

Checkstyle in DemoteResponseOperation

9cf6fa9

JamesHazelcast added this to the 5.4.0 milestone Oct 3, 2023

JamesHazelcast added the Add to Release Notes label Oct 3, 2023

JamesHazelcast self-requested a review October 3, 2023 13:07

JamesHazelcast approved these changes Oct 3, 2023

View reviewed changes

vbekiaris approved these changes Oct 4, 2023

View reviewed changes

srknzl approved these changes Oct 9, 2023

View reviewed changes

vbekiaris merged commit e5e13fd into hazelcast:master Oct 19, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Demote local data member #24617

Demote local data member #24617

tommyk-gears commented May 22, 2023

devOpsHazelcast commented May 22, 2023 •

edited

devOpsHazelcast commented May 22, 2023

devOpsHazelcast commented May 22, 2023

vbekiaris commented May 22, 2023

vbekiaris left a comment

srknzl commented Sep 7, 2023

hz-devops-test commented Sep 7, 2023

vbekiaris commented Sep 27, 2023

hz-devops-test commented Sep 27, 2023

JamesHazelcast left a comment

JamesHazelcast commented Sep 28, 2023 •

edited

tommyk-gears commented Sep 29, 2023

JamesHazelcast commented Oct 2, 2023

hz-devops-test commented Oct 2, 2023

JamesHazelcast commented Oct 2, 2023

JamesHazelcast left a comment

JamesHazelcast commented Oct 3, 2023

vbekiaris left a comment

vbekiaris commented Oct 19, 2023

Demote local data member #24617

Demote local data member #24617

Conversation

tommyk-gears commented May 22, 2023

devOpsHazelcast commented May 22, 2023 • edited

devOpsHazelcast commented May 22, 2023

devOpsHazelcast commented May 22, 2023

vbekiaris commented May 22, 2023

vbekiaris left a comment

Choose a reason for hiding this comment

srknzl commented Sep 7, 2023

hz-devops-test commented Sep 7, 2023

vbekiaris commented Sep 27, 2023

hz-devops-test commented Sep 27, 2023

JamesHazelcast left a comment

Choose a reason for hiding this comment

JamesHazelcast commented Sep 28, 2023 • edited

tommyk-gears commented Sep 29, 2023

JamesHazelcast commented Oct 2, 2023

hz-devops-test commented Oct 2, 2023

JamesHazelcast commented Oct 2, 2023

JamesHazelcast left a comment

Choose a reason for hiding this comment

JamesHazelcast commented Oct 3, 2023

vbekiaris left a comment

Choose a reason for hiding this comment

vbekiaris commented Oct 19, 2023

devOpsHazelcast commented May 22, 2023 •

edited

JamesHazelcast commented Sep 28, 2023 •

edited