Hazelcast listeners are lost during cluster disconnects - Backport PR #8848 #9183

ihsandemir · 2016-10-26T17:37:25Z

Fixes the client listener registration if the client owner connection is lost and reconnected (e.g. due to heartbeat or network problems).

Authentication response returns a list of members at which the client is unregistered so that the listener service acts upon this list. The smart listener service re-registers the listeners to these unregistered members. The non-smart listener service re-registers the listeners only if the reconnected member is not the same as the previous member or the previous member client resources were cleaned up.

The ClientDisconnectionOperation also acts smart so that it only cleans client resources if the dead member uuid is the current owner member uuid of the client and there is no alive connection endpoint to the client.

For compatibility, when client is connected to earlier versions the smart listener service re-registers all existing listeners.

Fixes the client listener registration if the client owner connection is lost and reconnected (e.g. due to heartbeat or network problems). Authentication response returns a list of members at which the client is unregistered so that the listener service acts upon this list. The smart listener service re-registers the listeners to these unregistered members. The non-smart listener service re-registers the listeners only if the reconnected member is not the same as the previous member or the previous member client resources were cleaned up. The ClientDisconnectionOperation also acts smart so that it only cleans client resources if the dead member uuid is the current owner member uuid of the client and there is no alive connection endpoint to the client. For compatibility, when client is connected to earlier versions the smart listener service re-registers all existing listeners.

…thBinaryInMemoryFormat and putToCacheAndGetInvalidationEventWhenNodeShutdownWithObjectInMemoryFormat as indicated at hazelcast#9172 (comment)

devOpsHazelcast · 2016-10-31T09:36:15Z

Test PASSed.

devOpsHazelcast · 2016-10-31T10:42:59Z

Test PASSed.

… before instead of creating a new one. This eliminates a lot of endpoint to getEndpoint() changes.

devOpsHazelcast · 2016-11-01T16:57:41Z

Test PASSed.

…on for 3.7.x minor version compatibility of the operation. Instead of this field just uses any existing alive connection to the client to decide if ClientDisconnectionOperation shoud run or not.

devOpsHazelcast · 2016-11-01T18:07:47Z

Test PASSed.

devOpsHazelcast · 2016-11-02T07:09:45Z

Test PASSed.

alparslanavci · 2016-11-02T08:04:01Z

Backport of #8848. Just mentioning here to connect the PRs.

devOpsHazelcast · 2016-11-02T19:48:39Z

Test PASSed.

devOpsHazelcast · 2016-11-04T00:11:38Z

Test FAILed.
https://hazelcast-l337.ci.cloudbees.com/job/new-lab-fast-pr/5574/

vbekiaris · 2016-11-04T06:59:49Z

hazelcast/src/main/java/com/hazelcast/client/impl/operations/ClientReAuthOperation.java

@@ -55,7 +55,7 @@ public void run() throws Exception {

    @Override
    public boolean returnsResponse() {
-        return Boolean.TRUE;
+        return Boolean.FALSE;


just nitpicking here, this is unnecessary unboxing, we could just return the primitive false directly.

OK, corrected.

devOpsHazelcast · 2016-11-04T07:33:04Z

Test FAILed.
https://hazelcast-l337.ci.cloudbees.com/job/new-lab-fast-pr/5577/

…n any response and changed the smart listener to reregister all the listeners on cluster reconnect. The Smart listener is not using the cleanedup member list and this list is not beign populated for backward compatibility.

vbekiaris

👍 with minor comments

vbekiaris · 2016-11-04T09:07:51Z

hazelcast/src/main/java/com/hazelcast/util/FutureUtil.java

+     * This ExceptionHandler rethrows {@link java.util.concurrent.ExecutionException}s and logs
+     * {@link com.hazelcast.core.MemberLeftException}s to the log.
+     */
+    public static final ExceptionHandler RETHROW_ALL_EXCEPT_MEMBER_LEFT = new ExceptionHandler() {


This exception handler doesn't seem to have any usages, maybe remove it from the final PR?

vbekiaris · 2016-11-04T09:09:55Z

...c/test/java/com/hazelcast/client/protocol/compatibility/ClientCompatibilityNullTest_1_0.java

@@ -5751,3 +5705,6 @@ public void handle(java.util.Collection<com.hazelcast.map.impl.querycache.event.
    }
 }

+


newlines leftover

This is a generated test, i need to remove them in its ftl file in client-protocol. It does not look urgent, hence i will just open an issue for this and correct ftl later.
Issue: hazelcast/hazelcast-client-protocol#33

ihsandemir · 2016-11-04T20:33:00Z

run-lab-run

devOpsHazelcast · 2016-11-04T21:13:00Z

Test PASSed.

devOpsHazelcast · 2016-11-05T00:26:29Z

Test PASSed.

ihsandemir · 2016-11-07T11:30:38Z

Some more details:

When the client for some reason loses the connection the member which is the owner for the client, it becomes CLIENT_DISCONNECTED and client reconnects to another member and then its lifecycle state becomes CLIENT_CONNECTED again. When this reconnection happen, it does re-register all the listeners to the members.

It is different when losing connection to non-owner member. If non owner connection is lost, no action needs to be taken in terms of listeners.

The listeners are registered to any new members which join the cluster.

Hence, you do not need to reregister any existing listeners since we are taking care of it at the background.

* Fixes the client listener registration if the client owner connection is lost and reconnected (e.g. due to heartbeat or network problems). Authentication response returns a list of members at which the client is unregistered so that the listener service acts upon this list. The smart listener service re-registers the listeners to these unregistered members. The non-smart listener service re-registers the listeners only if the reconnected member is not the same as the previous member or the previous member client resources were cleaned up. The ClientDisconnectionOperation also acts smart so that it only cleans client resources if the dead member uuid is the current owner member uuid of the client and there is no alive connection endpoint to the client. * Forward port of fixes at #9183

* Fixes the client listener registration if the client owner connection is lost and reconnected (e.g. due to heartbeat or network problems). Authentication response returns a list of members at which the client is unregistered so that the listener service acts upon this list. The smart listener service re-registers the listeners to these unregistered members. The non-smart listener service re-registers the listeners only if the reconnected member is not the same as the previous member or the previous member client resources were cleaned up. The ClientDisconnectionOperation also acts smart so that it only cleans client resources if the dead member uuid is the current owner member uuid of the client and there is no alive connection endpoint to the client. * Forward port of fixes at hazelcast#9183

ihsandemir added the Team: Client label Oct 26, 2016

ihsandemir added this to the 3.7.3 milestone Oct 26, 2016

ihsandemir self-assigned this Oct 26, 2016

Ignored the tests putToCacheAndGetInvalidationEventWhenNodeShutdownWi…

4102c25

…thBinaryInMemoryFormat and putToCacheAndGetInvalidationEventWhenNodeShutdownWithObjectInMemoryFormat as indicated at hazelcast#9172 (comment)

ihsandemir force-pushed the maint/backports/PR8848 branch 2 times, most recently from 92960a5 to 4102c25 Compare October 30, 2016 11:08

ihsandemir force-pushed the maint/backports/PR8848 branch from 3762539 to 060d8ff Compare October 31, 2016 09:59

Changed to use the existing endpoint if the client were authenticated…

060d8ff

… before instead of creating a new one. This eliminates a lot of endpoint to getEndpoint() changes.

Removed the added field memberUuid from the ClientDisconnectedOperati…

3a30386

…on for 3.7.x minor version compatibility of the operation. Instead of this field just uses any existing alive connection to the client to decide if ClientDisconnectionOperation shoud run or not.

Changes per review comments.

4bf4faa

Changes per review comments. Minor correction.

3c6149d

vbekiaris reviewed Nov 4, 2016

View reviewed changes

ihsandemir force-pushed the maint/backports/PR8848 branch from 9c844df to b7e82b0 Compare November 4, 2016 07:29

vbekiaris reviewed Nov 4, 2016

View reviewed changes

Checkstyle fix.

65fe21d

ihsandemir mentioned this pull request Nov 4, 2016

Extra new lines at the end of the generated tests hazelcast/hazelcast-client-protocol#33

Closed

ihsandemir added 3 commits November 4, 2016 13:31

Review correction.

0c1d62e

Test fix.

6c0a261

Fix for the event handler removal problem and no event handler logs.

ee24eb2

Fix for deadlock while reregistering.

2edd0ec

Fix for client freezes due to invocation timeouts.

8ec45de

ihsandemir merged commit f27eda4 into hazelcast:maintenance-3.x Nov 5, 2016

ihsandemir deleted the maint/backports/PR8848 branch November 5, 2016 08:20

ihsandemir added a commit to ihsandemir/hazelcast that referenced this pull request Nov 14, 2016

Forward port of fixes at hazelcast#9183

83cbb18

This was referenced May 4, 2017

Invalid connection type: NONE #10480

Closed

Node is not connected to the Management Center #10482

Closed

mmedenjak added the Source: Internal PR or issue was opened by an employee label Apr 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hazelcast listeners are lost during cluster disconnects - Backport PR #8848 #9183

Hazelcast listeners are lost during cluster disconnects - Backport PR #8848 #9183

ihsandemir commented Oct 26, 2016

devOpsHazelcast commented Oct 31, 2016

devOpsHazelcast commented Oct 31, 2016

devOpsHazelcast commented Nov 1, 2016

devOpsHazelcast commented Nov 1, 2016

devOpsHazelcast commented Nov 2, 2016

alparslanavci commented Nov 2, 2016

devOpsHazelcast commented Nov 2, 2016

devOpsHazelcast commented Nov 4, 2016

vbekiaris Nov 4, 2016

ihsandemir Nov 4, 2016

devOpsHazelcast commented Nov 4, 2016

vbekiaris left a comment

vbekiaris Nov 4, 2016

vbekiaris Nov 4, 2016

ihsandemir Nov 4, 2016 •

edited

ihsandemir commented Nov 4, 2016

devOpsHazelcast commented Nov 4, 2016

devOpsHazelcast commented Nov 5, 2016

ihsandemir commented Nov 7, 2016

		@@ -5751,3 +5705,6 @@ public void handle(java.util.Collection<com.hazelcast.map.impl.querycache.event.
		}
		}

Hazelcast listeners are lost during cluster disconnects - Backport PR #8848 #9183

Hazelcast listeners are lost during cluster disconnects - Backport PR #8848 #9183

Conversation

ihsandemir commented Oct 26, 2016

devOpsHazelcast commented Oct 31, 2016

devOpsHazelcast commented Oct 31, 2016

devOpsHazelcast commented Nov 1, 2016

devOpsHazelcast commented Nov 1, 2016

devOpsHazelcast commented Nov 2, 2016

alparslanavci commented Nov 2, 2016

devOpsHazelcast commented Nov 2, 2016

devOpsHazelcast commented Nov 4, 2016

vbekiaris Nov 4, 2016

Choose a reason for hiding this comment

ihsandemir Nov 4, 2016

Choose a reason for hiding this comment

devOpsHazelcast commented Nov 4, 2016

vbekiaris left a comment

Choose a reason for hiding this comment

vbekiaris Nov 4, 2016

Choose a reason for hiding this comment

vbekiaris Nov 4, 2016

Choose a reason for hiding this comment

ihsandemir Nov 4, 2016 • edited

Choose a reason for hiding this comment

ihsandemir commented Nov 4, 2016

devOpsHazelcast commented Nov 4, 2016

devOpsHazelcast commented Nov 5, 2016

ihsandemir commented Nov 7, 2016

ihsandemir Nov 4, 2016 •

edited