Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WARNING: Client heartbeat is timed out , closing connection #8777

Closed
dmitrymz opened this issue Aug 25, 2016 · 18 comments

Comments

Projects
None yet
7 participants
@dmitrymz
Copy link

commented Aug 25, 2016

we have hazelcast cluster v3.6.4 deployed on aws,
and see the same symptoms as described in #5054

WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701

this message gets produced after we replace the client host , and logs gets polluted indefinitely
with new message above every 10sec

please, advise

Aug 25, 2016 6:19:51 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT
Aug 25, 2016 6:20:01 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT
Aug 25, 2016 6:20:11 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT
Aug 25, 2016 6:20:21 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT
Aug 25, 2016 6:20:31 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT
Aug 25, 2016 6:20:41 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT
Aug 25, 2016 6:20:51 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT
Aug 25, 2016 6:21:01 AM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[10.1.16.211]:53760, alive=false, type=JAVA_CLIENT

two nodes cluster
the only remedy now is to delete the bad node(s) from the cluster, and spin a new one.

looks like other people have same issue as well - see in google groups
https://groups.google.com/forum/#!topic/hazelcast/R1stzCWfm68

our server config:

    <properties>
        <property name="hazelcast.logging.type">slf4j</property>
        <property name="hazelcast.icmp.enabled">true</property>
    </properties>

    <network>
        <port auto-increment="false">5701</port>
        <join>
            <multicast enabled="false"/>
            <tcp-ip enabled="false"/>
            <aws enabled="true">
                <access-key>xxx</access-key>
                <secret-key>xxx</secret-key>
                <region>us-west-2</region>
                <host-header>ec2.amazonaws.com</host-header>
                <security-group-name>xxx</security-group-name>
                <tag-key>tag</tag-key>
                <tag-value>stage</tag-value>
            </aws>
        </join>
    </network>
@ihsandemir

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2016

Do you have a reproducer test?

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Aug 29, 2016

not really a test. what happens, when we shutdown a client, one or more hazelcast members starts producing these messages indefinitely. So really, to reproduce it - start for example a 2 node cluster, add couple of clients, then remove a client, and add a new one. by the way we are running cluster in aws if that helps

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Aug 31, 2016

this is ongoing issue for us as we have daily client builds and deployments, so we have to replace the failing hazelcast cluster nodes frequently.

@ihsandemir ihsandemir self-assigned this Aug 31, 2016

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Sep 9, 2016

hi. any updates on this issue or ETA? thanks

@jjacobj84

This comment has been minimized.

Copy link

commented Sep 16, 2016

Hello , I was able to recreate this issue using a very simple test code.
Trigger a data partition event ( by adding a new node to cluster ) and disconnect clients. In our case the client that got disconnected had a distributed lock and it was never released.
Hope it helps in fixing it.

Logs:

INFO: [127.0.0.1]:5702 [dev] [3.6]

Members [2] {
    Member [127.0.0.1]:5702 this
    Member [127.0.0.1]:5703
}

Sep 16, 2016 11:58:14 AM com.hazelcast.transaction.TransactionManagerService
INFO: [127.0.0.1]:5702 [dev] [3.6] Committing/rolling-back alive transactions of Member [127.0.0.1]:5701, UUID: f39c7cc8-476a-48eb-adfe-573a9b4f6b14
Sep 16, 2016 11:58:29 AM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Partition balance is ok, no need to re-partition cluster data...
Sep 16, 2016 11:58:36 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5702 [dev] [3.6] Connection [Address[127.0.0.1]:53918] lost. Reason: java.io.EOFException[Remote socket closed!]
Sep 16, 2016 11:58:36 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketReader
WARNING: [127.0.0.1]:5702 [dev] [3.6] hz.Root-Instance.IO.thread-in-0 Closing socket to endpoint Address[127.0.0.1]:53918, Cause:java.io.EOFException: Remote socket closed!
Sep 16, 2016 11:58:43 AM com.hazelcast.spi.OperationService
INFO: [127.0.0.1]:5702 [dev] [3.6] Invocations:1 timeouts:0 backup-timeouts:1
Sep 16, 2016 11:58:58 AM com.hazelcast.spi.OperationService
INFO: [127.0.0.1]:5702 [dev] [3.6] Invocations:1 timeouts:0 backup-timeouts:1
Sep 16, 2016 11:59:13 AM com.hazelcast.nio.tcp.SocketAcceptorThread
INFO: [127.0.0.1]:5702 [dev] [3.6] Accepting socket connection from /127.0.0.1:54356
Sep 16, 2016 11:59:13 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [127.0.0.1]:5702 [dev] [3.6] Established socket connection between /127.0.0.1:5702 and /127.0.0.1:54356
11:59:22.769 [hz.Root-Instance.event-2] INFO  c.c.g.c.i.CoordinatorInstanceQuorumFunction - Current Quorum Size 2 and clusterNodes count 3
11:59:22.770 [hz.Root-Instance.event-2] INFO  c.c.g.c.i.CoordinatorInstanceQuorumFunction - Current Quorum Size 2 and clusterNodes count 3
11:59:22.770 [hz.Root-Instance.event-2] INFO  c.c.g.c.i.CoordinatorInstanceQuorumFunction - Current Quorum Size 2 and clusterNodes count 3
Sep 16, 2016 11:59:22 AM com.hazelcast.cluster.ClusterService
INFO: [127.0.0.1]:5702 [dev] [3.6]

Members [3] {
    Member [127.0.0.1]:5702 this
    Member [127.0.0.1]:5703
    Member [127.0.0.1]:5701
}

Sep 16, 2016 11:59:28 AM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Re-partitioning cluster data... Migration queue size: 90
Sep 16, 2016 11:59:35 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5702 [dev] [3.6] Connection [Address[127.0.0.1]:53913] lost. Reason: java.io.EOFException[Remote socket closed!]
Sep 16, 2016 11:59:35 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketReader
WARNING: [127.0.0.1]:5702 [dev] [3.6] hz.Root-Instance.IO.thread-in-2 Closing socket to endpoint Address[127.0.0.1]:53913, Cause:java.io.EOFException: Remote socket closed!
Sep 16, 2016 11:59:43 AM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 87
Sep 16, 2016 11:59:57 AM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 83
Sep 16, 2016 12:00:12 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 77
Sep 16, 2016 12:00:27 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 74
Sep 16, 2016 12:00:42 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 71
Sep 16, 2016 12:00:57 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 67
Sep 16, 2016 12:01:12 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 63
Sep 16, 2016 12:01:27 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 59
Sep 16, 2016 12:01:42 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 56
Sep 16, 2016 12:01:57 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 52
Sep 16, 2016 12:02:13 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 47
Sep 16, 2016 12:02:27 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 45
Sep 16, 2016 12:02:42 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 43
Sep 16, 2016 12:02:57 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 40
Sep 16, 2016 12:03:13 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 37
Sep 16, 2016 12:03:28 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 32
Sep 16, 2016 12:03:44 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 26
Sep 16, 2016 12:03:44 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:03:54 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:03:57 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 21
Sep 16, 2016 12:04:04 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:12 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 18
Sep 16, 2016 12:04:14 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:24 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:27 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 15
Sep 16, 2016 12:04:34 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:42 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 7
Sep 16, 2016 12:04:44 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53913, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:44 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:54 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53913, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:54 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:04:57 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] Remaining migration tasks in queue => 1
Sep 16, 2016 12:05:04 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53913, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:05:04 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:05:10 PM com.hazelcast.partition.InternalPartitionService
INFO: [127.0.0.1]:5702 [dev] [3.6] All migration tasks have been completed, queues are empty.
Sep 16, 2016 12:05:14 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53913, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:05:14 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53918, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:05:24 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out , closing connection to Connection [0.0.0.0/0.0.0.0:5702 -> null], endpoint=Address[127.0.0.1]:53913, alive=false, type=JAVA_CLIENT
Sep 16, 2016 12:05:24 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Sep 16, 2016

@jjacobj84: many thanks, this should help a lot!

@jjacobj84

This comment has been minimized.

Copy link

commented Sep 18, 2016

@jerrinot: Any Work Around or possible fix expected ? Thanks

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Oct 4, 2016

any updates on ETA? thanks

@gionn

This comment has been minimized.

Copy link

commented Oct 4, 2016

Hazelcast 3.7.1, some problem, get warning about a disconnected client of hours before, repeating indefinitely.

Oct 04, 2016 2:02:04 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
Oct 04, 2016 2:02:14 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
Oct 04, 2016 2:02:24 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
Oct 04, 2016 2:02:34 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
Oct 04, 2016 2:02:44 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
Oct 04, 2016 2:02:54 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
Oct 04, 2016 2:03:04 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
Oct 04, 2016 2:03:14 PM com.hazelcast.client.impl.ClientHeartbeatMonitor
WARNING: Client heartbeat is timed out, closing connection to Connection[id=3, 0.0.0.0/0.0.0.0:5700->null, endpoint=[172.18.0.1]:55152, alive=false, type=JAVA_CLIENT]. Now: 2016-10-04 10:34:22.789. LastTimePacketReceived: 2016-10-04 10:34:22.789
@dmitrymz

This comment has been minimized.

Copy link
Author

commented Oct 20, 2016

two months passed on this issue... is it possible to fix it soon?

@ihsandemir

This comment has been minimized.

Copy link
Contributor

commented Nov 15, 2016

@jjacobj84 Can you explain how you disconnect the clients when you said "disconnect clients" at #8777 (comment) ? Is it physical cable disconnections?

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 15, 2016

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 16, 2016

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 17, 2016

@ihsandemir

This comment has been minimized.

Copy link
Contributor

commented Nov 17, 2016

I was able to generate the case in this way:

Start server with the option:
config.setProperty(GroupProperty.CLIENT_HEARTBEAT_TIMEOUT_SECONDS.getName(), "3")

Connect with C++ client while debugging, pause the process.

I see the repeated warning logs as described.

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 17, 2016

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 17, 2016

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 18, 2016

@ihsandemir ihsandemir changed the title WARNING: Client heartbeat is timed out , closing connection to Connection WARNING: Client heartbeat is timed out , closing connection Nov 23, 2016

@mdogan mdogan added this to the 3.8 milestone Nov 23, 2016

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 23, 2016

Fix for hazelcast#8777. Changed so that only the endpoints without ou…
…tstanding listeners or transactions is destroyed.

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 23, 2016

Fix for hazelcast#8777. Changed so that only the endpoints without ou…
…tstanding listeners or transactions is destroyed.

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 24, 2016

Fix for hazelcast#8777. Changed so that only the endpoints without ou…
…tstanding listeners or transactions is destroyed.

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Nov 24, 2016

Fix for hazelcast#8777. Changed so that only the endpoints without ou…
…tstanding listeners or transactions is destroyed.

ihsandemir added a commit that referenced this issue Nov 24, 2016

ihsandemir added a commit that referenced this issue Nov 24, 2016

tkountis added a commit to tkountis/hazelcast that referenced this issue Nov 30, 2016

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Dec 12, 2016

is there any update for this issue?
I see v3.7.4 has fixes #9308, #9287 potentially related to this.
we definitely need this in v3.6.x

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Dec 22, 2016

ping

@ihsandemir

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

Helo,

This issue should have been fixed by #9271 and #9308.

We do not have it for 3.6.x yet.

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Dec 22, 2016

please, backport to v3.6.x. we can't transition to higher major version now, and suffering from this with frequent client deployments.

@sancar

This comment has been minimized.

Copy link
Member

commented Dec 26, 2016

Sorry. We can not apply this to 3.6.x easily because of codebase change. We are actively developing 3.8 features and doing patch fixes for only 3.7.x.

@sancar sancar closed this Dec 26, 2016

@dmitrymz

This comment has been minimized.

Copy link
Author

commented Jan 12, 2017

very disappointing... as this issue was opened with 3.6, and as you know it's pretty difficult to upgrade to new major version with hazelcast.

sancar added a commit to sancar/hazelcast that referenced this issue Nov 14, 2017

Fix client endpoint update when promoting to owner
When a client connection is promoted to be the owner connection,
we were failing to update the existing endpoint on server side.
This results in no clientEndpoint being the owner. Therefore,
when endpoints are removed, none of them will result in
`client disconnected operation` and resources of the client will
leak.

Issue is found when investigating an old report.
hazelcast#8777

sancar added a commit to sancar/hazelcast that referenced this issue Nov 14, 2017

Fix client endpoint update when promoting to owner
When a client connection is promoted to be the owner connection,
we were failing to update the existing endpoint on server side.
This results in no clientEndpoint being the owner. Therefore,
when endpoints are removed, none of them will result in
`client disconnected operation` and resources of the client will
leak.

Issue is found when investigating an old report.
hazelcast#8777
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.