split brain client side oome, com.hazelcast.internal.util.executor.StripedExecutor$Worker,

[Danny-Hazelcast]

clients are just connected to the cluster, clients perform no operations.

http://jenkins.hazelcast.com/view/split/job/split-rest/61/console

/disk1/jenkins/workspace/split-rest/4.1-SNAPSHOT/2020_03_06-08_51_34/rest Failed

output/HZ/HzClient2HZAA/HzClient2HZAA.hprof
output/HZ/HzClient2HZAA/HzClient2HZAA.oome

http://54.147.27.51/~jenkins/workspace/split-rest/4.1-SNAPSHOT/2020_03_06-08_51_34/rest

GC
http://54.147.27.51/~jenkins/workspace/split-rest/4.1-SNAPSHOT/2020_03_06-08_51_34/rest/gc.html

Problem Suspect 1

The thread com.hazelcast.internal.util.executor.StripedExecutor$Worker @ 0xfda98ac0 hz.client_1.event-4 keeps local variables with total size 8,688,072 (26.61%) bytes.

The memory is accumulated in one instance of "java.util.concurrent.LinkedBlockingQueue$Node" loaded by "".
The stacktrace of this Thread is available. See stacktrace.

Keywords
java.util.concurrent.LinkedBlockingQueue$Node
Details »
Problem Suspect 2

The thread com.hazelcast.internal.util.executor.StripedExecutor$Worker @ 0xfda98790 hz.client_1.event-5 keeps local variables with total size 8,621,360 (26.41%) bytes.

The memory is accumulated in one instance of "java.util.concurrent.LinkedBlockingQueue$Node" loaded by "".
The stacktrace of this Thread is available. See stacktrace.

[Danny-Hazelcast]

Mar 06, 2020 1:10:35 PM com.hazelcast.client.impl.spi.ClientListenerService
SEVERE: hz.client_1 [HZ] [4.1-SNAPSHOT] hz.client_1.event-5 caught an exception while processing:com.hazelcast.client.impl.spi.impl.listener.ClientListenerServiceImpl$$Lambda$640/684600674@2de6e977
java.lang.OutOfMemoryError: GC overhead limit exceeded

Mar 06, 2020 1:10:36 PM com.hazelcast.client.impl.connection.ClientConnectionManager
I

WARNING: hz.client_1 [HZ] [4.1-SNAPSHOT] Could not connect to any cluster, shutting down the client: Unable to connect to any cluster.
Mar 06, 2020 1:17:33 PM com.hazelcast.core.LifecycleService
INFO: hz.client_1 [HZ] [4.1-SNAPSHOT] HazelcastClient 4.1-SNAPSHOT (20200306 - afe56c8, 357b137) is SHUTTING_DOWN
Mar 06, 2020 1:17:38 PM com.hazelcast.core.LifecycleService
INFO: hz.client_1 [HZ] [4.1-SNAPSHOT] HazelcastClient 4.1-SNAPSHOT (20200306 - afe56c8, 357b137) is SHUTDOWN

[Danny-Hazelcast]

last passed http://jenkins.hazelcast.com/view/split/job/split-rest/60/console
http://jenkins.hazelcast.com/view/split/job/split-rest/60/
4.1-SNAPSHOT
Build (Feb 28, 2020 8:51:15 AM)
Started 7 days 5 hr ago
Took 20 hr on stabilizer

[Danny-Hazelcast]

after the reported oome in this issue, runs with latest master are passing http://jenkins.hazelcast.com/view/split/job/split-rest/

[sancar]

Hi @Danny-Hazelcast , I have investigated the heap dumps. The packets that are piling are the partition table updates.
It is possibly fixed here #16939
and backport #16938

Can we close the issue since we have identified the problem and tests are passing now ?

[sancar]

Confirmed by @Danny-Hazelcast . Closing the issue.