Hazelcast (3.8.4) closing listeners

[ghost]

Issue

As discussed 2017-08-29 with Al

We are losing notification from a hazelcast node through ITopic listeners (MessageListener). Infrequently (but often enough to be a concern) we lose all notifications for new messages posted to an ITopic that are stored on one node in a cluster. Other nodes in the cluster seem to continue to work normally and the client continues to receive new messages from those nodes. This happens with no exceptions/errors seen at the client side.

This is not an issue with ReliableTopics and associated listeners.

Replication

We can replicate this issue as follows:

1. Run a cluster of 2 nodes and one client application subscribed to the ITopics.
2. Post messages (increasing integers) in an ITopic
3. Repeatedly shut down and restart one node until we see the client logging messages with obvious gaps.

[pcortizona]

We have observed the same behavior.

…

On 30 August 2017 at 09:33, cryptomorph ***@***.***> wrote: *Issue* As discussed 2017-08-29 with Al We are losing notification from a hazelcast node through ITopic listeners (MessageListener). Infrequently (but often enough to be a concern) we lose all notifications for new messages posted to an ITopic that are stored on one node in a cluster. Other nodes in the cluster seem to continue to work normally and the client continues to receive new messages from those nodes. This happens with no exceptions/errors seen at the client side. This is not an issue with ReliableTopics and associated listeners. *Replication* We can replicate this issue as follows: 1. Run a cluster of 2 nodes and one client application subscribed to the ITopics. 2. Post messages (increasing integers) in an ITopic 3. Repeatedly shut down and restart one node until we see the client logging messages with obvious gaps. In the test logged we shut down instance-1 at 14:13:40 and it was back up at 10:14:30. Client connection to instance-2 appears to be lost. *Logs* Attached (as requested by Al yesterday) are the Hazelcast logs for our 2 nodes and a brief cutting of our client application. instance-2.log.txt <https://github.com/hazelcast/hazelcast/files/1263904/instance-2.log.txt> instance-2.log.1.txt <https://github.com/hazelcast/hazelcast/files/1263905/instance-2.log.1.txt> client.log.txt <https://github.com/hazelcast/hazelcast/files/1263907/client.log.txt> instance-1.log.txt <https://github.com/hazelcast/hazelcast/files/1263902/instance-1.log.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11264>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AR0Z7vTWZLxX635yQQtXLh6XlH8skSqAks5sdXKrgaJpZM4PHe1K> .

[sancar]

Same problem also reported here
#10101 (comment)
Pasting here as future reference.

[sancar]

The issue caused by using hostnames to connect.
For a workaround using ip addresses instead of hostnames will work.

fix is almost ready. It will be applied to 3.9 and backported to 3.8.6
Related issue : #11226

[sancar]

fixed by #11368