Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Joining members cannot form cluster when existing members are killed #14051

Closed
tezc opened this Issue Oct 26, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@tezc
Copy link
Contributor

tezc commented Oct 26, 2018

Config :

Config config = new Config();
JoinConfig join = config.getNetworkConfig().getJoin();
join.getMulticastConfig().setEnabled(false);
join.getAwsConfig().setEnabled(false);
join.getTcpIpConfig().setEnabled(true);
join.getTcpIpConfig().addMember("127.0.0.1:5701");
join.getTcpIpConfig().addMember("127.0.0.1:5702");
join.getTcpIpConfig().addMember("127.0.0.1:5703");
join.getTcpIpConfig().addMember("127.0.0.1:5704");

  • Member A and Member B up and running,
  • Start Member C and Member D
  • Kill Member A and B before C and D actually join
  • C and D will not form cluster

Easily reproducible on my local.

@Danny-Hazelcast also reproduced the issue here :
https://hazelcast-l337.ci.cloudbees.com/view/kill/job/cluster-formation/4/console

Logs from Member C like :

FINE: [127.0.0.1]:5703 [dev] [3.11] Will send master question to each address in: [[127.0.0.1]:5702, [127.0.0.1]:5704, [127.0.0.1]:5701]
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] NOT sending master question to blacklisted endpoints: {[127.0.0.1]:5704=false}
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] NOT sending master question to blacklisted endpoints: {[127.0.0.1]:5704=false}
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:47 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:47 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpAcceptor
FINE: [127.0.0.1]:5703 [dev] [3.11] Accepting socket connection from /127.0.0.1:60399
Oct 26, 2018 9:50:47 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/127.0.0.1:5703->/127.0.0.1:60399}.inboundPipeline, NioChannel{/127.0.0.1:5703->/127.0.0.1:60399}.outboundPipeline
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Established socket connection between /127.0.0.1:5703 and /127.0.0.1:60399
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5703 [dev] [3.11] Initialized new cluster connection between /127.0.0.1:5703 and /127.0.0.1:60399
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
FINEST: [127.0.0.1]:5703 [dev] [3.11] Binding Connection[id=3, /127.0.0.1:5703->/127.0.0.1:60399, endpoint=null, alive=true, type=MEMBER] to [127.0.0.1]:5704, reply is true
Oct 26, 2018 9:50:47 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5704 is removed from the blacklist.
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
FINEST: [127.0.0.1]:5703 [dev] [3.11] Sending bind packet to [127.0.0.1]:5704
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] Resetting connection monitor for endpoint [127.0.0.1]:5704
Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5703 [dev] [3.11] Connection[id=2, /0:0:0:0:0:0:0:0:47511->/127.0.0.1:5701, endpoint=[127.0.0.1]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
java.io.EOFException: Remote socket closed!
	at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:116)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:368)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:353)
	at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:279)
	at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:234)

Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.io.EOFException {Remote socket closed!}, Error-Count: 1
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Removed connection to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] NOT sending master question to blacklisted endpoints: {}
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5702
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5704
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:53193->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:53193->null}.outboundPipeline
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5702
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
WARNING: [127.0.0.1]:5703 [dev] [3.11] Could not create connection to possible master [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:47511->/127.0.0.1:5701}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:47511->/127.0.0.1:5701}.outboundPipeline
Oct 26, 2018 9:50:49 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:53193->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:53193->null}.outboundPipeline
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:454)
	at sun.nio.ch.Net.connect(Net.java:446)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
	at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:49 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 2
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5703 [dev] [3.11] Connection[id=1, /0:0:0:0:0:0:0:0:39283->/127.0.0.1:5702, endpoint=[127.0.0.1]:5702, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
java.io.EOFException: Remote socket closed!
	at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:116)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:368)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:353)
	at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:279)
	at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:234)

Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5702 Cause => java.io.EOFException {Remote socket closed!}, Error-Count: 1
Oct 26, 2018 9:50:49 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Removed connection to [127.0.0.1]:5702
Oct 26, 2018 9:50:49 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:39283->/127.0.0.1:5702}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:39283->/127.0.0.1:5702}.outboundPipeline
Oct 26, 2018 9:50:49 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:49 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true


---------------------------------


FINE: [127.0.0.1]:5703 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:54 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:54 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Joining to master [127.0.0.1]:5701
Oct 26, 2018 9:50:54 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:54 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:54 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:50683->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:50683->null}.outboundPipeline
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:454)
	at sun.nio.ch.Net.connect(Net.java:446)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
	at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:55 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:50683->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:50683->null}.outboundPipeline
Oct 26, 2018 9:50:55 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 8
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [127.0.0.1]:5703 [dev] [3.11] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 9
Oct 26, 2018 9:50:55 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:55 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:55 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Joining to master [127.0.0.1]:5701
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:55 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:58555->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:58555->null}.outboundPipeline
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:56 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:58555->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:58555->null}.outboundPipeline
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:454)
	at sun.nio.ch.Net.connect(Net.java:446)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
	at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:56 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 9
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [127.0.0.1]:5703 [dev] [3.11] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 10
Oct 26, 2018 9:50:56 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:56 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations

Logs from Member D like :

INFO: [127.0.0.1]:5704 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:50 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:44785->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:44785->null}.outboundPipeline
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:51 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:44785->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:44785->null}.outboundPipeline
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:454)
	at sun.nio.ch.Net.connect(Net.java:446)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
	at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:51 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5704 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5704 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 3
Oct 26, 2018 9:50:51 AM com.hazelcast.internal.cluster.impl.operations.JoinMastershipClaimOp
FINE: [127.0.0.1]:5704 [dev] [3.11] Sending 'false' for master claim of node: [127.0.0.1]:5703
Oct 26, 2018 9:50:51 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5704 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:51 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5704 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:51 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:33433->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:33433->null}.outboundPipeline
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:52 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:33433->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:33433->null}.outboundPipeline
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Connection refused to address /127.0.0.1:5701


-----------------------------------


FINE: [127.0.0.1]:5704 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:52 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5704 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:52 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5704 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:52 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:32933->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:32933->null}.outboundPipeline
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:53 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:32933->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:32933->null}.outboundPipeline
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:454)
	at sun.nio.ch.Net.connect(Net.java:446)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
	at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
	at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:53 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5704 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5704 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 5
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [127.0.0.1]:5704 [dev] [3.11] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 6
Oct 26, 2018 9:50:53 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5704 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
@Danny-Hazelcast

This comment has been minimized.

@mdogan

This comment has been minimized.

Copy link
Member

mdogan commented Jan 4, 2019

@Danny-Hazelcast

This comment has been minimized.

Copy link
Member

Danny-Hazelcast commented Jan 4, 2019

Yes i will run with the branch above

@tezc

This comment has been minimized.

Copy link
Contributor Author

tezc commented Jan 4, 2019

@mdogan I can't reproduce it with your branch.

@Danny-Hazelcast

This comment has been minimized.

@mdogan

This comment has been minimized.

Copy link
Member

mdogan commented Jan 4, 2019

@mdogan I can't reproduce it with your branch.

looks like its fixed https://hazelcast-l337.ci.cloudbees.com/view/kill/job/cluster-formation/12/console

Thank you both. I'll prepare a PR for the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.