Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker networking - "This node is not requested endpoint" error #4537

Closed
jjongsma opened this issue Feb 4, 2015 · 33 comments

Comments

Projects
None yet
@jjongsma
Copy link

commented Feb 4, 2015

Unless you configure containers to net=host and explicitly configure Hazelcast to bind to the host's IP range, it listens on a Docker-local IP that is port-mapped to the host, and every connection request is rejected with a "This node is not requested endpoint" error, because the public IP that other nodes connect to doesn't match what Hazelcast sees inside the container.

Host networking mode is a workaround but is not scalable for running many containers on one machine.

Possible solutions:

  • Add an option to configure separate private and public node addresses. This is how Cassandra addresses the issue (listen vs. broadcast.)
  • Provide option to disable this check entirely. There are other ways to validate node identity if that is a concern (SSL, etc.)
@mesutcelik

This comment has been minimized.

Copy link
Contributor

commented Feb 9, 2015

Thanks @jchanning for the point.

We are not fully supporting Docker for the time being but planning to better integrate Hazelcast into cloud technologies... We will definitely consider your solution proposals..

@bwzhang2011

This comment has been minimized.

Copy link

commented Feb 9, 2015

@mesutcelik, thanks for feed back. hope hz team could follow up such review and try best to make hazelcast compatible to docker environment.

@decoomanj

This comment has been minimized.

Copy link

commented Mar 10, 2015

Are there any plans for this one? I currently run into the situation where I cannot use multicast to detect other nodes. The TCP-IP detection works, but I have to supply IPs to the cluster.xml. I want to replace this by DNS-names, but I keep running into this error (This node is not requested endpoint).

I was already looking in the code myself to fix this. Maybe you could give me a good starting point?

@decoomanj

This comment has been minimized.

Copy link

commented Mar 11, 2015

I have a fix for Consul and it solves all problems. Contact me for more info.

@bwzhang2011

This comment has been minimized.

Copy link

commented Mar 20, 2015

@decoomanj, well done and hope your work could be back-port into all versions of hazelcast.

@bilalyasar bilalyasar added this to the 3.6 milestone May 12, 2015

@bilalyasar bilalyasar self-assigned this May 12, 2015

@smecsia

This comment has been minimized.

Copy link

commented Jul 9, 2015

+1 We'd be glad to see better Docker support as well. For now we're having difficulties with ipv6-only hosts and Hazelcast within Docker containers. I could not find the workaround to make it work.
At the same time for dual-stack hosts Dockerized Hazelcast works well. I have no time to investigate this deeper.

@mesutcelik mesutcelik modified the milestones: 3.6, 3.7 Nov 2, 2015

@jdjasperson

This comment has been minimized.

Copy link

commented Jan 18, 2016

Might there also be a solution to dynamically create the HZ cluster via service discovery ala Eureka, etc.? Docker support is paramount right now and is the difference between using Hazelcast vs other offerings.

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2016

hi @jdjasperson,

Hazelcast 3.6 has a discovery SPI you can use to implemented own discovery strategy. See this as an example: https://bitsofinfo.wordpress.com/2015/11/20/hazelcast-discovery-with-consul/

@bitsofinfo

This comment has been minimized.

Copy link

commented Mar 25, 2016

Note that using a discovery SPI, is not necessarily a work around for the original poster's issue (@jjongsma) i.e. "This node is not requested endpoint" warnings/errors when you are running an hazelcast app inside a docker container... You can publish whatever info you want to a discovery registry backend and obtain it all day, but if hazelcast rejects legit inbound NAT'ed connections via the docker host while running inside a container...well its an issue.

Can someone explain a workaround for this? Or a way to disable that rejection? I've read various posts (such as: https://groups.google.com/forum/#!msg/orient-database/ehfKcInWZXs/wxs8xLUOGB8J) that mention something like

Add <properties><property name="hazelcast.local.localAddress">IPV4ADDRESSHERE</property></properties> into hazelcast.xml
Add <network><public-address>IPV4ADDRESSHERE</public-address></network>

but where is this in the official docs

@bitsofinfo

This comment has been minimized.

Copy link

commented Mar 25, 2016

Even if I set the docker host IP as the hz interface that it binds to, because the docker port mapping is dynamic (i.e. container is listening on 5701) but the docker host maps a high port to 5701 you still get this annoyance... Wrong bind request from Address[192.168.99.100]:5701! This node is not requested endpoint: Address[192.168.99.100]:32886

whole point here is that all of this is totally dynamic. Launching a containerized hazelcast app in a swarm cluster for example. The goal here is dynamic up/down scaling of instances where one does not know the ip's and ports that other hz instances will be available at as they are allocated and mapped by the docker hosts.

You can certain glue all this together with discovery tools like the following:

https://github.com/bitsofinfo/docker-discovery-registrator-consul
https://github.com/bitsofinfo/hazelcast-consul-discovery-spi

However they are sort of if hazelcast itself rejects properly discovered, dynamically configured and legitimately nat'ed connections like this in a containerized environment.

seems not even a question of "docker support", hazelcast should try to be completely ignorant and unaware it is running inside or outside of a container, but just provide clear sets of options to workaround some of the baked in assumptions around networking.

From what I could dig up, setting this = false when hazelcast is in a containerized app....

-Dhazelcast.socket.server.bind.any=false -Dhazelcast.socket.bind.any=false

works to bypass the check below, however still not getting expected result, something else w/ the host gateway

if (ioService.isSocketBindAny() && !connection.isClient() && !thisAddress.equals(localEndpoint)) {

@bilalyasar

This comment has been minimized.

Copy link
Collaborator

commented Mar 25, 2016

@bitsofinfo you can find public-address configuration docs from here: hazelcast.local.localAddress property is not used anymore.

And your problem is that, when docker service starts, it creates docker0 interface. Then all hazelcast instances gets ip from docker0 or its another virtual interface.
Here my solutions:

  • i tried delete docker0 and started containers with flag --net=host flag.. It worked, now containers started to connect physical interface which is eth0 in my machine.
  • i tried to add bip property to docker daemon and started again. In bio property, i gave my eth0 inet address to bip. Also add flag --net=host again.
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -bip 172.30.0.49/24

172.30.0.49 is my private address of ec2 machine.

You can one of that solutions, then hazelcast instances will binds correct ip.

@bitsofinfo

This comment has been minimized.

Copy link

commented Mar 31, 2016

Unfortunately --net=host does not work with Registrator which is the core for the dynamic discovery we are attempting (see) gliderlabs/registrator#140

Also if I try to start multiple containers on the same host w/ --net and they all expose the same ports (regardless of mapping) you can only run one per host with errors like the below. (i.e. w/ a container scheduler where placement of multiple hazelcast app containers could be on same host, this breaks)

Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)

Without --net=host and each container on separate docker hosts:

When 2nd container starts, the 1st node that is already up on a separate docker host gets: Established socket connection between /172.17.0.2:5701 and /192.168.99.101:55276 , then followed by Connection [Address[172.17.0.2]:5701] lost. Reason: java.io.EOFException[Remote socket closed!]

On the 2nd container this is output: I just get "[3.6] Could not find a matching address to start with! Picking one of non-loopback addresses."

At the end of the day, I am at a loss as to how to get hazelcast to work with dynamically mapped ports, where multiple containers are on the same host or across separate hosts (again, without using --net=host) and avoiding this error:

if (ioService.isSocketBindAny() && !connection.isClient() && !thisAddress.equals(localEndpoint)) {

@peloncano

This comment has been minimized.

Copy link

commented Apr 5, 2016

+1

3 similar comments
@bmudda

This comment has been minimized.

Copy link

commented Apr 5, 2016

+1

@nfedyk

This comment has been minimized.

Copy link

commented Apr 5, 2016

+1

@gabadi

This comment has been minimized.

Copy link

commented Jun 5, 2016

+1

@eminn eminn assigned eminn and unassigned bilalyasar Jul 20, 2016

@eminn

This comment has been minimized.

Copy link
Collaborator

commented Jul 20, 2016

Hi All,

We already have a method to configure public and private addresses of a hazelcast instance. And if public address is configured for hazelcast instance running inside a docker container This node is not requested endpoint error will go away.

You can check out the documentation for configuring public address : http://docs.hazelcast.org/docs/latest-dev/manual/html-single/index.html#public-address

I also knew that our out of the box Docker experience is sub-optimal. We have implemented pluggable service discovery spi and community responded with various implementations like ZooKeeper, Eureka, etcd, consul. But we haven't utilized them in our docker images yet. I believe all the aforementioned problems will go away when we utilize them. So you could be able to run an arbitrary number of node hazelcast cluster in docker environment by just configuring the URL of the service discovery mechanism.

I took the opportunity and created an enhancement proposal ( hazelcast/hazelcast-docker#10 )to the hazelcast-docker repository.

I am going to close this issue, since we have a workaround for the reported case. For other suggestions, I suggest you to open new issues, since it is very hard for us to track suggestions from comments.

@eminn eminn closed this Jul 20, 2016

@bitsofinfo

This comment has been minimized.

Copy link

commented Jul 20, 2016

@eminn hmmm, have there been changes in the latest releases since base 3.6? See above, the workarounds did not work...

@eminn

This comment has been minimized.

Copy link
Collaborator

commented Jul 20, 2016

@bitsofinfo I've made the statement above based on my previous work on a similar issue long time ago, and as far as I remember the issue was resolved with public-address workaround. I'm setting up my environment to double check the suggested workaround solves the problem. I will update the issue as soon as possible if that is not working.

@bitsofinfo

This comment has been minimized.

Copy link

commented Jul 20, 2016

Thanks please just see all my followup comments above, #4537 (comment)

@eminn

This comment has been minimized.

Copy link
Collaborator

commented Jul 21, 2016

In my setup, I have a virtual machine uses interface of 192.168.59.103 and runs the docker daemon and my local interface is 192.168.2.68.

The virtual machine runs two docker containers which are using their private addresses 172.17.0.5 and 172.17.0.6. I started two docker containers configured to listen on a public address outside of the docker interface, which are 192.168.59.103:5701 and 192.168.59.103:5702 .

To start the first container :

hazelcast1.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<!--
    The default Hazelcast configuration. This is used when no hazelcast.xml is present.
    Please see the schema for how to configure Hazelcast at https://hazelcast.com/schema/config/hazelcast-config-3.6.xsd 
    or the documentation at https://hazelcast.org/documentation/
-->
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.6.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <group>
        <name>dev</name>
        <password>dev-pass</password>
    </group>
    <network>
        <public-address>192.168.59.103:5701</public-address>
        <join>
            <tcp-ip enabled="true">
            </tcp-ip>
        </join>
    </network>
    <partition-group enabled="false"/>


    <services enable-defaults="true"/>

    <lite-member enabled="false"/>

</hazelcast>


docker run -e JAVA_OPTS="-Dhazelcast.config=/configFolder/hazelcast1.xml" -v /Users/emindemirci/Desktop/:/configFolder -ti -p 5701:5701 hazelcast/hazelcast:3.6

To start the second container :

hazelcast2.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<!--
    The default Hazelcast configuration. This is used when no hazelcast.xml is present.
    Please see the schema for how to configure Hazelcast at https://hazelcast.com/schema/config/hazelcast-config-3.6.xsd 
    or the documentation at https://hazelcast.org/documentation/
-->
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.6.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <group>
        <name>dev</name>
        <password>dev-pass</password>
    </group>
    <network>
        <public-address>192.168.59.103:5702</public-address>
        <join>
            <tcp-ip enabled="true">
                <members>192.168.59.103:5701</members>
            </tcp-ip>
        </join>
    </network>
    <partition-group enabled="false"/>


    <services enable-defaults="true"/>

    <lite-member enabled="false"/>

</hazelcast>

docker run -e JAVA_OPTS="-Dhazelcast.config=/configFolder/hazelcast2.xml" -v /Users/emindemirci/Desktop/:/configFolder -ti -p 5702:5701 hazelcast/hazelcast:3.6

I also added a third node from my local (192.168.2.68) computer. Which just uses a tcp-ip configuration to connect nodes which are running inside a docker container.

localMember.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<!--
    The default Hazelcast configuration. This is used when no hazelcast.xml is present.
    Please see the schema for how to configure Hazelcast at https://hazelcast.com/schema/config/hazelcast-config-3.6.xsd 
    or the documentation at https://hazelcast.org/documentation/
-->
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.6.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <group>
        <name>dev</name>
        <password>dev-pass</password>
    </group>
    <network>
        <join>
        <multicast enabled="false"></multicast>
        <tcp-ip enabled="true">
                <interface>192.168.2.68</interface>
                <members>192.168.59.103:5701</members>          
                <members>192.168.59.103:5702</members>          
        </tcp-ip>
        </join>
    </network>
    <partition-group enabled="false"/>

    <services enable-defaults="true"/>

    <lite-member enabled="false"/>

</hazelcast>

Logs from first container (172.17.0.5) :

$ docker run -e JAVA_OPTS="-Dhazelcast.config=/configFolder/hazelcast1.xml" -v /Users/emindemirci/Desktop/:/configFolder -ti -p 5701:5701 hazelcast/hazelcast:3.6
########################################
# RUN_JAVA=
# JAVA_OPTS=-Dhazelcast.config=/configFolder/hazelcast1.xml
# starting now....
########################################
Process id for hazelcast instance is written to location:  /opt/hazelcast/hazelcast_instance.pid
Jul 21, 2016 8:16:17 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading configuration /configFolder/hazelcast1.xml from System property 'hazelcast.config'
Jul 21, 2016 8:16:17 AM com.hazelcast.config.XmlConfigLocator
INFO: Using configuration file at /configFolder/hazelcast1.xml
Jul 21, 2016 8:16:17 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: []
Jul 21, 2016 8:16:17 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Prefer IPv4 stack is true.
Jul 21, 2016 8:16:17 AM com.hazelcast.instance.DefaultAddressPicker
WARNING: [LOCAL] [dev] [3.6] Could not find a matching address to start with! Picking one of non-loopback addresses.
Jul 21, 2016 8:16:17 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Picked Address[172.17.0.5]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Jul 21, 2016 8:16:17 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Using public address: Address[192.168.59.103]:5701
Jul 21, 2016 8:16:17 AM com.hazelcast.spi.OperationService
INFO: [192.168.59.103]:5701 [dev] [3.6] Backpressure is disabled
Jul 21, 2016 8:16:17 AM com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor
INFO: [192.168.59.103]:5701 [dev] [3.6] Starting with 2 generic operation threads and 4 partition operation threads.
Jul 21, 2016 8:16:18 AM com.hazelcast.system
INFO: [192.168.59.103]:5701 [dev] [3.6] Hazelcast 3.6 (20160125 - 13fea63) starting at Address[192.168.59.103]:5701
Jul 21, 2016 8:16:18 AM com.hazelcast.system
INFO: [192.168.59.103]:5701 [dev] [3.6] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
Jul 21, 2016 8:16:18 AM com.hazelcast.system
INFO: [192.168.59.103]:5701 [dev] [3.6] Configured Hazelcast Serialization version : 1
Jul 21, 2016 8:16:18 AM com.hazelcast.instance.Node
INFO: [192.168.59.103]:5701 [dev] [3.6] Creating TcpIpJoiner
Jul 21, 2016 8:16:18 AM com.hazelcast.core.LifecycleService
INFO: [192.168.59.103]:5701 [dev] [3.6] Address[192.168.59.103]:5701 is STARTING
Jul 21, 2016 8:16:18 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
INFO: [192.168.59.103]:5701 [dev] [3.6] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
Jul 21, 2016 8:16:18 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [192.168.59.103]:5701 [dev] [3.6] 


Members [1] {
    Member [192.168.59.103]:5701 this
}

Jul 21, 2016 8:16:18 AM com.hazelcast.core.LifecycleService
INFO: [192.168.59.103]:5701 [dev] [3.6] Address[192.168.59.103]:5701 is STARTED
^[[AJul 21, 2016 8:16:26 AM com.hazelcast.nio.tcp.SocketAcceptorThread
INFO: [192.168.59.103]:5701 [dev] [3.6] Accepting socket connection from /172.17.42.1:40949
Jul 21, 2016 8:16:26 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.59.103]:5701 [dev] [3.6] Established socket connection between /172.17.0.5:5701 and /172.17.42.1:40949
Jul 21, 2016 8:16:33 AM com.hazelcast.cluster.ClusterService
INFO: [192.168.59.103]:5701 [dev] [3.6] 

Members [2] {
    Member [192.168.59.103]:5701 this
    Member [192.168.59.103]:5702
}

Jul 21, 2016 8:16:49 AM com.hazelcast.nio.tcp.SocketAcceptorThread
INFO: [192.168.59.103]:5701 [dev] [3.6] Accepting socket connection from /192.168.59.3:57447
Jul 21, 2016 8:16:49 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.59.103]:5701 [dev] [3.6] Established socket connection between /172.17.0.5:5701 and /192.168.59.3:57447
Jul 21, 2016 8:16:55 AM com.hazelcast.cluster.ClusterService
INFO: [192.168.59.103]:5701 [dev] [3.6] 

Members [3] {
    Member [192.168.59.103]:5701 this
    Member [192.168.59.103]:5702
    Member [192.168.2.68]:5701
}

Jul 21, 2016 8:16:57 AM com.hazelcast.partition.InternalPartitionService
INFO: [192.168.59.103]:5701 [dev] [3.6] Initializing cluster partition table arrangement...

Logs from second container (172.17.0.6) :

$ docker run -e JAVA_OPTS="-Dhazelcast.config=/configFolder/hazelcast2.xml" -v /Users/emindemirci/Desktop/:/configFolder -ti -p 5702:5701 hazelcast/hazelcast:3.6
########################################
# RUN_JAVA=
# JAVA_OPTS=-Dhazelcast.config=/configFolder/hazelcast2.xml
# starting now....
########################################
Process id for hazelcast instance is written to location:  /opt/hazelcast/hazelcast_instance.pid
Jul 21, 2016 8:16:25 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading configuration /configFolder/hazelcast2.xml from System property 'hazelcast.config'
Jul 21, 2016 8:16:25 AM com.hazelcast.config.XmlConfigLocator
INFO: Using configuration file at /configFolder/hazelcast2.xml
Jul 21, 2016 8:16:25 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [192.168.59.103]
Jul 21, 2016 8:16:25 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Prefer IPv4 stack is true.
Jul 21, 2016 8:16:25 AM com.hazelcast.instance.DefaultAddressPicker
WARNING: [LOCAL] [dev] [3.6] Could not find a matching address to start with! Picking one of non-loopback addresses.
Jul 21, 2016 8:16:25 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Picked Address[172.17.0.6]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Jul 21, 2016 8:16:25 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Using public address: Address[192.168.59.103]:5702
Jul 21, 2016 8:16:26 AM com.hazelcast.spi.OperationService
INFO: [192.168.59.103]:5702 [dev] [3.6] Backpressure is disabled
Jul 21, 2016 8:16:26 AM com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor
INFO: [192.168.59.103]:5702 [dev] [3.6] Starting with 2 generic operation threads and 4 partition operation threads.
Jul 21, 2016 8:16:26 AM com.hazelcast.system
INFO: [192.168.59.103]:5702 [dev] [3.6] Hazelcast 3.6 (20160125 - 13fea63) starting at Address[192.168.59.103]:5702
Jul 21, 2016 8:16:26 AM com.hazelcast.system
INFO: [192.168.59.103]:5702 [dev] [3.6] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
Jul 21, 2016 8:16:26 AM com.hazelcast.system
INFO: [192.168.59.103]:5702 [dev] [3.6] Configured Hazelcast Serialization version : 1
Jul 21, 2016 8:16:26 AM com.hazelcast.instance.Node
INFO: [192.168.59.103]:5702 [dev] [3.6] Creating TcpIpJoiner
Jul 21, 2016 8:16:26 AM com.hazelcast.core.LifecycleService
INFO: [192.168.59.103]:5702 [dev] [3.6] Address[192.168.59.103]:5702 is STARTING
Jul 21, 2016 8:16:26 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
INFO: [192.168.59.103]:5702 [dev] [3.6] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
Jul 21, 2016 8:16:26 AM com.hazelcast.nio.tcp.InitConnectionTask
INFO: [192.168.59.103]:5702 [dev] [3.6] Connecting to /192.168.59.103:5701, timeout: 0, bind-any: true
Jul 21, 2016 8:16:26 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.59.103]:5702 [dev] [3.6] Established socket connection between /172.17.0.6:36282 and /192.168.59.103:5701
Jul 21, 2016 8:16:33 AM com.hazelcast.cluster.ClusterService
INFO: [192.168.59.103]:5702 [dev] [3.6] 

Members [2] {
    Member [192.168.59.103]:5701
    Member [192.168.59.103]:5702 this
}

Jul 21, 2016 8:16:35 AM com.hazelcast.core.LifecycleService
INFO: [192.168.59.103]:5702 [dev] [3.6] Address[192.168.59.103]:5702 is STARTED
Jul 21, 2016 8:16:49 AM com.hazelcast.nio.tcp.SocketAcceptorThread
INFO: [192.168.59.103]:5702 [dev] [3.6] Accepting socket connection from /192.168.59.3:57449
Jul 21, 2016 8:16:49 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.59.103]:5702 [dev] [3.6] Established socket connection between /172.17.0.6:5701 and /192.168.59.3:57449
Jul 21, 2016 8:16:55 AM com.hazelcast.cluster.ClusterService
INFO: [192.168.59.103]:5702 [dev] [3.6] 

Members [3] {
    Member [192.168.59.103]:5701
    Member [192.168.59.103]:5702 this
    Member [192.168.2.68]:5701
}

Logs from local machine (192.168.2.68)

$ ./console.sh 
Jul 21, 2016 11:16:49 AM com.hazelcast.config.FileSystemXmlConfig
INFO: Configuring Hazelcast from '/Users/emindemirci/Downloads/hazelcast-3.6/demo/hazelcast.xml'.
Jul 21, 2016 11:16:49 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [192.168.2.68, 192.168.59.103]
Jul 21, 2016 11:16:49 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Prefer IPv4 stack is true.
Jul 21, 2016 11:16:49 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6] Picked Address[192.168.2.68]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Jul 21, 2016 11:16:49 AM com.hazelcast.spi.OperationService
INFO: [192.168.2.68]:5701 [dev] [3.6] Backpressure is disabled
Jul 21, 2016 11:16:49 AM com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor
INFO: [192.168.2.68]:5701 [dev] [3.6] Starting with 4 generic operation threads and 8 partition operation threads.
Jul 21, 2016 11:16:50 AM com.hazelcast.system
INFO: [192.168.2.68]:5701 [dev] [3.6] Hazelcast 3.6 (20160125 - 13fea63) starting at Address[192.168.2.68]:5701
Jul 21, 2016 11:16:50 AM com.hazelcast.system
INFO: [192.168.2.68]:5701 [dev] [3.6] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
Jul 21, 2016 11:16:50 AM com.hazelcast.system
INFO: [192.168.2.68]:5701 [dev] [3.6] Configured Hazelcast Serialization version : 1
Jul 21, 2016 11:16:50 AM com.hazelcast.instance.Node
INFO: [192.168.2.68]:5701 [dev] [3.6] Creating TcpIpJoiner
Jul 21, 2016 11:16:50 AM com.hazelcast.core.LifecycleService
INFO: [192.168.2.68]:5701 [dev] [3.6] Address[192.168.2.68]:5701 is STARTING
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
INFO: [192.168.2.68]:5701 [dev] [3.6] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.InitConnectionTask
INFO: [192.168.2.68]:5701 [dev] [3.6] Connecting to /192.168.59.103:5702, timeout: 0, bind-any: true
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.InitConnectionTask
INFO: [192.168.2.68]:5701 [dev] [3.6] Connecting to /192.168.2.68:5703, timeout: 0, bind-any: true
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.InitConnectionTask
INFO: [192.168.2.68]:5701 [dev] [3.6] Connecting to /192.168.59.103:5701, timeout: 0, bind-any: true
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.InitConnectionTask
INFO: [192.168.2.68]:5701 [dev] [3.6] Connecting to /192.168.2.68:5702, timeout: 0, bind-any: true
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.InitConnectionTask
INFO: [192.168.2.68]:5701 [dev] [3.6] Could not connect to: /192.168.2.68:5703. Reason: SocketException[Connection refused to address /192.168.2.68:5703]
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.InitConnectionTask
INFO: [192.168.2.68]:5701 [dev] [3.6] Could not connect to: /192.168.2.68:5702. Reason: SocketException[Connection refused to address /192.168.2.68:5702]
Jul 21, 2016 11:16:50 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [192.168.2.68]:5701 [dev] [3.6] Address[192.168.2.68]:5703 is added to the blacklist.
Jul 21, 2016 11:16:50 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [192.168.2.68]:5701 [dev] [3.6] Address[192.168.2.68]:5702 is added to the blacklist.
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.2.68]:5701 [dev] [3.6] Established socket connection between /192.168.59.3:57449 and /192.168.59.103:5702
Jul 21, 2016 11:16:50 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.2.68]:5701 [dev] [3.6] Established socket connection between /192.168.59.3:57447 and /192.168.59.103:5701
Jul 21, 2016 11:16:56 AM com.hazelcast.cluster.ClusterService
INFO: [192.168.2.68]:5701 [dev] [3.6] 

Members [3] {
    Member [192.168.59.103]:5701
    Member [192.168.59.103]:5702
    Member [192.168.2.68]:5701 this
}


Jul 21, 2016 11:16:58 AM com.hazelcast.core.LifecycleService
INFO: [192.168.2.68]:5701 [dev] [3.6] Address[192.168.2.68]:5701 is STARTED
hazelcast[default] > hazelcast[default] > 
hazelcast[default] > m.putmany 1000
size = 1000, 1225 evt/s, 957 Kbit/s, 97 KB added
hazelcast[default] > 

It appears that I can create a cluster by using public address configuration, without using --net=host flag or is there anything that I am missing ?

@bitsofinfo

This comment has been minimized.

Copy link

commented Jul 21, 2016

Thanks for the info, example above. The scenario I was having a ton of problems with was a scenario where nothing was hardwired in pre-existing configurations per your example, nor were the IP's and ports "pre-known".

Rather each container would register and both discover both it's host/port mapping related info and that of its peers, (via consul) then take this info and feed it into hazelcast via the discovery spi. From the comments above you can see the issues/errors we were running into.

The challenge was basically getting a hazelcast application to properly work in an orchestrated container environment where containers could be placed on any host and have no-pre-awareness of anything, and the port mappings are totally dynamic. Containers can come up and down randomly and be placed/moved to any host without warning.

I have not re-visited this in sometime, perhaps something else was fixed, will have to try it again. I just saw this issue being closed with no clear resolution and I just know this caused a lot of headaches for us. (will be revisiting this soon)

@eminn

This comment has been minimized.

Copy link
Collaborator

commented Jul 21, 2016

Hi @bitsofinfo ,

Thanks for your comment. I understand your use case, I will try to create a setup to see where we are standing in that scenario.

But I've think that is a different scenario and should be in scope of this : hazelcast/hazelcast-docker#10

So, I think, as far as host ip addresses are known and hazelcast nodes configured to listen on those public host addresses without enabling --net=host flag, it works, even though that configuration process is a pain.

@bitsofinfo

This comment has been minimized.

Copy link

commented Jul 21, 2016

Ok, I'll take the conversation over to that issue when I revisit this whole thing which will be in the next few months.

@gsaslis

This comment has been minimized.

Copy link

commented Sep 21, 2016

Just a quick note to say I also crashed into the same wall here. In my case (running hazelcast on AWS ECS, with an Auto-Scaling Group for the container hosts), the host IPs are NOT known beforehand, so I can't see how above solution would work me around the This node is not requested endpoint issue... : /

Note:
I did use the --net==host flag, but no luck there either. My error message is:
WARNING: [172.17.0.1]:5701 [abcdef] [3.7.1] Wrong bind request from [172.17.0.1]:5701! This node is not requested endpoint: [172.31.18.96]:5701

@bmudda

This comment has been minimized.

Copy link

commented Sep 21, 2016

+1

@nfedyk

This comment has been minimized.

Copy link

commented Oct 8, 2016

+1

@jacohend

This comment has been minimized.

Copy link

commented Oct 14, 2016

+1

jjongsma opened this issue on Feb 4, 2015

@jeacott1

This comment has been minimized.

Copy link

commented Oct 26, 2016

@eminn, try your config using hostnames instead of ip's for the network join members.
for me works fine with ips, fails miserably with domain names.

@pankajtakawale

This comment has been minimized.

Copy link

commented Nov 2, 2016

+1

@bitsofinfo

This comment has been minimized.

Copy link

commented Nov 4, 2016

All, well its been months and we are now revisiting this, ran into this again, perhaps summarized here: #9219

@bitsofinfo

This comment has been minimized.

Copy link

commented Jun 21, 2017

@bitsofinfo

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.