EC2 Auto discovery and WAN replication with public ip addresses.

[fuadm]

When we use EC2 Auto discovery, Hazelcast uses private ip addresses. But when we set WAN replication across regions it is not able to communicate over private IP addresses. An option is to force hazelcast to bind to public ip by setting

ec2-xx.us-west-1.compute.amazonaws.com:5701

or

ec2-54-193-63-117.us-west-1.compute.amazonaws.com ec2-54-193-63-117.us-west-1.compute.amazonaws.com

but this requires

1. static setting of IP
2. it brakes the EC2 discovery as it uses the private ip's to discover.

So both EC2 discovery and WAN replication needs to be fixed to make the process clean and seamless.

[imranbohoran]

Is this something that can be expected to be fixed any time soon?

Assuming that the following error message is caused by this issue;
[3.2] Wrong bind request from Address[private-ip]:5701! This node is not requested endpoint: Address[public-ip]:5701

Code related to this being; TcpIpConnectionManager.bind() method.

On a WAN replication scenario, does it actually make sense to do the check that results this. From what I've seen this occurs in the replicating cluster, which gets the private (in aws)/local (NAT) IP of the sender.
I assume the current check in the bind() method is a safety net to make sure replication event is from the source it was intended from. Does it make sense to let the Node make aware of the public IP of it self as well, so that secondary check can be done against the public IP if the IP from IOService.getThisAddress() fails.
I believe this is the same issue raised here - #370

This currently means replicating between 2 aws regions is not possible. While EC2 discovery is quite useful and is the primary replication that would be needed for a cluster, having cross region back-ups is useful for those dreadful moments where a whole region is not accessible.

[pveentjer]

I believe for 3.3 or 3.4 AWS fixes are planned. The client is suffering from exactly the same problems.

[jjongsma]

This is also a big issue for Docker containerization (which should maybe be a separately tracked issue). Unless you configure containers to net=host and explicitly tell Hazelcast to bind to the host's IP, Hazelcast listens on a Docker-local IP that is port-mapped to the host, and every connection request is rejected with the same "This node is not requested endpoint" error because the host's IP that other nodes connect to doesn't match what Hazelcast sees inside the container.

Host networking mode is a workaround, but requiring that and explicit IP ranges for Hazelcast to listen on is not scalable for a containerized environment. At minimum there should probably be an option to configure separate private and public node addresses, which is how Cassandra addresses this issue (listen vs. broadcast). I'd prefer disabling this check completely though - there are other ways to validate node identity if that is a concern (SSL certs, etc).

[sbuettner]

We are seeing the same issue as described by @jjongsma when trying to deploy Hazelcast inside AWS Elastic Beanstalk using Docker.

2015-06-20 11:19:36.302  INFO 1 --- [thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptor     : [172.17.0.2]:5701 [dev] [3.5] Accepting socket connection from /10.0.1.209:41004
2015-06-20 11:19:36.303  INFO 1 --- [        cached5] c.h.nio.tcp.TcpIpConnectionManager       : [172.17.0.2]:5701 [dev] [3.5] Established socket connection between /172.17.0.2:5701
2015-06-20 11:19:36.303  WARN 1 --- [.IO.thread-in-1] c.h.nio.tcp.TcpIpConnectionManager       : [172.17.0.2]:5701 [dev] [3.5] Wrong bind request from Address[172.17.0.2]:5701! This node is not requested endpoint: Address[10.0.1.227]:5701
2015-06-20 11:19:36.304  INFO 1 --- [.IO.thread-in-1] com.hazelcast.nio.tcp.TcpIpConnection    : [172.17.0.2]:5701 [dev] [3.5] Connection [/10.0.1.209:41004] lost. Reason: Socket explicitly closed

[sbuettner]

Amazon provides a meta-data service on each instance which seems to be also accessible from inside docker containers. This service can be used to get the public ip address of the ec2 instance. Since hazelcast already provides tight integration with aws it could use this service when a certain config flag has been set to get this address.

[mmedenjak]

There are docker issues encountering the same problem. Some of them:
#10801
#9219
We are working on adding a new SPI to 3.9 which will allow the instance to "discover" and set it's own public address without it explicitly being set by the user.

[mmedenjak]

@fuadm @imranbohoran @pveentjer @jjongsma @sbuettner there is a new SPI that has just been merged and which will be released in the 3.9 version. It will allow you to define the bind and public address when starting the hazelcast instance and if used correctly it can be used to avoid issues when forming a cluster.
For now you will have to use the SPI yourself and write an implementation which will fix your issue but we are planning on releasing implementations of our own which will be bundled into plugins such as the docker or AWS plugin for easier deployment.
Please check out the new SPI:
https://github.com/hazelcast/hazelcast/blob/3cede71cad1fe87312f0901ff77f903ed2d4383d/hazelcast/src/main/java/com/hazelcast/spi/MemberAddressProvider.java
Please create a new issue or reopen this one if this does not suit your use case.