Autowire should only use addresses that exist on an interface

[chrismccracken]

Follow-up to #6709

https://github.com/dotCMS/enterprise-2.x/blob/626f6b321d9aac6c52d0bc96b9ce947aa89a81c3/src/com/dotcms/enterprise/cluster/ClusterFactory.java#L241
InetAddress.getLocalHost() can use DNS resolution to find an IP matching the hostname depending on system name resolution (defaults to hosts file, then DNS). Due to DNS and other factors, this can return an address that does not exist on any interface and cache/ES binds will fail. We should be verifying that the address exists on an interface no matter what. If it doesn't exist, pick the first usable (non-local, non-link-layer) address from an interface.

http://docs.oracle.com/javase/7/docs/api/java/net/InetAddress.html#getLocalHost()
"Returns the address of the local host. This is achieved by retrieving the name of the host from the system, then resolving that name into an InetAddress."
"Host name-to-IP address resolution is accomplished through the use of a combination of local machine configuration information and network naming services such as the Domain Name System (DNS) and Network Information Service(NIS). The particular naming services(s) being used is by default the local machine configured one."

Ran into this issue at http://jira.dotmarketing.net/browse/DOTCMSINC-1245

For testing we need to ensure that we can reproduce the issue. I think elastic cloud is the best place to do this. Let's get with @chrismccracken to help here.

@DeanGonzalez we need to doc the tested example here.

[dsilvam]

Main effort of this issue would be in testing it. Will get with Chris to figure this out.

[dsilvam]

PR core: #9101
PR enterprise: https://github.com/dotCMS/enterprise-2.x/pull/368

[wezell]

I don't know if this is an issue or not but I still see this starting up dotCMS

INFO  2016-05-29 13:05:18,364 com.dotmarketing.startup.StartupTasksExecutor:info(54): Running: Task00009ClusterInitialize
INFO  2016-05-29 13:05:18,382 com.dotmarketing.business.ChainableCacheAdministratorImpl:info(54): ***    Starting JGroups Cluster Setup
INFO  2016-05-29 13:05:18,382 com.dotmarketing.business.ChainableCacheAdministratorImpl:info(54): Using automatic port placement as CLUSTER_AUTOWIRE is ON
INFO  2016-05-29 13:05:18,384 com.dotmarketing.business.ChainableCacheAdministratorImpl:info(54): ***    Using 192.168.86.107 as the bindaddress
INFO  2016-05-29 13:05:18,384 com.dotmarketing.business.ChainableCacheAdministratorImpl:info(54): ***    Using 7802 as the bindport
INFO  2016-05-29 13:05:18,384 com.dotmarketing.business.ChainableCacheAdministratorImpl:info(54): ***    Setting up TCP initial hosts: 192.168.86.107[7802]
INFO  2016-05-29 13:05:18,384 com.dotmarketing.business.ChainableCacheAdministratorImpl:info(54): ***    Prefer IPv4: enabled
INFO  2016-05-29 13:05:18,385 com.dotmarketing.business.jgroups.JGroupsCacheTransport:info(54): ***  Setting up JChannel
INFO  2016-05-29 13:05:18,385 com.dotmarketing.business.jgroups.JGroupsCacheTransport:info(54): ***  Going to load JGroups with this Classpath file cache-jgroups-tcp.xml
WARN  2016-05-29 13:05:18,578 com.dotcms.repackage.org.jgroups.stack.Configurator:warn(79): JGRP000014: TP.loopback has been deprecated: enabled by default
WARN  2016-05-29 13:05:18,579 com.dotcms.repackage.org.jgroups.stack.Configurator:warn(79): JGRP000014: TP.discard_incompatible_packets has been deprecated: incompatible packets are discarded anyway
WARN  2016-05-29 13:05:18,580 com.dotcms.repackage.org.jgroups.stack.Configurator:warn(79): JGRP000014: TP.enable_bundling has been deprecated: will be ignored as bundling is on by default
WARN  2016-05-29 13:05:18,586 com.dotcms.repackage.org.jgroups.stack.Configurator:warn(79): JGRP000014: Discovery.timeout has been deprecated: GMS.join_timeout should be used instead
WARN  2016-05-29 13:05:18,587 com.dotcms.repackage.org.jgroups.stack.Configurator:warn(79): JGRP000014: Discovery.num_initial_members has been deprecated: will be ignored
WARN  2016-05-29 13:05:18,610 com.dotcms.repackage.org.jgroups.stack.Configurator:warn(79): JGRP000014: UNICAST.setTimeout has been deprecated: not used anymore
INFO  2016-05-29 13:05:18,670 com.dotcms.repackage.org.jgroups.protocols.UFC:info(67): UFC is not needed (and can be removed) as we're running on a TCP transport
WARN  2016-05-29 13:05:18,686 com.dotcms.repackage.org.jgroups.protocols.TCP:warn(75): failed to join /224.0.75.75:7500 on awdl0: java.net.SocketException: Can't assign requested address  <-----------------

-------------------------------------------------------------------
GMS: address=cabbage-39822, cluster=dotCMSCluster, physical address=192.168.86.107:7802
-------------------------------------------------------------------
INFO  2016-05-29 13:05:21,710 com.dotmarketing.business.jgroups.JGroupsCacheTransport:info(54): Method view: Cluster View is : [cabbage-39822|0] (1) [cabbage-39822]
INFO  2016-05-29 13:05:21,710 com.dotmarketing.business.ChainableCacheAdministratorImpl:info(62): viewAccepted + Cluster View is : [cabbage-39822|0] (1) [cabbage-39822]
INFO  2016-05-29 13:05:21,722 com.dotmarketing.business.jgroups.JGroupsCacheTransport:info(54): ***  local_addr=cabbage-39822
cluster_name=dotCMSCluster