ZOOKEEPER-3758: Leader reachability check fails with single address

[symat]

Since ZooKeeper 3.6.0 we can specify multiple addresses for each ZooKeeper server instance
(this can increase availability when multiple physical network interfaces can be used parallel
in the cluster). ZooKeeper will perform ICMP ECHO requests or try to establish a TCP connection
on port 7 (Echo) of the destination host in order to find the reachable addresses. This should
happen only if the user provide multiple addresses in the configuration, in a single address is
used then ZooKeeper shouldn’t send any ICMP requests.

This works as we expected for the leader election connections, but in this Jira issue we found
a bug when the reachability check was performed even with a single address when the Follower
tries to connect to the newly elected Leader.

The fix follows the same approach we discussed for the election protocol before (see
ZOOKEEPER-3698). We avoid the reachability check for single addresses. Also when we have
multiple addresses and none of them can be reached, we still start to connect to all addresses
in parallel.

[eolivelli]

Isn't this disabled by default?

[symat]

no, unfortunately not. There are two separate parameters:

• multiAddress.enabled : this is for enabling/disabling the whole multiaddress feature, this is set to false to default
• multiAddress.reachabilityCheckEnabled : this one can be used to disable the ICMP messages when multiple addresses are used. But this is enabled by default, as disabling it actually makes the multiAddress recovery unreliable (as ZK will not really be able to select among the addresses)

There is already a logic in the MultiAddress.getReachableOrOne() method which will skip the ICMP check if there is only a single address is provided:

zookeeper/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/MultipleAddresses.java

Lines 151 to 158 in a5a4743

	public InetSocketAddress getReachableOrOne() {
	InetSocketAddress address;
	// if there is only a single address provided then we don't do any reachability check
	if (addresses.size() == 1) {
	return getOne();
	}

basically the same logic was missing from the MultipleAddresses.getAllReachableAddresses method.

I should have thinking of it, but forget when I fixed ZOOKEEPER-3698.

[eolivelli]

@nkalmar this is a good candidate for 3.6.1

please take a look

[nkalmar]

LGTM

[eolivelli]

LGTM

@nkalmar can you please merge ?

[nkalmar]

I messed up my python env, and the script is not running, I'll try to fix it. I can merge after.

[symat]

@eolivelli @nkalmar thanks for checking this so quickly!

[nkalmar]

pushed to master and branch-3.6, thanks @symat