[dns] failure to resolve some names

[awh]

Observed on a Weave 0.10.0 test system owned by @yaronr:

• Three CoreOS hosts running weave and weaveDNS
• A single subnet (10.0.0.0/8) used for both weaveDNS and application containers
• Three named application containers (zoo1, zoo2, zoo3) running Zookeeper, using weaveDNS for discovery

The problem manifested as Zookeeper nodes being unable to peer with each other correctly due to name resolution errors. The following diagnostic steps were performed:

• weave status on each host confirmed that weave had peered correctly, and that the correct names were registered in each weaveDNS peer
• docker exec was used to confirm that the resolv.conf was configured correctly in each Zookeeper container, that weaveDNS was reachable, and that the other Zookeeper containers were reachable
• ip netns exec was used to confirm IP reachability between the weaveDNS containers
• docker exec getent was used to perform lookups inside each Zookeeper container. The following results were observed:
  • zoo1 could resolve zoo3 but not zoo2
  • zoo3 could resolve zoo1 and zoo2
  • zoo3 could resolve zoo2 but not zoo1

Furthermore, in the course of our investigation, the problem resolved itself and did not recur. The only change we had deliberately made in the meantime was to shut down some other containers that were generating DNS traffic that was hampering our ability to analyse the weaveDNS logs in real time. ps confirmed that none of the weaveDNS containers had been restarted.

[inercia]

@awh Do you have any logs we could use from the WeaveDNS containers?

[awh]

Unfortunately not, we were in the process of shutting down other containers to obtain some clean traces when the problem resolved itself.

[awh]

Closing due to inability to reproduce; it's likely that this defect is obsoleted by the upcoming mDNS->gossip transition anyway.

[yaronr]

Update:
Probably related to https://issues.apache.org/jira/browse/ZOOKEEPER-1506