Fully resolve and randomize the client host list at connect time #372
Conversation
|
FYI - There appears to be a problem with the tests. The Zookeeper mirrors have removed 3.4.6 from the repository, and this is causing huge problems. |
|
Hmmm, that's not good. |
|
OK, I got this to work by changing the URL for pulling ZK to point to archive.apache.org, which apparently where the older versions are being stored. May want to separately consider changing the tests to use 3.4.7 instead of 3.4.6, at which point the script can be repointed to a mirror. Also, apologize for the commit spam on this to get it done. My git-fu is not mighty. |
|
Thanks much (for also fixing the zk archive location to). |
| for rhost in socket.getaddrinfo(host.strip(), port, 0, 0, socket.IPPROTO_TCP): | ||
| host_ports.append((rhost[4][0], rhost[4][1])) | ||
| except socket.gaierror: | ||
| # Skip hosts that don't resolve |
There was a problem hiding this comment.
What do u think about logging this? Seems like a warning would at least be useful (as someone giving an unresolveable host seems not that good).
|
Any status on this? I'd love for this to clear, I'm using project that relies on this. I'd be happy to pitch in! |
|
Sorry about that... all for the want of a log statement! I've updated the PR with the logging (good idea), but I'm having problems with intermittent failures in Travis, of course. Nothing related to this code. I'll try and get a clean test. |
|
Apologize for all the email spam on this as I tried to get a clean set of tests. I finally did. Can we merge this one now? |
|
Hey @harlowja, can we get this merged? We're now running into some serious issues even with our workaround internally because of IPv6 (and this would resolve it) |
|
LGTM |
When using a round-robin DNS record set for a Zookeeper ensemble (i.e. a single A record that points to all the IP addresses of the servers in the ensemble), we found that Kazoo was always connecting to the same server. It appears that the underlying socket calls that Kazoo uses to set up connections do not rotate the addresses in a round robin properly (if at all).
This is a refactor of PR #360, which resolved the hosts at client setup time. As pointed out, that can generate some issues if the DNS records change while an application is running. A cleaner solution, that more closely matches the current behavior, is to expand the host list in the same way at connect time. In order to do this properly, randomization of the host list was moved to connect time as well. Otherwise we could end up with one host in the hostlist that is a DNS round robin, and the connect loop would always pick the first item in the RRset.