Mac OSX resolver domains do not resolve when monkey_patch applied

[coffeegist]

Versions Effected

$ python --version
Python 2.7.13

$ pip show eventlet
Name: eventlet
Version: 0.20.1
Summary: Highly concurrent networking library
Home-page: http://eventlet.net
Author: Linden Lab
Author-email: eventletdev@lists.secondlife.com
License: UNKNOWN
Location: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requires: enum-compat, greenlet

Scenario

I am using a private DNS server to resolve database hostnames in our production environment. When I use eventlet.monkey_patch(), these hostnames no longer resolve.

To set this up for replication, I installed a local DNS server to resolve the hostname somewherespecial.tech to 127.0.0.1. The specific commands I ran are listed below:

$ sudo -H port install dnsmasq
$ sudo mkdir /etc/resolver
$ echo -n "nameserver 127.0.0.1" | sudo tee /etc/resolver/tech
$ echo "address=/somewherespecial.tech/127.0.0.1" | sudo tee -a /opt/local/etc/dnsmasq.conf
$ sudo dnsmasq

# Verify everything is working
$ ping somewherespecial.tech
PING somewherespecial.tech (127.0.0.1): 56 data bytes

$ ping coderunner.tech # this is a real website, shows that it's unaffected
PING coderunner.tech (68.65.122.145): 56 data bytes

$ dig +short @localhost somewherespecial.tech
127.0.0.1

$ scutil --dns
.
.
.
resolver #13
  domain   : tech
  nameserver[0] : 127.0.0.1
  flags    : Request A records, Request AAAA records
Reachable, Local Address, Directly Reachable Address

Example

The following code shows the change in resolution that monkey_patching causes.

$ python
Python 2.7.13 (default, Dec 18 2016, 05:35:35) 
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import eventlet
>>> import socket
>>> print "somewherespecial.tech -> " + socket.gethostbyname('somewherespecial.tech')
>>> somewherespecial.tech -> 127.0.0.1
>>> print "coderunner.tech -> " + socket.gethostbyname('coderunner.tech')
>>> coderunner.tech -> 68.65.122.145

>>> eventlet.monkey_patch()
>>> print "somewherespecial.tech -> " + socket.gethostbyname('somewherespecial.tech')
>>> Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/eventlet/support/greendns.py", line 518, in gethostbyname
    rrset = resolve(hostname)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/eventlet/support/greendns.py", line 405, in resolve
    raise EAI_NODATA_ERROR
socket.gaierror: [Errno 7] No address associated with hostname

>>> print "coderunner.tech -> " + socket.gethostbyname('coderunner.tech')
>>> coderunner.tech -> 68.65.122.145

I'm more than willing to provide any other info that might be useful.

[temoto]

python -c 'import eventlet ; print(eventlet.__version__)'

[coffeegist]

$ python -c 'import eventlet ; print(eventlet.__version__)'
0.20.1

[temoto]

It's because patched version of socket module uses custom resolver which was trained to look into /etc/resolv.conf but not in /etc/resolver/tech. Thanks for lesson, I didn't know that OSX uses another resolvers configuration file tree.

I'm still studying how it works in OSX and will implement this.

[coffeegist]

Sure thing! Thanks for getting back so fast.

For anyone in the future that stumbles across this and needs a refresher on how /etc/resolver works, here's an example.

/etc/resolver/tech:

nameserver 127.0.0.1

This tells us that for every *.tech domain requested, it should use the nameserver located at 127.0.0.1 to do the resolution.

[temoto]

Linking similar issue in Go, just for fun: golang/go#12524

[temoto]

Okay, this is probably your best option for today

pip install https://github.com/eventlet/eventlet/archive/f9e3ff7f06c4e099795efe6660019fa6c6d848f3.zip

Run python with EVENTLET_NO_GREENDNS=yes environment variable. Seems like it's a development thing anyway, so blocking resolving wouldn't hurt that much.

A better solution is yet to come, I'm tired.

[coffeegist]

Thanks for the hard work. Interestingly enough, this solved it for my example, but not for our actual production issue. I'm having a difficult time recreating this configuration for a test case. I know our production development setup does work with gevent's monkey.patch_all(). I need to figure out a way to get you an even better example. Until then, I can offer this bit of information from the error:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
    worker.init_process()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 48, in init_process
    super(EventletWorker, self).init_process()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
    self.wsgi = self.app.wsgi()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
    self.callable = self.load()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
    return self.load_wsgiapp()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/util.py", line 365, in import_app
    app = eval(obj, mod.__dict__)
  File "<string>", line 1, in <module>
  File "/Users/User.Name/Development/webd.py", line 49, in create_app
    app.db = database.db_from_config('my-database')
  File "/Users/User.Name/Development/tools/db/database.py", line 41, in db_from_config
    return db_from_connection(config.db_uri, config.db_name)
  File "/Users/User.Name/Development/tools/db/database.py", line 32, in db_from_connection
    con = MongoClient(host=uri, tz_aware=True)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mongokit/connection.py", line 107, in __init__
    PymongoConnection.__init__(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/mongo_client.py", line 426, in __init__
    raise ConnectionFailure(str(e))
ConnectionFailure: [Errno 8] No address found

[temoto]

You have OSX in production?

[temoto]

"No address found" looks like a valid error to me. Do you think the address was resolvable at the time?

Could you inject the following code into create_app() just before db_from_config()?

import eventlet
import socket
so = eventlet.patcher.original('socket')
assert eventlet.patcher.is_monkey_patched('socket')
from eventlet.support import greendns
assert socket.gethostbyname is greendns.gethostbyname
db_host = config.db_uri[7:28]  # fix indexes to point to hostname
print('% host__ %%%%%%%%%%%%', db_host)
print('% system %%%%%%%%%%%%', so.gethostbyname(db_host))
print('% green_ %%%%%%%%%%%%', socket.gethostbyname(db_host))

[coffeegist]

Sorry, production was not the right word, development is the environment that our Mac's are in :)

As for the "No address found" message, I'm positive that the address was resolvable (testing with our old versions and with gevent works as expected).

I injected the code above, and the findings are very interesting indeed. so.gethostbyname resolves, while socket.gethostbyname does not. Results below:

('% host__ %%%%%%%%%%%%', 'my.special.hostname.com')
('% system %%%%%%%%%%%%', '192.168.10.1') # fake IP here
2017-02-02 09:18:39 [3801] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
    worker.init_process()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 48, in init_process
    super(EventletWorker, self).init_process()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
    self.wsgi = self.app.wsgi()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
    self.callable = self.load()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
    return self.load_wsgiapp()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gunicorn/util.py", line 365, in import_app
    app = eval(obj, mod.__dict__)
  File "<string>", line 1, in <module>
  File "/Users/Adam.Brown/Development/webd.py", line 60, in create_app
    print('% green_ %%%%%%%%%%%%', socket.gethostbyname(db_host))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/eventlet/support/greendns.py", line 518, in gethostbyname
    rrset = resolve(hostname)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/eventlet/support/greendns.py", line 405, in resolve
    raise EAI_NODATA_ERROR
gaierror: [Errno 7] No address associated with hostname

[temoto]

I forgot to ask you to print os.environ.get('EVENTLET_NO_GREENDNS') in that same code. But what I'm actually trying to get at, is if that environment variable was set before this python process started. Maybe you didn't set it, or maybe gunicorn doesn't pass everything to workers.

[temoto]

And this assert socket.gethostbyname is greendns.gethostbyname should've failed with that env variable set to yes.

[coffeegist]

Very interesting, but I tried the above patch again this morning (the only thing I thought I changed was adding a print for the environment variable), but the assertion did indeed fail this morning as you said it should, and the patch seems to have worked.

[temoto]

Yup, computers are precise only in theory.

Real solution is coming some time later.