eventlet is incompatible with dnspython 2.0.0rc1

[frenzymadness]

Hello.

dnspython version 2.0.0 will be released soon (it's a release candidate now, rthalley/dnspython#466) and it seems that eventlet is not compatible with it.

# pip list
Package    Version
---------- --------
dnspython  2.0.0rc1
eventlet   0.25.2
greenlet   0.4.16
monotonic  1.5
pip        20.1.1
setuptools 47.3.1
six        1.15.0
wheel      0.34.2

# python
Python 3.8.3 (default, May 29 2020, 00:00:00) 
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import eventlet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lbalhar/.virtualenvs/tmp-722a19f0bf97ea6/lib/python3.8/site-packages/eventlet/__init__.py", line 10, in <module>
    from eventlet import convenience
  File "/home/lbalhar/.virtualenvs/tmp-722a19f0bf97ea6/lib/python3.8/site-packages/eventlet/convenience.py", line 7, in <module>
    from eventlet.green import socket
  File "/home/lbalhar/.virtualenvs/tmp-722a19f0bf97ea6/lib/python3.8/site-packages/eventlet/green/socket.py", line 21, in <module>
    from eventlet.support import greendns
  File "/home/lbalhar/.virtualenvs/tmp-722a19f0bf97ea6/lib/python3.8/site-packages/eventlet/support/greendns.py", line 64, in <module>
    setattr(dns, pkg, import_patched('dns.' + pkg))
  File "/home/lbalhar/.virtualenvs/tmp-722a19f0bf97ea6/lib/python3.8/site-packages/eventlet/support/greendns.py", line 59, in import_patched
    return patcher.import_patched(module_name, **modules)
  File "/home/lbalhar/.virtualenvs/tmp-722a19f0bf97ea6/lib/python3.8/site-packages/eventlet/patcher.py", line 123, in import_patched
    return inject(
  File "/home/lbalhar/.virtualenvs/tmp-722a19f0bf97ea6/lib/python3.8/site-packages/eventlet/patcher.py", line 100, in inject
    module = __import__(module_name, {}, {}, module_name.split('.')[:-1])
ModuleNotFoundError: No module named 'dns.asyncquery.py'; 'dns.asyncquery' is not a package

Could you please take a look on that? Let me know if I can help anyhow.

[rthalley]

this might be a problem in the all list in RC1 as I see it lists asyncquery.py in there. I will fix in dnspython.

[jstasiak]

@frenzymadness @rthalley thank you for the heads up. I can't look into this right now (maybe someone else will) but thank you nonetheless!

[eacmen]

For what it is worth dnspython 2.0.0 release dropped yesterday in pypi and this was a breaking change for us. Our container could no longer resolve rabbitmq host. We are using nameko 2.12.0 with eventlet 0.25.2. If we pinned dnspython to 1.16.0 our issues were fixed.

Apologies for the screenshot but it is hard to get full stack trace from our log collector.

[rthalley]

The problem here is that the APIs to dns.query.udp() and dns.query.tcp() changed in dnspython 2.0.0, and eventlet is monkey patching them with the old versions. In particular, dns.query.udp() got a new raise_on_truncation parameter which the resolver is trying to use, and this is causing a TypeError "udp() got an unexpected keyword argument 'raise_on_truncation'" exception any time the resolver tries to call udp(). The resolver treats such exceptions as a "try again" type error, and hence the resolution ultimately times out.

Eventlet will need to detect which version of dnspython is running (import dns.version) and monkey patch appropriately. Note that the raise_on_truncation is not the only change, as the af parameter is now gone, and you can also pass sockets to both udp() and tcp().

/Bob

[gibizer]

OpenStack Nova also found the same eventlet - dnspython 2.0.0 incompatibility tracked in https://bugs.launchpad.net/nova/+bug/1888237

[SeanMooney]

i closed #632 since dnspython seams to be the root cause there too so the new release seams to cause multiple failures
fell free to reopen #632 if it make it easier to track compatibility with dnspython 2.0.0

[ralonsoh]

I've pushed 46fc185 for #629. This should work for dnspython versions <2 and >=2.

[temoto]

@ralonsoh hi, I'm surprised, how did you push a branchless commit to this repo, I don't see you in commiters list. No offense here, pure curiosity. And thanks for fix.

[temoto]

Everybody affected by this problem with a free minute is warmly invited to test Rodolfo solution.

(this is his commit rebased on master 0.26)
pip install https://github.com/eventlet/eventlet/archive/adb2eebc14cbdd00f572a8f36b5e9cb6994b6d42.zip

[rthalley]

I don't think this will fix the underlying issues, because the dnspython resolver is still used, and it calls dns.udp.query() with a parameter, raise_on_truncation, that the monkey patched replacements don't understand. (see my description above)

[SeanMooney]

issue #632 which we closed as a duplicaite of this issue fails with dnspython 2.0.0
for compltely different reason then what @ralonsoh adresses in commit 46fc185

it was failing because TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket
so there are multiple incompatibilities between eventlet and dnspython

it was not clear what the conenct between the dnspython package and ssl was to me looking at the trace back but downgradign dnspython resolved the issue. perhaps we shoudl reopen #632 to track it seperatly otherwise i think commit 46fc185 is not a compete fix.

we certenly could not uncap dnspython in openstack just by addressing the issue that neutron hit which @ralonsoh is fixing. nova would still fail when using websockify with tls for our vm console proxy server.

[temoto]

v0.26.1 just uploaded to PyPI with temporary workaround by @eacmen pin dnspython<2

[frenzymadness]

v0.26.1 just uploaded to PyPI with temporary workaround by @eacmen pin dnspython<2

This workaround might help when eventlet is installed from PyPI but it doesn't help to distribution RPM/DEB packages. For example, we updated dnspython to the latest version in Fedora and now we are no longer able to build eventlet in RPM package.

[ShaneHarvey]

perhaps we shoudl reopen #632 to track it seperatly otherwise i think commit 46fc185 is not a compete fix.

I agree. The ssl module still does not work with dnspython 2.0 even with the fix in #639:

$ pip install --upgrade https://github.com/jayvdb/eventlet/archive/dnspython2.zip
$ pip install dnspython==2.0
$ PYMONGO_MUST_CONNECT=1 python green_framework_test.py eventlet -s test.test_client_context
...
  File "/Users/shane/git/mongo-python-driver/pymongo/pool.py", line 1180, in connect
    sock = _configured_socket(self.address, self.opts)
  File "/Users/shane/git/mongo-python-driver/pymongo/pool.py", line 1002, in _configured_socket
    sock = ssl_context.wrap_socket(sock, server_hostname=host)
  File "/Users/shane/git/mongo-python-driver/venvPYTHON-1854/lib/python3.8/site-packages/eventlet/green/ssl.py", line 448, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/Users/shane/git/mongo-python-driver/venvPYTHON-1854/lib/python3.8/site-packages/eventlet/green/ssl.py", line 69, in __new__
    ret = _original_sslsocket._create(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1031, in _create
    self._sslobj = self._context._wrap_socket(
TypeError: _wrap_socket() argument 'sock' must be _socket.socket, not SSLSocket
...

[rthalley]

FWIW, I looked at that SSL error a bit. It's hard to figure out not really understanding either the original python SSL code nor eventlet's wrapping, but it seems like a double wrapping. Not sure how that could happen or what dnspython could have to do with it, but maybe some change in error paths?

At any rate, I repeat what I said above, which is that none of the fixes I've seen so far address the raise_on_truncation problem, so I expect that eventlet's patching of getaddrinfo, etc. will be getting timeouts after around 5 seconds. The test program I wrote along with an instrumented dnspython still shows this problem exists with the master branch of eventlet. I suggest fixing this problem might be a good first start, as we know it is wrong.

[rthalley]

If someone wants to test "does fixing the truncation problem help with the SSL problem", here is a hack for the truncation issue. This is NOT a fix, as it doesn't actually raise_on_truncation, which will break resolutions with truncation. This is merely a way to test relatedness to the SSL issue. I can't test with mongodb myself as I don't have it set up.

This hack basically takes the #639 compute_expiration change, just for UDP, and then changes the function signature to be compatible with dnspython 2.0. (And thus INCOMPATIBLE with < 2.0.) With this hack, basic resolutions now work for me again instead of timing out.

@@ -658,8 +658,16 @@ def _net_write(sock, data, expiration):
                 raise dns.exception.Timeout


+def compute_expiration(query, timeout):
+     if hasattr(query, '_compute_expiration'):
+         return query._compute_expiration(timeout)
+     else:
+         return query._compute_times(timeout)[1]
+
+
 def udp(q, where, timeout=DNS_QUERY_TIMEOUT, port=53,
-        af=None, source=None, source_port=0, ignore_unexpected=False):
+        source=None, source_port=0, ignore_unexpected=False,
+        raise_on_truncation=False):
     """coro friendly replacement for dns.query.udp
     Return the response obtained after sending a query via UDP.

@@ -687,6 +695,7 @@ def udp(q, where, timeout=DNS_QUERY_TIMEOUT, port=53,
     @type ignore_unexpected: bool"""

     wire = q.to_wire()
+    af = None
     if af is None:
         try:
             af = dns.inet.af_for_address(where)
@@ -709,7 +718,7 @@ def udp(q, where, timeout=DNS_QUERY_TIMEOUT, port=53,
     s = socket.socket(af, socket.SOCK_DGRAM)
     s.settimeout(timeout)
     try:
-        expiration = dns.query._compute_expiration(timeout)
+        expiration = compute_expiration(dns.query, timeout)
         if source is not None:
             s.bind(source)
         while True
:```

[rthalley]

Oh, and here is the test program for basic resolutions. Without the hack, both invocations fail due to triggering the
udp() got an unexpected keyword argument 'raise_on_truncation' TypeError

import dns.resolver

import eventlet.support.greendns as gd
print('eventlet')
p = gd.ResolverProxy()
try:
    a = p.query('www.google.com')
    print(a.rrset)
except Exception as e:
    print('bad', e)

print('dnspython')
a = dns.resolver.resolve(dns.name.from_text('www.google.com'), dns.rdatatype.A, dns.rdataclass.IN, False, None, raise_on_no_answer=False)
print(a.rrset)

[jpegxguy]

Wait so is the bug going to be fixed? I think it's great to see workarounds and limiting to <2.0 to stop breaking things, but what about the future? Sorry if I missed something

[Xronis]

Hi,

I've tested this with eventlet version 0.3 and dnspython 1.16.0 and I'm still having the same error

Is there a workaround for this at the moment?

[temoto]

@Xronis 0.3?

[Xronis]

@temoto I was trying with v0.31.0 and then after trying this

Everybody affected by this problem with a free minute is warmly invited to test Rodolfo solution.

(this is his commit rebased on master 0.26)
pip install https://github.com/eventlet/eventlet/archive/adb2eebc14cbdd00f572a8f36b5e9cb6994b6d42.zip

I got v0.26.0 that still produces the error. Am I doing something wrong here?

[temoto]

@Xronis you seem to be doing correct actions. Did you recreate virtualenv or somehow made sure that dnspython <2 is installed? Can you show pip freeze |fgrep -i dns ?

[Xronis]

Since I'm using Pycharm I used the integrated package manager to make sure that I got dnspython to 1.16.0.

I'm using windows 10 and I'm not really sure if there is an fgrep for it but using grep, the output of what you asked of is

dnspython==1.16.0

Let me know if I need to perform something extra.

[temoto]

@Xronis so people say downgrading dnspython to <2 helps, there is nothing I can help with right now.

[Xronis]

Hmmm interesting. Thanks for trying @temoto. May I ask, if there is a plan for supporting of dnspython => 2.0?

[temoto]

@Xronis yes as in "prioritized task", no as in "expect to be done by X date".

[Xronis]

That's understandable. Thank you for your response! 👍

[tipabu]

As a heads-up, it looks like supporting newer dnspython will be a necessary prerequisite to supporting Python 3.10. dnspython<2.0.0 has a reference to collections.MutableMapping, which was deprecated in favor of collections.abc.MutableMapping in py33 and removed in py310.

[temoto]

Everyone subscribed to this issue is very welcome to test another fix at #722

pip install https://github.com/felixonmars/eventlet/archive/refs/heads/dnspython2.zip

[temoto]

Hopefully we're fully compatible with dnspython2 by now. Please reopen if there are problems still.