CFFI bindings for c-ares

[thedrow]

It might be nice to provide better DNS resolving engines for PyPy as well.

[jamadden]

The existing Cython-based c-ares bindings seem to work fine, as far as the test-suite is concerned. I've enabled them for PyPy (as an option, not as a default, the same for cpython). Note that I haven't attempted any performance comparisons because That Would Be Hard.

[thedrow]

CPython extensions are generally slower on PyPy so I think that there is value in creating a CFFI extension for it.
I also don't know how to measure performance of c-ares in PyPy with a C extension and a CFFI extension.
Maybe we could use the vmprof for that?

[jamadden]

CPython extensions are generally slower on PyPy so I think that there is value in creating a CFFI extension for it.

That can certainly be true, depending on how much time is spent interfacing between C and Python. Given that's it's cython-ized code, my guess is "quite a lot". But then again, it's DNS lookups. How often is that actually a bottleneck in a Python program? More to the point, how often is the local interpreter overhead a bottleneck compared to network overhead in DNS resolution?

That said, I wasn't trying to say that a CFFI binding would never be useful (although it does double the maintenance requirements). I was just happy to be able to say that the functionality does exist and can be used, even if it doesn't perform optimally.

[jamadden]

I wasn't trying to say that a CFFI binding would never be useful (although it does double the maintenance requirements)

Hmm. After having gotten the corecffi loop running on CPython, it appears there's very little performance penalty relative to the Cython loop (~5-10%, depending on platform). If that turns out to hold true (and it looks like it should) then I would be in favor of dropping the Cython c-ares bindings in favor of CFFI bindings that are usable and stable on both platforms (assuming we don't run into any windows-related issues, as I did with the CFFI loop). (That's certainly where I intend to start investigating a libuv loop :) )

[thedrow]

Sounds like a plan.

[jamadden]

#774 is probably a bit of a blocker for this. And as a side note, a libuv loop could dispense with c-ares altogether in favor of the async DNS builtin to libuv.

[thedrow]

Are there plans to move from libev to libuv?

[jamadden]

I wouldn't call them exactly plans at this point, but it's something I want to take a serious look at.

[jgehrcke]

IMO one of the nicest things libuv would bring along is IOCP support.
Also see
https://gehrcke.de/gipc/#notes-for-windows-users
https://github.com/saghul/uvent
https://twitter.com/gevent/status/251870755187478529

[jamadden]

I've been looking at libuv, and it's not quite the panacea one would hope.

First, what's relevant for this thread, the DNS support, sadly, is also no better: it just shoves requests off onto its own internal threadpool. Making it no different than gevent's default resolver.

Other minor problems include no support for fork watchers or child watchers; those can both be handled at the gevent level though, at least on POSIX (but the situation with children and SIGCHLD won't get any better).

The biggest impedance mismatch, though, is in the handling of IO. The libuv equivalent to libev's IO watchers, "poll" watchers, pretty much matches exactly what libev does, including only working for sockets on Windows (and not using IOCP, AFAICS). If you want the much-vaunted Windows improvements you have to use functions and types specific to TCP, UDP or pipes, and these do not have (exposed) read or write events, relying instead on a series of non-blocking callbacks that are handled internally to the event loop. This doesn't fit well with gevent's "wait for a read event then try to read again" approach (in fact, gevent wouldn't be able to use the Python-level socket object at all). It may be possible to paper over this with some higher-level abstractions, but they don't currently exist.

So in the short term, I'm still interested in looking into a libuv port based on the poll watchers. I think that's pretty low hanging fruit, and I'm interested to compare performance. For another thing, this should make CFFI possible on Windows. But any other Windows-specific improvements that would come from redesigning the socket abstractions are further down the line.

[jamadden]

Moved libuv discussion to #790. Short sad story: I hit a wall with libuv breaking terribly on a fork().

[jamadden]

Perhaps another option is to adopt dnspython, a pure-python DNS implementation, to replace c-ares. That's what eventlet just did.

[thedrow]

It'd be a shame since it's a very good opportunity to release the GIL earlier in case you are working with both gevent and threading.
We can fallback to dnspython in PyPy if there's no other choice.

[thedrow]

I see that we're currently not releasing the GIL anyway. No idea why though.

[thedrow]

See #910 :)