set a timeout for DNS lookups

[kevinburke]

BPO	22889
Nosy	@bitdancer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-11-17.18:54:09.964>
created_at = <Date 2014-11-17.05:38:20.593>
labels = ['type-feature', 'library']
title = 'set a timeout for DNS lookups'
updated_at = <Date 2015-01-23.10:27:32.142>
user = 'https://bugs.python.org/kevinburke'

bugs.python.org fields:

activity = <Date 2015-01-23.10:27:32.142>
actor = 'piotr.dobrogost'
assignee = 'none'
closed = True
closed_date = <Date 2014-11-17.18:54:09.964>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2014-11-17.05:38:20.593>
creator = 'kevinburke'
dependencies = []
files = []
hgrepos = []
issue_num = 22889
keywords = []
message_count = 4.0
messages = ['231266', '231273', '231290', '231291']
nosy_count = 3.0
nosy_names = ['r.david.murray', 'piotr.dobrogost', 'kevinburke']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22889'
versions = ['Python 3.5']

[kevinburke]

It would be nice to be able to set a timeout for DNS lookups in the event of a DNS server failure.

1. Set your DNS to something like 123.123.123.123 or any other host that is not listening for DNS queries on port 53.

2. Run requests.get('http://jsonip.com', timeout=3), or similar using urllib/http.client/httplib

3. Observe that the wall clock time is 2 minutes or higher.

It's known that a timeout value does not correspond to wall clock time, but this can be an unexpected cause of latency in http client requests, and is completely uncontrollable from client code (unless the user resolves DNS themselves).

For a comparison, Go provides this functionality, see for example https://code.google.com/p/go/source/browse/src/pkg/net/lookup.go?name=release#55

[bitdancer]

As far as I know, the libc dns timeout is controlled by /etc/resolv.conf (or whatever the Windows equivalent is), and libc doesn't provide any way for an application to override this. There is no mention of timeout on the resolver(3) man page. Do you know of a way? (One that doesn't change global state.) (In theory one could re-implement a DNS resolver client, but that is not something CPython is going to do :)

[kevinburke]

Hi,
You are correct, there's no way to set it in the C API. Go works around this by resolving addresses in a goroutine. I was wondering if something similar would be possible in Python (for people who set a timeout in the interface).

[bitdancer]

Oh, my apologies. I totally missed your link to the go example when I first read your message.

Yes, Python supports the equivalent. In the asyncio module, which is our closest equivalent to goroutines, the functionality exists implicitly: the base event loop has a getaddrinfo coroutine that is used for DNS lookup, and it is run in a separate thread to keep from blocking the event loop. If you time out (wait_for with a timeout) a coroutine that ends up doing a DNS request, your wait_for call will time out whether or not the getaddrinfo DNS lookup has timed out.

Using asyncio isn't quite as simple as prefixing a function name with 'go', but it isn't all that hard, either. Nevertheless, if you aren't using asyncio for the rest of your program, it probably makes more sense to write a function that launches a thread to do the getaddrinfo and does a wait on the thread with a timeout. (But you might want to check out if asyncio applies to your overall problem.)

So yes, Python can do this, but I don't think it makes sense to build it in to the stdlib's socket module (say), since unlike go async and threading aren't part of the core language but are instead libraries themselves. asyncio already has it built in. For the thread based version, putting a recipe on one of the recipe sites might be good, if there isn't one already.