Explicit DNSError subclass for failed DNS lookups

[techtonik]

There is no cross-platform way to catch DNS lookup errors with requests. At least I see that error message on Windows https://stackoverflow.com/questions/40145631/precisely-catch-dns-error-with-python-requests looks different from Linux in https://github.com/kennethreitz/requests/issues/3550

Would be nice to get dedicated exception for this use case.

[Lukasa]

So this is somewhat reasonable, but made more opaque by the fact that urllib3 gets in the way here. urllib3 will basically always turn a connection setup problem like this one into a MaxRetriesError (because of its retry processing). That means for us to fire off an appropriate error in all cases means that we need to go and look at every possible exception that can fire.

I think, as a first step, you should propose that urllib3 raise special exceptions when it gets problems from getaddrinfo. From there, Requests can introspect and look for such an exception, which gives us a better shot at raising an appropriate error. And at the very least, you will then be able to dig in to the exceptions to pull out the original (as all of those exceptions are just wrappers).

[techtonik]

Reported above. Do you have access to OS X machine to test what requests.head('http://wowsucherror') returns there currently?

[Lukasa]

In 2.7.12:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='wowsucherror', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x10e22ff10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

[techtonik]

The hack so far:

import requests

def sitecheck(url):
    status = None
    message = ''
    try:
        resp = requests.head('http://' + url)
        status = str(resp.status_code)
    except requests.ConnectionError as exc:
        # filtering DNS lookup error from other connection errors
        # (until https://github.com/shazow/urllib3/issues/1003 is resolved)
        if type(exc.message) != requests.packages.urllib3.exceptions.MaxRetryError:
            raise
        reason = exc.message.reason    
        if type(reason) != requests.packages.urllib3.exceptions.NewConnectionError:
            raise
        if type(reason.message) != str:
            raise
        if ("[Errno 11001] getaddrinfo failed" in reason.message or     # Windows
            "[Errno -2] Name or service not known" in reason.message or # Linux
            "[Errno 8] nodename nor servname " in reason.message):      # OS X
            message = 'DNSLookupError'
        else:
            raise

    return url, status, message

print sitecheck('wowsucherror')
print sitecheck('google.com')

('wowsucherror', None, 'DNSLookupError')
('google.com', '302', '')

[dnx-seek]

I'm not sure what the status of this is a few years later... I found the above didn't work for me on current python, so came up with this. Note I'm only returning the OK status here.

def check_live(url):
    try:
        r = requests.get(url)
        live = r.ok
    except requests.ConnectionError as e:
        if 'MaxRetryError' not in str(e.args) or 'NewConnectionError' not in str(e.args):
            raise
        if "[Errno 8]" in str(e) or "[Errno 11001]" in str(e) or ["Errno -2"] in str(e):
            print('DNSLookupError')
            live = False
        else:
            raise
    except:
        raise
    return live

[gdubicki]

In case someone is wondering what is the status as of now: urllib3 v2.0 will have a separate exception for this, see urllib3/urllib3@1831327 and urllib3/urllib3@8a1ac9f.

[sethmlarson]

In an effort to clean up the issue tracker to only have issues that are still relevant to the project we've done a quick pass and decided this issue may no longer be relevant for a variety of potential reasons, including:

• Applies to a much older version, unclear whether the issue still applies.
• Change requires a backwards incompatible release and it's unclear if the benefits are worth the migration effort from the community.
• There isn't a clear demand from the community on the change landing in Requests.

If you think the issue should remain open, please comment so below or open a new issue and link back to the original issue. Again, thank you for opening the issue and for the discussion, it's much appreciated.