Network failure leads to TypeError in retry

[JohannesBuchner]

I got this traceback after a network failure with urllib3 v1.10 on Linux:

Exception:
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 135, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/util/connection.py", line 66, in create_connection
    for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
  File "/usr/lib64/python3.4/socket.py", line 533, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 782, in _validate_conn
    conn.connect()
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 215, in connect
    conn = self._new_conn()
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 144, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f86e18d8198>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/usr/lib64/python3.4/site-packages/pip/commands/install.py", line 294, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 334, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 461, in _prepare_file
    req_to_install.populate_link(finder, self.upgrade)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_install.py", line 250, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 486, in find_requirement
    all_versions = self._find_all_versions(req.name)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 404, in _find_all_versions
    index_locations = self._get_index_urls_locations(project_name)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 378, in _get_index_urls_locations
    page = self._get_page(main_index_url)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 818, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 928, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib64/python3.4/site-packages/pip/download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib64/python3.4/site-packages/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/lib64/python3.4/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/util/retry.py", line 226, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

[Lukasa]

Hmm, yeah, this is less than ideal. How does Retry.total ever end up being an instance of Retry? Let me dive in and see what's going on here.

[Lukasa]

I can really only see one flow where this might happen at the moment, which is if Retry.from_int is called and we somehow have a retries instance that is not an instance of Retry when it is passed into the function, but is when we instantiate the class. Which seems crazy and should be impossible.

Oh! But maybe it isn't! I think this isn't strictly a urllib3 bug, it's a requests/pip bug. Are your tracebacks complete, @JohannesBuchner, or did you trim them?

[JohannesBuchner]

I used pip3.4; I did not trim the traceback.

[Lukasa]

Hmm. It feels like parts of the traceback are missing, but I guess they only go as far back as they got unwound to. Regardless, let me ask this: where did you install pip from?

[sigmavirus24]

I've definitely seen this before. That said, this traceback is interesting. urllib3 is vendored into requests, but requests isn't vendored into pip. Very weird.

Also, pip passes its own Retry object iirc. Let me go digging.

[JohannesBuchner]

Gentoo repos. I used pip to install a local egg directory, which loaded some dependencies off the web (sqlalchemy to be precise)

[sigmavirus24]

So pip creates its own Retry object. cachecontrol is involved in this traceback though.

  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib64/python3.4/site-packages/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/lib64/python3.4/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout

I wonder if there's a bug in the version of cachecontrol that gentoo has packaged.

[Lukasa]

It's possible, but my concern is more that if pip/requests is providing a Retry object, that because of the shenanigans in kennethreitz/requests#2870, the isinstance check might fail.

[JohannesBuchner]

I have CacheControl version 0.11.5

[sigmavirus24]

@Lukasa from the traceback it would appear gentoo is not unvendoring urllib3. That bug shouldn't be the cause here.

[shazow]

So sounds like this is a pip/requests bug? (Feel free to re-open if I'm mistaken.)