Timeouts when calling client is causing casting errors

[sbenderli]

It looks like our call to the client to get the images is timing out. I'm guessing that the requests package is used to make calls to the host machine, and that the Timeout scenario is causing a casting issue.

[exec] self = <test_dev_containers.TestContainers object at 0x7efc662a9eb8>
     [exec] 
     [exec]     def test_container_sizes(self):
     [exec]         # Test assumes that images have been built in the system. The gradle task that runs these tests
     [exec]         # has a dependency on a task that runs build_images.py
     [exec]     
     [exec]         # Assert sizes for all images on the system
     [exec] >       images = client.images.list()
     [exec] 
     [exec] docker/test/test_dev_containers.py:50: 
     [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/docker/models/images.py:223: in list
     [exec]     resp = self.client.api.images(name=name, all=all, filters=filters)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/docker/api/image.py:93: in images
     [exec]     res = self._result(self._get(self._url("/images/json"), params=params),
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/docker/utils/decorators.py:47: in inner
     [exec]     return f(self, *args, **kwargs)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/docker/api/client.py:183: in _get
     [exec]     return self.get(url, **self._set_request_timeout(kwargs))
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/sessions.py:515: in get
     [exec]     return self.request('GET', url, **kwargs)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/sessions.py:502: in request
     [exec]     resp = self.send(prep, **send_kwargs)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/sessions.py:612: in send
     [exec]     r = adapter.send(request, **kwargs)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/adapters.py:440: in send
     [exec]     timeout=timeout
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py:587: in urlopen
     [exec]     timeout_obj = self._get_timeout(timeout)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py:302: in _get_timeout
     [exec]     return Timeout.from_float(timeout)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/packages/urllib3/util/timeout.py:154: in from_float
     [exec]     return Timeout(read=timeout, connect=timeout)
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/packages/urllib3/util/timeout.py:94: in __init__
     [exec]     self._connect = self._validate_timeout(connect, 'connect')
     [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
     [exec] 
     [exec] cls = <class 'requests.packages.urllib3.util.timeout.Timeout'>
     [exec] value = <urllib3.util.timeout.Timeout object at 0x7efc6629bf98>
     [exec] name = 'connect'
     [exec] 
     [exec]     @classmethod
     [exec]     def _validate_timeout(cls, value, name):
     [exec]         """ Check that a timeout attribute is valid.
     [exec]     
     [exec]             :param value: The timeout value to validate
     [exec]             :param name: The name of the timeout attribute to validate. This is
     [exec]                 used to specify in error messages.
     [exec]             :return: The validated and casted version of the given value.
     [exec]             :raises ValueError: If it is a numeric value less than or equal to
     [exec]                 zero, or the type is not an integer, float, or None.
     [exec]             """
     [exec]         if value is _Default:
     [exec]             return cls.DEFAULT_TIMEOUT
     [exec]     
     [exec]         if value is None or value is cls.DEFAULT_TIMEOUT:
     [exec]             return value
     [exec]     
     [exec]         if isinstance(value, bool):
     [exec]             raise ValueError("Timeout cannot be a boolean value. It must "
     [exec]                              "be an int, float or None.")
     [exec]         try:
     [exec]             float(value)
     [exec]         except (TypeError, ValueError):
     [exec]             raise ValueError("Timeout value %s was %s, but it must be an "
     [exec] >                            "int, float or None." % (name, value))
     [exec] E           ValueError: Timeout value connect was Timeout(connect=None, read=None, total=None), but it must be an int, float or None.
     [exec] 
     [exec] /tmp/docker-container-test-20170614011257/py35/lib/python3.5/site-packages/requests/packages/urllib3/util/timeout.py:127: ValueError

[mattoberle]

I found that this block of code from urllib3 was the failure point in my case:

    def _get_timeout(self, timeout):
        """ Helper that always returns a :class:`urllib3.util.Timeout` """
        if timeout is _Default:
            return self.timeout.clone()

        if isinstance(timeout, Timeout):
            return timeout.clone()
        else:
            # User passed us an int/float. This is for backwards compatibility,
            # can be removed later
            return Timeout.from_float(timeout)

timeout was an instance of urllib3.util.timeout.Timeout
Timeout was an instance of requests.packages.urllib3.util.timeout.Timeout

When the isinstance(timeout, Timeout) check fails the urllib3.util.timeout.Timeout object is passed as the connection argument to requests.packages.urllib3.util.timeout.Timeout.from_float.

In the docker-py repository this import business seems to be an integral part of the issue:

try:
    import requests.packages.urllib3 as urllib3
except ImportError:
    import urllib3

Commenting out the try/except and only using import urllib3 prevented the error from occurring.

It looks like a recent commit to the requests library breaks compatibility with this commit:
psf/requests@168109f

In the meantime I've found that pinning requests to version 1.17.3 side-steps the issue.

[sbenderli]

Looks like requests released twice today: 2.18.0 was problematic, but 2.18.1 seems not to have this issue
https://github.com/requests/requests/commits/master

edit: We don't have set requirements for the requests module, so we get whatever docker-py brings in. Previously (prior to 10am today) requests 2.17.3 was coming along, and things were ok. After 10am, we started getting 2.18.0 while we saw the errors, and finally around 1pm 2.18.1 came along and fixed the issue.

[mattoberle]

Here is the issue in the requests project regarding the problem:
psf/requests#4160

PR #1646 places the version on the list of incompatible versions.

[shin-]

I keep getting flak when I want to restrict the range of versions for requests on this repo and compose, but this kind of issue is the exact reason I want to do so.

Thanks for the report! I'll take a look at that PR asap.

[sbenderli]

Mystery explained, thanks @mattoberle

[shin-]

Should be fixed by #1646