Dependency on urllib3?

[mlissner]

I just upgraded the packages on my system, and this package (which must be a dependency for another package) broke my system. I'm getting the error that:

ImportError: No module named urllib3.response

And I don't have urllib3 installed, so that's the problem. However, why don't I have it installed? I only use pip to install things, so I think the fact that urllib3 isn't installed means that it's not listed in a requirements.txt file somewhere.

It doesn't look like this project has a requirements list, but I don't know. It does seem like the urllib3 requirement might be a new one?

[mlissner]

Yeah, this looks like the culprit from three days ago: b6c62d8

I imagine this will be breaking a fair number of people's systems soon?

[dhermes]

@mlissner Can you be a bit more explicit about how this broke your system?

FWIW, urllib3 comes in as a dependency of requests, which is an "extra" rather than being a direct dependency. In other words, a perfectly reasonable way to install this library would be

pip install google-resumable-media[requests]

/cc @jonparrott

[john2x]

We're currently affected by this issue. I was able to track it down to having an explicit dependency to the following:

requests==2.10.0
google-cloud==0.27

requests 2.10.0 doesn't depend on urllib3 yet, so it isn't installed automatically.

Updating our requirements to

requests==2.18.4
google-cloud==0.27

addresses the issue.

[mlissner]

Yep, we have requests pinned with:

requests==2.9.1

I think the fix here is to require a version of requests that contains urllib3 (or not use urllib3).

[mlissner]

Can you be a bit more explicit about how this broke your system?

And to respond to this....it broke my system because as part of deployment we always upgrade the packages in our requirements.txt file. In there we have requests pinned, but not google-cloud. So when we upgraded everything, google-cloud suddenly needed urllib3, but didn't have it.

[dhermes]

google-cloud suddenly needed urllib3

How do you mean? The dependency is optional because it can only be triggered if the google.resumable_media.requests package is imported. So you wouldn't see any issue at install time, just at run time. If you have code that is failing at run time, can you share a stacktrace? Also, can you share the pinned versions for requests and any package that begins with google?

---

As for google-cloud, I strongly recommend you just depend on the services that you need. The google-cloud package includes every single google-cloud-* package from PyPI.

Instead, you could depend only on google-cloud-storage.

---

Checking out the actual dependencies of the current google-cloud:

google-cloud==0.27.0 (latest) depends on google-cloud-storage >= 1.3.0, < 1.4dev. The latest version that will satisfy this (google-cloud-storage==1.3.2) depends on requests >= 2.18.0.

In addition, that version of google-cloud, depends on google-cloud-core >= 0.26.0, < 0.27dev, which has only one match 0.26.0 that depends on requests >= 2.4.0, < 3.0.0dev.

FWIW, the release (2.10.0) of requests that includes the urllib3 dependency came out in April 2016.

Can you share the pinned versions?

[mlissner]

If you have code that is failing at run time, can you share a stacktrace? Also, can you share the pinned versions for requests and any package that begins with google?

Here's the stacktrace. It's a doozy:

In [21]: from celery.__main__ import main

In [22]: main()
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-22-58ca95c5b364> in <module>()
----> 1 main()

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/__main__.pyc in main()
     28         maybe_patch_concurrency()
     29     from celery.bin.celery import main
---> 30     main()
     31 
     32 

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/celery.pyc in main(argv)
     79         from billiard import freeze_support
     80         freeze_support()
---> 81         cmd.execute_from_commandline(argv)
     82     except KeyboardInterrupt:
     83         pass

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/celery.pyc in execute_from_commandline(self, argv)
    768         try:
    769             sys.exit(determine_exit_status(
--> 770                 super(CeleryCommand, self).execute_from_commandline(argv)))
    771         except KeyboardInterrupt:
    772             sys.exit(EX_FAILURE)

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/base.pyc in execute_from_commandline(self, argv)
    309         argv = self.setup_app_from_commandline(argv)
    310         self.prog_name = os.path.basename(argv[0])
--> 311         return self.handle_argv(self.prog_name, argv[1:])
    312 
    313     def run_from_argv(self, prog_name, argv=None, command=None):

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/celery.pyc in handle_argv(self, prog_name, argv)
    760         except IndexError:
    761             command, argv = 'help', ['help']
--> 762         return self.execute(command, argv)
    763 
    764     def execute_from_commandline(self, argv=None):

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/celery.pyc in execute(self, command, argv)
    692                 no_color=self.no_color, quiet=self.quiet,
    693                 on_usage_error=partial(self.on_usage_error, command=command),
--> 694             ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
    695         except self.UsageError as exc:
    696             self.on_usage_error(exc)

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/base.pyc in run_from_argv(self, prog_name, argv, command)
    313     def run_from_argv(self, prog_name, argv=None, command=None):
    314         return self.handle_argv(prog_name,
--> 315                                 sys.argv if argv is None else argv, command)
    316 
    317     def maybe_patch_concurrency(self, argv=None):

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/base.pyc in handle_argv(self, prog_name, argv, command)
    375         options, args = self.prepare_args(
    376             *self.parse_options(prog_name, argv, command))
--> 377         return self(*args, **options)
    378 
    379     def prepare_args(self, options, args):

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/base.pyc in __call__(self, *args, **kwargs)
    272         self.verify_args(args)
    273         try:
--> 274             ret = self.run(*args, **kwargs)
    275             return ret if ret is not None else EX_OK
    276         except self.UsageError as exc:

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/bin/celery.pyc in run(self, force_ipython, force_bpython, force_python, without_tasks, eventlet, gevent, **kwargs)
    560         import celery
    561         import celery.task.base
--> 562         self.app.loader.import_default_modules()
    563         self.locals = {'app': self.app,
    564                        'celery': self.app,

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/loaders/base.pyc in import_default_modules(self)
    114 
    115     def import_default_modules(self):
--> 116         signals.import_modules.send(sender=self.app)
    117         return [
    118             self.import_task_module(m) for m in (

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/utils/dispatch/signal.pyc in send(self, sender, **named)
    164 
    165         for receiver in self._live_receivers(_make_id(sender)):
--> 166             response = receiver(signal=self, sender=sender, **named)
    167             responses.append((receiver, response))
    168         return responses

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/amqp/utils.pyc in __call__(self, *args, **kwargs)
     40             )
     41         except Exception as exc:
---> 42             self.set_error_state(exc)
     43         else:
     44             if self.on_success:

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/amqp/utils.pyc in __call__(self, *args, **kwargs)
     37             self.value = self.fun(
     38                 *self.args + args if self.args else args,
---> 39                 **dict(self.kwargs, **kwargs) if self.kwargs else kwargs
     40             )
     41         except Exception as exc:

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/app/base.pyc in _autodiscover_tasks(self, packages, related_name, **kwargs)
    328         # argument may be lazy
    329         packages = packages() if callable(packages) else packages
--> 330         self.loader.autodiscover_tasks(packages, related_name)
    331 
    332     def send_task(self, name, args=None, kwargs=None, countdown=None,

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/loaders/base.pyc in autodiscover_tasks(self, packages, related_name)
    249         self.task_modules.update(
    250             mod.__name__ for mod in autodiscover_tasks(packages or (),
--> 251                                                        related_name) if mod)
    252 
    253     @property

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/loaders/base.pyc in autodiscover_tasks(packages, related_name)
    270     _RACE_PROTECTION = True
    271     try:
--> 272         return [find_related_module(pkg, related_name) for pkg in packages]
    273     finally:
    274         _RACE_PROTECTION = False

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/loaders/base.pyc in find_related_module(package, related_name)
    296         return
    297 
--> 298     return importlib.import_module('{0}.{1}'.format(package, related_name))

/usr/lib/python2.7/importlib/__init__.pyc in import_module(name, package)
     35             level += 1
     36         name = _resolve_name(name[level:], package, level)
---> 37     __import__(name)
     38     return sys.modules[name]

/var/www/courtlistener/cl/audio/tasks.py in <module>()
      8 from celery.canvas import chain
      9 from django.conf import settings
---> 10 from google.cloud import storage
     11 from google.cloud.exceptions import Forbidden, NotFound
     12 from google.cloud.storage import Blob

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/google/cloud/storage/__init__.py in <module>()
     36 
     37 from google.cloud.storage.batch import Batch
---> 38 from google.cloud.storage.blob import Blob
     39 from google.cloud.storage.bucket import Bucket
     40 from google.cloud.storage.client import Client

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/google/cloud/storage/blob.py in <module>()
     38 
     39 from google import resumable_media
---> 40 from google.resumable_media.requests import ChunkedDownload
     41 from google.resumable_media.requests import Download
     42 from google.resumable_media.requests import MultipartUpload

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/google/resumable_media/requests/__init__.py in <module>()
    661 
    662 
--> 663 from google.resumable_media.requests.download import ChunkedDownload
    664 from google.resumable_media.requests.download import Download
    665 from google.resumable_media.requests.upload import MultipartUpload

/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/google/resumable_media/requests/download.py in <module>()
     19 import logging
     20 
---> 21 import urllib3.response
     22 
     23 from google.resumable_media import _download

ImportError: No module named urllib3.response

In [23]: import urllib3
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-23-dbe53ea169b1> in <module>()
----> 1 import urllib3

ImportError: No module named urllib3

I think it just comes from here though:

https://github.com/GoogleCloudPlatform/google-resumable-media-python/blob/b6c62d80ad8415626fade6caf4282939862996bd/google/resumable_media/requests/download.py#L21

I don't actually use that code, but I guess it gets imported somehow.

I honestly don't know how pip handles dependency conflicts. I'd expect that if I tried to upgrade google-cloud and it required a version of requests that's newer than the one I have installed, that I'd get a failure, but apparently not?

[dhermes]

OK, that stacktrace is what I expect. It really just reflects an "out-of-compliance" dependency on requests.

The "fix" is for you is to update your dependency. I doubt (but don't know) that upgrading requests will break you in any way.

---

Unfortunately, pip will allow you to break dependency requirements (it's probably an np-hard problem not to):

$ virtualenv venv
$ venv/bin/pip install google-resumable-media[requests]
$ venv/bin/pip show requests
Name: requests
Version: 2.18.4
...
$ venv/bin/python
>>> import google.resumable_media.requests
>>> google.resumable_media.requests
<module 'google.resumable_media.requests' from '.../venv/lib/python3.6/site-packages/google/resumable_media/requests/__init__.py'>
$
$ venv/bin/pip uninstall urllib3
$ venv/bin/pip install 'requests==2.9.1'
$ venv/bin/pip show requests
Name: requests
Version: 2.9.1
...
$ venv/bin/python
>>> import google.resumable_media.requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../venv/lib/python3.6/site-packages/google/resumable_media/requests/__init__.py", line 663, in <module>
    from google.resumable_media.requests.download import ChunkedDownload
  File ".../venv/lib/python3.6/site-packages/google/resumable_media/requests/download.py", line 21, in <module>
    import urllib3.response
ModuleNotFoundError: No module named 'urllib3'

---

I am going to close this out, as it seems the issue is caused by you explicitly pinning to an earlier version of requests than what is required by google-cloud-storage.

Let's continue the discussion if you think there is more to resolve. I'm happy to help.

[mlissner]

I guess I'd expect pip to make a dependency graph and sort these things out. Apt can handle such things. Anyway, I've got tests...I already just installed urllib3, but I'll upgrade requests and see what happens. They're pretty careful stewards.

Thanks again.

[dhermes]

Sure thing, sorry for the pain!