Conflicting Dependencies #3579

lukesneeringer · 2017-07-05T14:57:22Z

This is reposted from an internal ticket. (Googlers: b/63062329)

Relevant PRs:

Making requirements tighter for "stable" packages in 0.25.1 release. #3584
Making requirements tighter for "stable" packages in 0.26.1 release. #3586
Making requirements tighter for "stable" packages in error reporting 0.24.3 release. #3587
Making requirements tighter for "stable" packages in error reporting 0.25.1 release. #3588

(Added by @dhermes for posterity)

Issue summary:

# Install google-cloud 0.25.0 in Ubuntu (16.04, 16.10, 17.04)
sudo apt-get update
sudo apt-get install python-pip
pip install google-cloud==0.25.0

# Its dependencies have conflicts
python -c "__requires__=['google-cloud']; import pkg_resources" <-- will receive error:

Here is the traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 637, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 650, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 834, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (google-cloud-core 0.24.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('google-cloud-core<0.26dev,>=0.25.0'), set(['google-cloud-storage', 'google-cloud-logging', 'google-cloud-datastore']))

Workaround: Customer has reverted to 0.23.0 to build.

Very similar to #3331.

The text was updated successfully, but these errors were encountered:

dhermes · 2017-07-05T17:57:05Z

@lukesneeringer ISTM the issue is that

They had an existing installation that conflicts with the latest version
The bug is somewhat in pip / setuptools?

Here is the latest state of things:

google-cloud==0.26.0
- google-cloud-core==0.25.*
- google-cloud-datastore >= 1.1.0, < 2.0dev
- google-cloud-logging >= 1.1.0, < 2.0dev
- google-cloud-storage >= 1.2.0, < 2.0dev
google-cloud-logging==1.1.0
- google-cloud-core==0.25.*
google-cloud-storage==1.2.0
- google-cloud-core==0.25.*
google-cloud-datastore==1.1.0
- google-cloud-core==0.25.*

tseaver · 2017-07-05T17:58:59Z

I cannot reproduce. For google-cloud==0.25.0:

$ /opt/Python-3.6.0/bin/virtualenv /tmp/gcp-3579-0.25
Using base prefix '/opt/Python-3.6.0'
New python executable in /tmp/gcp-3579-0.25/bin/python3.6
Also creating executable in /tmp/gcp-3579-0.25/bin/python
Installing setuptools, pip, wheel...done.
$ /tmp/gcp-3579-0.25/bin/pip install google-cloud==0.25.0
Collecting google-cloud==0.25.0
...
Successfully installed cachetools-2.0.0 certifi-2017.4.17 chardet-3.0.4 dill-0.2.6 future-0.16.0 gapic-google-cloud-datastore-v1-0.15.3 gapic-google-cloud-error-reporting-v1beta1-0.15.3 gapic-google-cloud-logging-v2-0.91.3 gapic-google-cloud-pubsub-v1-0.15.4 gapic-google-cloud-spanner-admin-database-v1-0.15.3 gapic-google-cloud-spanner-admin-instance-v1-0.15.3 gapic-google-cloud-spanner-v1-0.15.3 gapic-google-cloud-speech-v1-0.15.3 google-auth-1.0.1 google-auth-httplib2-0.0.2 google-cloud-0.26.0 google-cloud-bigquery-0.25.0 google-cloud-bigtable-0.25.0 google-cloud-core-0.25.0 google-cloud-datastore-1.1.0 google-cloud-dns-0.25.0 google-cloud-error-reporting-0.25.0 google-cloud-language-0.25.0 google-cloud-logging-1.1.0 google-cloud-monitoring-0.25.0 google-cloud-pubsub-0.26.0 google-cloud-resource-manager-0.25.0 google-cloud-runtimeconfig-0.25.0 google-cloud-spanner-0.25.0 google-cloud-speech-0.26.0 google-cloud-storage-1.2.0 google-cloud-translate-0.25.0 google-cloud-videointelligence-0.25.0 google-cloud-vision-0.25.0 google-gax-0.15.13 google-resumable-media-0.1.1 googleapis-common-protos-1.5.2 grpc-google-iam-v1-0.11.1 grpcio-1.4.0 httplib2-0.10.3 idna-2.5 oauth2client-3.0.0 ply-3.8 proto-google-cloud-datastore-v1-0.90.4 proto-google-cloud-error-reporting-v1beta1-0.15.3 proto-google-cloud-logging-v2-0.91.3 proto-google-cloud-pubsub-v1-0.15.4 proto-google-cloud-spanner-admin-database-v1-0.15.3 proto-google-cloud-spanner-admin-instance-v1-0.15.3 proto-google-cloud-spanner-v1-0.15.3 proto-google-cloud-speech-v1-0.15.3 protobuf-3.3.0 pyasn1-0.2.3 pyasn1-modules-0.0.9 requests-2.18.1 rsa-3.4.2 six-1.10.0 urllib3-1.21.1

For google-cloud==0.26.0:

$ /opt/Python-3.6.0/bin/virtualenv /tmp/gcp-3579-0.26
Using base prefix '/opt/Python-3.6.0'
New python executable in /tmp/gcp-3579-0.26/bin/python3.6
Also creating executable in /tmp/gcp-3579-0.26/bin/python
Installing setuptools, pip, wheel...done.
$ /tmp/gcp-3579-0.26/bin/pip install google-cloud==0.26.0
Collecting google-cloud==0.26.0
...
Successfully installed cachetools-2.0.0 certifi-2017.4.17 chardet-3.0.4 dill-0.2.6 future-0.16.0 gapic-google-cloud-datastore-v1-0.15.3 gapic-google-cloud-error-reporting-v1beta1-0.15.3 gapic-google-cloud-logging-v2-0.91.3 gapic-google-cloud-pubsub-v1-0.15.4 gapic-google-cloud-spanner-admin-database-v1-0.15.3 gapic-google-cloud-spanner-admin-instance-v1-0.15.3 gapic-google-cloud-spanner-v1-0.15.3 gapic-google-cloud-speech-v1-0.15.3 google-auth-1.0.1 google-auth-httplib2-0.0.2 google-cloud-0.26.0 google-cloud-bigquery-0.25.0 google-cloud-bigtable-0.25.0 google-cloud-core-0.25.0 google-cloud-datastore-1.1.0 google-cloud-dns-0.25.0 google-cloud-error-reporting-0.25.0 google-cloud-language-0.25.0 google-cloud-logging-1.1.0 google-cloud-monitoring-0.25.0 google-cloud-pubsub-0.26.0 google-cloud-resource-manager-0.25.0 google-cloud-runtimeconfig-0.25.0 google-cloud-spanner-0.25.0 google-cloud-speech-0.26.0 google-cloud-storage-1.2.0 google-cloud-translate-0.25.0 google-cloud-videointelligence-0.25.0 google-cloud-vision-0.25.0 google-gax-0.15.13 google-resumable-media-0.1.1 googleapis-common-protos-1.5.2 grpc-google-iam-v1-0.11.1 grpcio-1.4.0 httplib2-0.10.3 idna-2.5 oauth2client-3.0.0 ply-3.8 proto-google-cloud-datastore-v1-0.90.4 proto-google-cloud-error-reporting-v1beta1-0.15.3 proto-google-cloud-logging-v2-0.91.3 proto-google-cloud-pubsub-v1-0.15.4 proto-google-cloud-spanner-admin-database-v1-0.15.3 proto-google-cloud-spanner-admin-instance-v1-0.15.3 proto-google-cloud-spanner-v1-0.15.3 proto-google-cloud-speech-v1-0.15.3 protobuf-3.3.0 pyasn1-0.2.3 pyasn1-modules-0.0.9 requests-2.18.1 rsa-3.4.2 six-1.10.0 urllib3-1.21.1

nhammond · 2017-07-05T20:14:34Z

Please double-check the repro steps before deciding this is not reproducible. Pip install succeeds, but running a pip-installed executable that requires google-cloud 0.25.0 fails.

The repro steps above have a nice easy way to reproduce that error.

dhermes · 2017-07-05T20:35:47Z

Ah, my bad! Confirmed:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial
$
$ virtualenv --python=/usr/bin/python venv
Running virtualenv with interpreter /usr/bin/python
New python executable in /home/dhermes/google_contracting/google-auth-library-python/venv/bin/python
Installing setuptools, pip, wheel...done.
$
$ source venv/bin/activate
(venv) $
(venv) $ venv/bin/pip install google-cloud==0.25.0
Collecting google-cloud==0.25.0
  Downloading google_cloud-0.25.0-py2.py3-none-any.whl
...
Successfully installed cachetools-2.0.0 ...
(venv) $ venv/bin/python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> __requires__ = ['google-cloud']
>>> import pkg_resources
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  ...
  File ".../pkg_resources/__init__.py", line 3037, in <module>
    @_call_aside
  File ".../pkg_resources/__init__.py", line 3021, in _call_aside
    f(*args, **kwargs)
  File ".../pkg_resources/__init__.py", line 3050, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File ".../pkg_resources/__init__.py", line 657, in _build_master
    return cls._build_from_requirements(__requires__)
  File ".../pkg_resources/__init__.py", line 670, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File ".../pkg_resources/__init__.py", line 860, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (google-cloud-core 0.24.1 (.../venv/lib/python2.7/site-packages), Requirement.parse('google-cloud-core<0.26dev,>=0.25.0'), set(['google-cloud-storage', 'google-cloud-logging', 'google-cloud-datastore']))
>>>

Any reason that google-cloud==0.25.0 is pinned?

A picture of the installed packages may give a hint:

(venv) $ venv/bin/pip freeze | grep -e ^google-cloud
google-cloud==0.25.0
google-cloud-bigquery==0.24.0
google-cloud-bigtable==0.24.0
google-cloud-core==0.24.1
google-cloud-datastore==1.1.0
google-cloud-dns==0.24.0
google-cloud-error-reporting==0.24.2
google-cloud-language==0.24.1
google-cloud-logging==1.1.0
google-cloud-monitoring==0.24.0
google-cloud-pubsub==0.25.0
google-cloud-resource-manager==0.24.0
google-cloud-runtimeconfig==0.24.0
google-cloud-spanner==0.24.2
google-cloud-speech==0.24.0
google-cloud-storage==1.2.0
google-cloud-translate==0.24.0
google-cloud-vision==0.24.0

dhermes · 2017-07-05T20:41:46Z

I am still sleuthing, but I'm pretty sure the issue is that our "stable" packages have looser bounds:

    'google-cloud-datastore >= 1.0.0, < 2.0dev',
    ...
    'google-cloud-logging >= 1.0.0, < 2.0dev',
    ...
    'google-cloud-storage >= 1.1.0, < 2.0dev',

so we get newer version of those packages that depend on google-cloud-core==0.25.x in this old release that depends on google.cloud.core>=0.24.1, < 0.25dev

dhermes · 2017-07-05T20:47:57Z

dhermes · 2017-07-05T20:52:17Z

OK, after clicking into all of the setup.py files above, I have found the 3 culprits (which confirms my theory):

duggelz · 2017-07-05T22:15:19Z

Could this be related to pip doesn't resolve dependencies correctly?

dhermes · 2017-07-05T22:24:33Z

@duggelz In this case, it's just our fault for being too "loose" with our restrictions on the "stable" packages.

UPDATE: It would be nice if pip would fail the install though, since our failure isn't clear until something after install.

dhermes · 2017-07-06T19:15:57Z

OK I'm going to send some fixes for this right now. I just pushed two branches:

corresponding to the existing "broken" tags:

$ git log -1 0.25.0 --pretty=%H
bd9fdfba07fac63a91847628613928a569250c0f
$ git log -1 0.26.0 --pretty=%H
214aba604fbcaa9e4936fa1798efde050389992a

I will send PRs (#3584, #3586, #3587, #3588) against them to create 0.25.1 and 0.26.1 tags, and then will delete those branches from the main repo once the new tags are created.

dhermes · 2017-07-06T19:25:35Z

We've got similar issues in 0.24.0:

$ git show 0.24.0:setup.py | grep -A 18 'REQUIREMENTS = \['
REQUIREMENTS = [
    'google-cloud-bigquery >= 0.24.0, < 0.25dev',
    'google-cloud-bigtable >= 0.24.0, < 0.25dev',
    'google-cloud-core >= 0.24.0, < 0.25dev',
    'google-cloud-datastore >= 1.0.0, < 2.0dev',
    'google-cloud-dns >= 0.24.0, < 0.25dev',
    'google-cloud-error-reporting >= 0.24.0, < 0.25dev',
    'google-cloud-language >= 0.24.0, < 0.25dev',
    'google-cloud-logging >= 0.24.0, < 0.25dev',
    'google-cloud-monitoring >= 0.24.0, < 0.25dev',
    'google-cloud-pubsub >= 0.24.0, < 0.25dev',
    'google-cloud-resource-manager >= 0.24.0, < 0.25dev',
    'google-cloud-spanner >= 0.24.0, < 0.25dev',
    'google-cloud-speech >= 0.24.0, < 0.25dev',
    'google-cloud-storage >= 1.0.0, < 2.0dev',
    'google-cloud-translate >= 0.24.0, < 0.25dev',
    'google-cloud-vision >= 0.24.0, < 0.25dev',
    'google-cloud-runtimeconfig >= 0.24.0, < 0.25dev',
]

I confirmed it "stops" there, the previous release was "pre-stable"

$ git show 0.23.0:setup.py | grep -A 18 'REQUIREMENTS = \['
REQUIREMENTS = [
    'google-cloud-bigquery >= 0.23.0, < 0.24dev',
    'google-cloud-bigtable >= 0.23.0, < 0.24dev',
    'google-cloud-core >= 0.23.1, < 0.24dev',
    'google-cloud-datastore >= 0.23.0, < 0.24dev',
    'google-cloud-dns >= 0.23.0, < 0.24dev',
    'google-cloud-error-reporting >= 0.23.0, < 0.24dev',
    'google-cloud-language >= 0.23.0, < 0.24dev',
    'google-cloud-logging >= 0.23.0, < 0.24dev',
    'google-cloud-monitoring >= 0.23.0, < 0.24dev',
    'google-cloud-pubsub >= 0.23.0, < 0.24dev',
    'google-cloud-resource-manager >= 0.23.0, < 0.24dev',
    'google-cloud-spanner >= 0.23.1, < 0.24dev',
    'google-cloud-speech >= 0.23.0, < 0.24dev',
    'google-cloud-storage >= 0.23.0, < 0.24dev',
    'google-cloud-translate >= 0.23.0, < 0.24dev',
    'google-cloud-vision >= 0.23.0, < 0.24dev',
    'google-cloud-runtimeconfig >= 0.23.0, < 0.24dev',
]

dhermes · 2017-07-06T21:06:24Z

OK the following releases have come as a result:

I am closing this as "mostly fixed".

@nhammond In the current form 0.25.0 will "never" be fixed and IIUC that's the "right" way to "fix" a PyPI issue (rather than overwriting the content). However 0.25.1 is "ready to go" as is 0.26.1.

nhammond · 2017-07-06T22:51:30Z

Thanks @dhermes, I understand not wanting to overwrite the release even if it leaves it in a somewhat broken state.

Given that this just happened a few months ago (#3331), are there any safeguards against repeating the mistake? It will become quite frustrating to users if this breaks in the same mode with every new release of google-cloud.

dhermes · 2017-07-06T22:56:05Z

I actually wondered that to myself as well. It'd involve starting with all matches for packages in a range, e.g.

google-cloud-storage >= 1.1.0, < 1.2dev

yields

google-cloud-storage==1.1.0
google-cloud-storage==1.1.1

then we just install all combinations and then "check" if OK (e.g. via pkg_resources as you provided). There are at least two issues with this though that come to mind:

Combinatorial explosion of valid packages
"Recursive" dependencies (i.e. following dependencies of dependencies that are within our control)

@jonparrott is actually on the PyPA (packaging authority) so he would be great here, but he is OOO for the next week or so.

duggelz · 2017-07-06T23:31:44Z

The combinatorial explosion is real. I recommended (internally) that we just test against 1) the latest of combination of "everything", 2) the earliest version of "everything" that still satisfies the dependency specs, and 3) a handful (or less) of particular combinations that are important. I think this would catch most of the problems with a minimum amount (2x or 3x) of the testing resources. And some kind of abbreviated test where we don't actually run the tests, we just import everything, might mitigate that as well.

theacodes · 2017-07-13T16:50:05Z

I agree with @duggelz's suggestion of installing all of the latest stuff and running a sanity check import and then another test to install the latest umbrella with the minimum versions of its dependencies. We could do this as a pre-release check.

I would also vote in favor of killing the umbrella package. It's going to become more and more useless as we go on.

I'm also in favor of figuring out a plan to v1.0.0 google-cloud-core (& merge it with gax) so that we can stop breaking packages because of that.

dhermes · 2017-07-13T16:52:17Z

I would also vote in favor of killing the umbrella package. It's going to become more and more useless as we go on.

The moment we had more sub-packages (like > 5) than a person could (reasonably) list from memory, the umbrella package became cumbersome. (Which is especially funny because the first release after the breakup of the gcloud package had 7 sub-packages.)

tseaver · 2017-07-13T17:26:09Z

I was always suspicious of the "give me everything" usecase for the umbrella package, so +1 for losing it.

…eapis#3591)

lukesneeringer assigned dhermes Jul 5, 2017

tseaver added the packaging label Jul 6, 2017

This was referenced Jul 6, 2017

Making requirements tighter for "stable" packages in 0.25.1 release. #3584

Merged

Making requirements tighter for "stable" packages in 0.26.1 release. #3586

Merged

This was referenced Jul 6, 2017

Making requirements tighter for "stable" packages in error reporting 0.24.3 release. #3587

Merged

Making requirements tighter for "stable" packages in error reporting 0.25.1 release. #3588

Merged

dhermes closed this as completed Jul 6, 2017

dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jul 6, 2017

Patch in the version updates made when fixing googleapis#3579.

ddd587d

dhermes mentioned this issue Jul 6, 2017

Patch in the version updates made when fixing #3579. #3591

Merged

dhermes added a commit that referenced this issue Jul 11, 2017

Patch in the version updates made when fixing #3579. (#3591)

569e739

Gwildor mentioned this issue Aug 14, 2017

gcloud==0.18.3 and google-cloud-datastore==1.2.0 run into dependency conflicts #3807

Closed

landrito pushed a commit to landrito/google-cloud-python that referenced this issue Aug 21, 2017

Patch in the version updates made when fixing googleapis#3579. (googl…

1c4f3c1

…eapis#3591)

landrito pushed a commit to landrito/google-cloud-python that referenced this issue Aug 22, 2017

Patch in the version updates made when fixing googleapis#3579. (googl…

4b4918a

…eapis#3591)

landrito pushed a commit to landrito/google-cloud-python that referenced this issue Aug 22, 2017

Patch in the version updates made when fixing googleapis#3579. (googl…

2db88f1

…eapis#3591)

lukesneeringer mentioned this issue Aug 28, 2017

ImportError: No module named 'google.api.core' #3885

Closed

theacodes unassigned dhermes Sep 28, 2018

JustinBeckwith assigned dhermes Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conflicting Dependencies #3579

Conflicting Dependencies #3579

lukesneeringer commented Jul 5, 2017 •

edited by dhermes

dhermes commented Jul 5, 2017

tseaver commented Jul 5, 2017

nhammond commented Jul 5, 2017

dhermes commented Jul 5, 2017

dhermes commented Jul 5, 2017 •

edited

dhermes commented Jul 5, 2017 •

edited

dhermes commented Jul 5, 2017

duggelz commented Jul 5, 2017

dhermes commented Jul 5, 2017 •

edited

dhermes commented Jul 6, 2017 •

edited

dhermes commented Jul 6, 2017

dhermes commented Jul 6, 2017

nhammond commented Jul 6, 2017

dhermes commented Jul 6, 2017

duggelz commented Jul 6, 2017

theacodes commented Jul 13, 2017 •

edited

dhermes commented Jul 13, 2017

tseaver commented Jul 13, 2017

Conflicting Dependencies #3579

Conflicting Dependencies #3579

Comments

lukesneeringer commented Jul 5, 2017 • edited by dhermes

dhermes commented Jul 5, 2017

tseaver commented Jul 5, 2017

nhammond commented Jul 5, 2017

dhermes commented Jul 5, 2017

dhermes commented Jul 5, 2017 • edited

dhermes commented Jul 5, 2017 • edited

dhermes commented Jul 5, 2017

duggelz commented Jul 5, 2017

dhermes commented Jul 5, 2017 • edited

dhermes commented Jul 6, 2017 • edited

dhermes commented Jul 6, 2017

dhermes commented Jul 6, 2017

nhammond commented Jul 6, 2017

dhermes commented Jul 6, 2017

duggelz commented Jul 6, 2017

theacodes commented Jul 13, 2017 • edited

dhermes commented Jul 13, 2017

tseaver commented Jul 13, 2017

lukesneeringer commented Jul 5, 2017 •

edited by dhermes

dhermes commented Jul 5, 2017 •

edited

dhermes commented Jul 5, 2017 •

edited

dhermes commented Jul 5, 2017 •

edited

dhermes commented Jul 6, 2017 •

edited

theacodes commented Jul 13, 2017 •

edited