Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datastore: protobuf dependency requirements incorrect #6390

Closed
aarontp opened this issue Nov 5, 2018 · 2 comments
Closed

Datastore: protobuf dependency requirements incorrect #6390

aarontp opened this issue Nov 5, 2018 · 2 comments
Assignees
Labels
api: core api: datastore Issues related to the Datastore API. priority: p2 Moderately-important priority. Fix may not be included in next release.

Comments

@aarontp
Copy link

aarontp commented Nov 5, 2018

I believe that the indirect dependency requirements for the protobuf package is not correct. This is similar to #6128 . Maybe the minimum version of google-api-core needs to be updated? This seems to work with google-cloud-datastore==1.7.0, but then breaks with version 1.7.1.

The breaking changes appear to be in this commit:
6a6f1e1#diff-677d93da9ffc63ac9fceae97c454bc55

OS type and version: Linux/Ubuntu 14.04.5 LTS
Python version and virtual environment information:

Python 2.7.6
$ pip --version
pip 9.0.1 from /home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages (python 2.7)```

4. google-cloud-<service> version:  `google-cloud-datastore==1.7.1`. 

5. Stacktrace if available
```Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/travis/build/log2timeline/dftimewolf/tests/lib/processors/turbinia.py", line 10, in <module>
    from dftimewolf.lib.processors import turbinia
  File "./dftimewolf/lib/processors/turbinia.py", line 10, in <module>
    from turbinia import client as turbinia_client
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/turbinia/client.py", line 29, in <module>
    from turbinia import task_manager
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/turbinia/task_manager.py", line 25, in <module>
    from turbinia import state_manager
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/turbinia/state_manager.py", line 38, in <module>
    from google.cloud import datastore
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/google/cloud/datastore/__init__.py", line 61, in <module>
    from google.cloud.datastore.batch import Batch
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/google/cloud/datastore/helpers.py", line 29, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/google/cloud/datastore_v1/__init__.py", line 17, in <module>
    from google.cloud.datastore_v1 import types
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/google/cloud/datastore_v1/types.py", line 26, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/google/cloud/datastore_v1/proto/datastore_pb2.py", line 17, in <module>
    from google.cloud.datastore_v1.proto import entity_pb2 as google_dot_cloud_dot_datastore__v1_dot_proto_dot_entity__pb2
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/google/cloud/datastore_v1/proto/entity_pb2.py", line 46, in <module>
    options=None, file=DESCRIPTOR)
TypeError: __new__() got an unexpected keyword argument 'file'
  1. Steps to reproduce:
    from google.cloud import datastore

  2. Code example
    from google.cloud import datastore

@tseaver tseaver added api: datastore Issues related to the Datastore API. priority: p2 Moderately-important priority. Fix may not be included in next release. api: clouddebugger Issues related to the Cloud Debugger API. labels Nov 5, 2018
@tseaver tseaver self-assigned this Nov 5, 2018
@tseaver
Copy link
Contributor

tseaver commented Nov 5, 2018

Thanks for the follow-up. #6391 fixes that first for datastore, and then adds normalization for dependencies in all our setup.py files to prevent similar issues. I also added an issue and a pull request for the gapic-generator to ensure that it emits correct minimum dependency in a synth-cycle.

@aarontp
Copy link
Author

aarontp commented Nov 5, 2018

Thanks much for the fast fix. The longer term fixes are appreciated as well, thanks!

aarontp added a commit to google/turbinia that referenced this issue Nov 5, 2018
The minimum library versions are not set correctly for some of the upstream cloud libraries so we'll need to pin them for now.  See the following for more details on the bug: 
googleapis/google-cloud-python#6390

This can be removed once grr-client-api updates its pinned version of protobuf.  The grr-client-api library gets pulled in from dfTimewolf, which also has Turbinia as a dependency.   The bug referenced above gets tickled when both of these libraries are installed at the same time (because the Datastore package doesn't correctly limit the minimum versions of its dependencies, an incompatible version gets installed).
Onager pushed a commit to google/turbinia that referenced this issue Nov 6, 2018
The minimum library versions are not set correctly for some of the upstream cloud libraries so we'll need to pin them for now.  See the following for more details on the bug: 
googleapis/google-cloud-python#6390

This can be removed once grr-client-api updates its pinned version of protobuf.  The grr-client-api library gets pulled in from dfTimewolf, which also has Turbinia as a dependency.   The bug referenced above gets tickled when both of these libraries are installed at the same time (because the Datastore package doesn't correctly limit the minimum versions of its dependencies, an incompatible version gets installed).
@tseaver tseaver added api: core and removed api: clouddebugger Issues related to the Cloud Debugger API. labels Nov 7, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: core api: datastore Issues related to the Datastore API. priority: p2 Moderately-important priority. Fix may not be included in next release.
Projects
None yet
Development

No branches or pull requests

2 participants