Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No module named 'google.protobuf' using --target on amazonlinux container #5154

Closed
ryantuck opened this issue Apr 5, 2018 · 2 comments
Closed

No module named 'google.protobuf' using --target on amazonlinux container #5154

ryantuck opened this issue Apr 5, 2018 · 2 comments
Assignees
@tseaver
Labels
api: bigquery Issues related to the BigQuery API. packaging

Comments

@ryantuck
Copy link

ryantuck commented Apr 5, 2018 •
edited

Problem

Installing google-cloud-bigquery via pip to a specific directory in an amazonlinux docker container does not work as expected.

$ pip3.6 install google-cloud-bigquery -t /tmp/py
$ python3.6
>>> import sys
>>> sys.path.append('/tmp/py')
>>> from google.cloud import bigquery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/py/google/cloud/bigquery/__init__.py", line 34, in <module>
    from google.cloud.bigquery._helpers import DEFAULT_RETRY
  File "/tmp/py/google/cloud/bigquery/_helpers.py", line 21, in <module>
    from google.cloud._helpers import UTC
  File "/tmp/py/google/cloud/_helpers.py", line 33, in <module>
    from google.protobuf import duration_pb2
ModuleNotFoundError: No module named 'google.protobuf'

Possibly helpful notes

  • Doing a 'normal' pip3.6 install google-cloud-bigquery on this container works as expected.
  • This method does work when running in a frolvlad/alpine-python3 container.
  • I've tried this for google-cloud-bigquery versions 0.28.0, 0.31.0, and 0.32.0, and they all fail.

What i'm trying to do

I need to install python packages to a specific directory so that I can create a deploy package for AWS lambda.

I do this by running the following within an amazonlinux-based docker container (to mirror any system-level .so file oddities that crop up with some libraries).

$ pip install -r requirements.txt -t my_python_packages

In my python code in lambda, I then do the following to ensure all modules installed to this directory are available to the rest of my code:

import sys
sys.path.insert(0, './my_python_packages')
# ... rest of code ...

This works fine for all other packages I've used, but breaks when attempting to install google-cloud-bigquery.

steps to reproduce

  1. Build the docker image specified by the below Dockerfile.
  2. Repeat my steps below.

Dockerfile

FROM amazonlinux

# python3.6
RUN yum install -y https://rhel6.iuscommunity.org/ius-release.rpm
RUN yum install -y \
        python36u \
        python36u-pip \
        python36u-devel

Stack Trace

This example shows how I can install requests just fine, but not google-cloud-bigquery.

⚡  ~/Desktop ⚡  docker run -it amazonlinux-python36 /bin/bash
bash-4.2# pip3.6 list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
setuptools (36.6.0)
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
bash-4.2# mkdir /tmp/py
bash-4.2# pip3.6 install requests -t /tmp/py
Collecting requests
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |################################| 92kB 1.3MB/s
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |################################| 133kB 2.3MB/s
Collecting certifi>=2017.4.17 (from requests)
  Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)
    100% |################################| 153kB 2.8MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |################################| 143kB 3.2MB/s
Collecting idna<2.7,>=2.5 (from requests)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |################################| 61kB 2.2MB/s
Installing collected packages: urllib3, certifi, chardet, idna, requests
Successfully installed certifi-2018.1.18 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
bash-4.2# python3.6
Python 3.6.4 (default, Dec 19 2017, 14:48:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/tmp/py')
>>> import requests
>>> exit()
bash-4.2# pip3.6 install google-cloud-bigquery -t /tmp/py
Collecting google-cloud-bigquery
  Downloading google_cloud_bigquery-0.32.0-py2.py3-none-any.whl (73kB)
    100% |################################| 81kB 1.2MB/s
Collecting google-resumable-media>=0.2.1 (from google-cloud-bigquery)
  Downloading google_resumable_media-0.3.1-py2.py3-none-any.whl
Collecting google-cloud-core<0.29dev,>=0.28.0 (from google-cloud-bigquery)
  Downloading google_cloud_core-0.28.1-py2.py3-none-any.whl
Collecting google-api-core<2.0.0dev,>=1.0.0 (from google-cloud-bigquery)
  Downloading google_api_core-1.1.0-py2.py3-none-any.whl (51kB)
    100% |################################| 61kB 4.1MB/s
Collecting six (from google-resumable-media>=0.2.1->google-cloud-bigquery)
  Downloading six-1.11.0-py2.py3-none-any.whl
Collecting requests<3.0.0dev,>=2.18.0 (from google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting pytz (from google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)
    100% |################################| 512kB 1.2MB/s
Collecting protobuf>=3.0.0 (from google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading protobuf-3.5.2.post1-cp36-cp36m-manylinux1_x86_64.whl (6.4MB)
    100% |################################| 6.4MB 99kB/s
Collecting setuptools>=34.0.0 (from google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading setuptools-39.0.1-py2.py3-none-any.whl (569kB)
    100% |################################| 573kB 880kB/s
Collecting google-auth<2.0.0dev,>=0.4.0 (from google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading google_auth-1.4.1-py2.py3-none-any.whl (65kB)
    100% |################################| 71kB 3.7MB/s
Collecting googleapis-common-protos<2.0dev,>=1.5.3 (from google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading googleapis-common-protos-1.5.3.tar.gz
Collecting idna<2.7,>=2.5 (from requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Using cached certifi-2018.1.18-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth<2.0.0dev,>=0.4.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading pyasn1_modules-0.2.1-py2.py3-none-any.whl (60kB)
    100% |################################| 61kB 3.5MB/s
Collecting cachetools>=2.0.0 (from google-auth<2.0.0dev,>=0.4.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading cachetools-2.0.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from google-auth<2.0.0dev,>=0.4.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |################################| 51kB 2.7MB/s
Collecting pyasn1<0.5.0,>=0.4.1 (from pyasn1-modules>=0.2.1->google-auth<2.0.0dev,>=0.4.0->google-api-core<2.0.0dev,>=1.0.0->google-cloud-bigquery)
  Downloading pyasn1-0.4.2-py2.py3-none-any.whl (71kB)
    100% |################################| 71kB 2.9MB/s
Installing collected packages: six, google-resumable-media, idna, certifi, chardet, urllib3, requests, pytz, setuptools, protobuf, pyasn1, pyasn1-modules, cachetools, rsa, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-bigquery
  Running setup.py install for googleapis-common-protos ... done
Successfully installed cachetools-2.0.1 certifi-2018.1.18 chardet-3.0.4 google-api-core-1.1.0 google-auth-1.4.1 google-cloud-bigquery-0.32.0 google-cloud-core-0.28.1 google-resumable-media-0.3.1 googleapis-common-protos-1.5.3 idna-2.6 protobuf-3.5.2.post1 pyasn1-0.4.2 pyasn1-modules-0.2.1 pytz-2018.3 requests-2.18.4 rsa-3.4.2 setuptools-39.0.1 six-1.11.0 urllib3-1.22
Target directory /tmp/py/chardet already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/requests already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/certifi-2018.1.18.dist-info already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/idna already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/requests-2.18.4.dist-info already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/idna-2.6.dist-info already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/urllib3-1.22.dist-info already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/urllib3 already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/chardet-3.0.4.dist-info already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/certifi already exists. Specify --upgrade to force replacement.
Target directory /tmp/py/google already exists. Specify --upgrade to force replacement.
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
bash-4.2# python3.6
Python 3.6.4 (default, Dec 19 2017, 14:48:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/tmp/py')
>>> from google.cloud import bigquery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/py/google/cloud/bigquery/__init__.py", line 34, in <module>
    from google.cloud.bigquery._helpers import DEFAULT_RETRY
  File "/tmp/py/google/cloud/bigquery/_helpers.py", line 21, in <module>
    from google.cloud._helpers import UTC
  File "/tmp/py/google/cloud/_helpers.py", line 33, in <module>
    from google.protobuf import duration_pb2
ModuleNotFoundError: No module named 'google.protobuf'
>>>
@ryantuck
Copy link
Author

ryantuck commented Apr 5, 2018

Found the following related issues that appear to relate to the fact that pip is installing some stuff in /usr/lib64/python3.6/site-packages/, and the rest in /usr/lib/python3.6/site-packages.

pypa/pip#4464

#4039

My workaround for this issue, in case anyone is interested, is to instead simply install to my system and copy those directory contents to where I need them:

$ pip3.6 install -r requirements.txt
$ mkdir /tmp/lib
$ mkdir /tmp/lib64
$ cp /usr/lib/python3.6/site-packages/* /tmp/lib
$ cp /usr/lib64/python3.6/site-packages/* /tmp/lib64

and then:

import sys
sys.path.insert(0, './lib64')
sys.path.insert(0, './lib')

@chemelnucfin chemelnucfin added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Apr 6, 2018
@tseaver tseaver added packaging and removed priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Apr 10, 2018
@tseaver
Copy link
Contributor

tseaver commented Apr 10, 2018

I can't see that this is an issue with google-cloud-bigquery at all: it is a bug in pip.

@tseaver tseaver closed this as completed Apr 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tseaver tseaver

Labels
api: bigquery Issues related to the BigQuery API. packaging
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tseaver @ryantuck @chemelnucfin