two copies of pyasn1_modules in $PYTHONPATH; doesn't work with remote_api_shell.py

[faried]

Python 2.7.13 on Mac OS 10.11.6, with google-cloud 0.22.0, Cloud SDK 138.0.0.

Installing google-cloud using pip downloads and installs pyasn1_modules as a dependency, but it is newer than the one that ships with the platform.

For my managed App Engine project, I use the vendor system. My appengine_config.py has

from google.appengine.ext import vendor
vendor.add("lib")

I installed google-cloud using mkdir lib; pip install -t lib google-python and my service works fine both when deployed and when I use dev_appserver on my computer. However, I can't seem to use google-cloud with remote_api_shell.py:

$ ~/google-cloud-sdk/platform/google_appengine/remote_api_shell.py \
    -s someproject.appspot.com
App Engine remote_api shell
Python 2.7.13 (default, Dec 17 2016, 23:03:43) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
The db, ndb, users, urlfetch, and memcache modules are imported.
s~someproject> from google.appengine.ext import vendor
s~someproject> vendor.add("lib")
s~someproject> from google.cloud import storage
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/fn/someproject/lib/google/cloud/storage/__init__.py", line 36, in <module>
    from google.cloud.storage.client import Client
  File "/Users/fn/someproject/lib/google/cloud/storage/client.py", line 19, in <module>
    from google.cloud.client import JSONClient
  File "/Users/fn/someproject/appengine/python/lib/google/cloud/client.py", line 18, in <module>
    from google.oauth2 import service_account
  File "/Users/fn/someproject/lib/google/oauth2/service_account.py", line 76, in <module>
    from google.auth import _service_account_info
  File "/Users/fn/someproject/lib/google/auth/_service_account_info.py", line 22, in <module>
    from google.auth import crypt
  File "/Users/fn/someproject/python/lib/google/auth/crypt.py", line 45, in <module>
    from pyasn1_modules import pem
ImportError: cannot import name pem
s~someproject> 

It looks like I have pyasn1_modules both in my lib directory (installed as a dependency for google-cloud) and in

/Users/fn/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules/pyasn1_modules/

The former has pem.py but the latter doesn't.

[dhermes]

High-level: why are you using the remote API shell to import google.cloud.storage? You shouldn't ever need to do that.

---

@jonparrott is our resident GAE expert, though I'll try to help. We should probably move this issue over to https://github.com/GoogleCloudPlatform/google-auth-library-python/ though since the issue is with the auth library (not really though, since it's just a PYTHONPATH issue).

From the remote API shell, execute the following:

>>> import pyasn1_modules
>>> pyasn1_modules
<module 'pyasn1_modules' from '/usr/local/lib/python2.7/dist-packages/pyasn1_modules/__init__.pyc'>
>>> pyasn1_modules.__file__
'/usr/local/lib/python2.7/dist-packages/pyasn1_modules/__init__.pyc'

[faried]

I recently had a bug in my deployed application where some datastore entities were out of sync with their related files on cloud storage. I used the remote API shell to loop over the affected objects, figure out if the referenced files existed, and fix up the objects. I ended up using stat() from the cloudstore package.

For me, your commands point to pyasn1_modules in the platform directory:

$ ~/google-cloud-sdk/platform/google_appengine/remote_api_shell.py -s someproject.appspot.com
App Engine remote_api shell
Python 2.7.13 (default, Dec 17 2016, 23:03:43) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
The db, ndb, users, urlfetch, and memcache modules are imported.
s~someproject> from google.appengine.ext import vendor
s~someproject> vendor.add("lib")
s~someproject> import pyasn1_modules
s~someproject> pyasn1_modules.__file__
'/Users/fn/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules/pyasn1_modules/__init__.pyc'
s~someproject> 

I don't have pyasn1_modules installed system-wide, just in my project's lib directory. Since the script prepends its own directory's path into sys.path, trying env PYTHONPATH=lib .../remote_api_shell.py doesn't help either.

[dhermes]

@faried Great, it's possible that sys.path has the platform libs before your lib, but I don't think that's true.

What I really think is happening: the remote shell pre-loads a bunch of stuff, some of which imports pyasn1_modules BEFORE you run any code (in particular, before you run vendor.add("lib")). That mean future import pyasn1_modules will just grab the already imported module in sys.modules.

You can run sys.modules.pop('pyasn1_modules') before importing google.cloud.storage and it should fix it? Also setting PYTHONPATH for the remote shell should fix it, but I can't vouch for what the shell is doing underneath.

[dhermes]

@faried I am pre-emptively closing out (so I don't forget to sweep it up later). LMK if this doesn't work and we can re-open.

[faried]

it's possible that sys.path has the platform libs before your lib, but I don't think that's true.

It does. I can't find a recent copy of remote_api_shell.py on github, but the one in my platform/google_appengine directory has this right at the top:

import os
import sys
import time

sys_path = sys.path
try:
  sys.path = [os.path.dirname(__file__)] + sys.path

  import wrapper_util

finally:
  sys.path = sys_path

If I run env PYTHONPATH=lib ~/google-cloud-sdk/platform/google_appengine/remote_api_shell.py -s someproject.appspot.com about 19 directories from inside /Users/fn/google-cloud-sdk/platform/google_appengine/ end up in sys.path before my lib directory, including platform/google_appengine/lib/pyasn1_modules.

Your suggestion of running sys.modules.pop('pyasn1_modules') does work around the problem.

I don't know where the appengine repository is, but I think an issue should be created there to update their copy of pyasn1_modules. In fact, platform/gsutil, platform/google_appengine and platform/bq all have different versions of pyasn1_modules, none of them the most recent one (0.0.8, released September 2015).

[dhermes]

Thanks for the info @faried.

@jonparrott Where to report gcloud CLI/SDK bugs?