"DistributionNotFound" error after deployment

[4lph4-Ph4un]

Context

After deploying I get an error stating:

Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

zappa tail provides this info:

The 'gcloud' distribution was not found and is required by the application: DistributionNotFound
Traceback (most recent call last):
  File "/var/task/handler.py", line 567, in lambda_handler
  return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 237, in lambda_handler
  handler = cls()
  File "/var/task/handler.py", line 129, in __init__
  self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmp/src/appstatic.py", line 2, in <module>
  from instababy import create_app
  File "/tmp/src/instababy/__init__.py", line 5, in <module>
  from . import utils
  File "/tmp/src/instababy/utils/__init__.py", line 1, in <module>
  from . import s3, hasher, firebase, authentication
  File "/tmp/src/instababy/utils/firebase.py", line 1, in <module>
  import pyrebase
  File "/tmp/src/pyrebase/__init__.py", line 1, in <module>
  from .pyrebase import initialize_app
  File "/tmp/src/pyrebase/pyrebase.py", line 18, in <module>
  from gcloud import storage
  File "/tmp/src/gcloud/__init__.py", line 19, in <module>
  __version__ = get_distribution('gcloud').version
  File "/tmp/src/pkg_resources/__init__.py", line 476, in get_distribution
  dist = get_provider(dist)
  File "/tmp/src/pkg_resources/__init__.py", line 352, in get_provider
  return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/tmp/src/pkg_resources/__init__.py", line 895, in require
  needed = self.resolve(parse_requirements(requirements))
  File "/tmp/src/pkg_resources/__init__.py", line 781, in resolve
  raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'gcloud' distribution was not found and is required by the application

Expected Behavior

Zappa deploys the packaged application while libraries are intact

Actual Behavior

Dependency is not found.

Your Environment

• Zappa version used: 0.46.2
• Operating System and Python version: ubuntu-17.10 and Python 3.6.1
• The output of pip freeze:

alembic==1.0.0
argcomplete==1.9.3
asn1crypto==0.24.0
atomicwrites==1.2.1
attrs==18.1.0
aws-xray-sdk==0.95
Babel==2.6.0
base58==1.0.0
blinker==1.4
boto==2.49.0
boto3==1.8.5
botocore==1.11.5
CacheControl==0.12.5
cachetools==2.1.0
certifi==2018.8.24
cffi==1.7.0
cfn-flip==1.0.3
chardet==3.0.4
click==6.7
cookies==2.2.1
cryptography==2.1.1
docker==3.5.0
docker-pycreds==0.3.0
docutils==0.14
durationpy==0.5
ecdsa==0.13
firebase-admin==2.13.0
Flask==1.0
Flask-Babel==0.11.2
Flask-Mail==0.9.1
Flask-Migrate==2.2.1
Flask-SQLAlchemy==2.2
future==0.16.0
gcloud==0.18.3
GeoAlchemy2==0.5.0
google-api-core==1.3.0
google-auth==1.5.1
google-cloud-core==0.28.1
google-cloud-firestore==0.29.0
google-cloud-storage==1.11.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
hashids==1.2.0
hjson==3.0.1
httplib2==0.11.3
idna==2.7
inflection==0.3.1
itsdangerous==0.24
Jinja2==2.10
jmespath==0.9.3
jsondiff==1.1.1
jsonpickle==0.9.6
jws==0.1.3
kappa==0.6.0
lambda-packages==0.20.0
Mako==1.0.7
MarkupSafe==1.0
marshmallow==2.15.3
mock==2.0.0
more-itertools==4.3.0
moto==1.3.5
msgpack==0.5.6
numpy==1.13.0
oauth2client==3.0.0
opencv-python==3.2.0.7
packaging==17.1
pbr==4.2.0
placebo==0.8.1
pluggy==0.7.1
protobuf==3.6.1
psycopg2==2.7.3.1
py==1.6.0
pyaml==17.12.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycparser==2.18
pycryptodome==3.4.3
PyJWT==1.5.0
pyparsing==2.2.0
Pyrebase==3.0.27
pytest==3.7.4
pytest-flask==0.11.0
python-dateutil==2.6.1
python-editor==1.0.3
python-google-places==1.4.0
python-jose==2.0.2
python-jwt==2.0.1
python-slugify==1.2.4
pytz==2018.5
PyYAML==3.12
raven==6.9.0
requests==2.19.1
requests-toolbelt==0.7.0
responses==0.9.0
rsa==3.4.2
s3transfer==0.1.13
six==1.11.0
SQLAlchemy==1.1.10
toml==0.9.4
tqdm==4.19.1
troposphere==2.3.1
Unidecode==1.0.22
urllib3==1.23
validate-email==1.3
websocket-client==0.51.0
Werkzeug==0.14.1
wrapt==1.10.11
wsgi-request-logger==0.4.6
xmltodict==0.11.0
zappa==0.46.2

• Your zappa_settings.py:

"dev": {
        "app_function": "appstatic.app",
        "s3_bucket": "zappa-2hcqm2bhh",
        "remote_env": "s3://zappa-2hcqm2bhh/config.json",
	    "manage_roles": false,
        "runtime": "python3.6",
        "memory_size": 1536,
        "role_name": "AppLambdaExcecuteRole",
        "log_level": "ERROR",
        "slim_handler": true,
        "vpc_config": {
            "SubnetIds": [ "subnet-828372e5","subnet-0d80b255","subnet-c457a48d" ],
            "SecurityGroupIds": [ "sg-b7f66ece" ]
        },
        "events": [{
            "function": "appstatic.process_new_media",
            "event_source": {
                  "arn":  "arn:aws:s3:::instababy-media-inbox",
                  "events": [
                    "s3:ObjectCreated:*"
                  ]
               }
            }]
    },

[paulina-mudano]

Seems to be similar to #1555 - this error is caused when slim_handler: true and I cannot find a reason why :(

[4lph4-Ph4un]

Some updates. This seems to be a environmental configuration issue, rather than simple Zappa issue. The deployment is succesfull if I run it locally from my dev machine, but currently fails on our CI -machines. I'll provide some extra info if someone else might suffer from the same issue. Also if someone here has experience configuring Circle2.0 with Zappa, some additional insight might be appreaciated :)

[paulina-mudano]

@4lph4-Ph4un I pinpointed the issue (see #1602). It depends on the system you're using, and whether pip installs dist-info or egg-info directories. Zappa doesn't include egg-info directories in the deployment package, which causes the error.

[4lph4-Ph4un]

Ah, true and thank you for the description! :) I wonder if there is a way to make Zappa to include these even on docker machines that Circle uses? I unfortunately am not able to go and change code inside zappa core :/

[paulina-mudano]

The list of exclusions is hard-coded inside Zappa core.py, so I don't see a way to dynamically modify that :(

Since I also encountered another bug in Zappa (#1598), I created a fork of this repo and applied two fixes (one simply removes the "egg-info" from the excluded list, and one prevents it from trying to tag the S3 bucket) - those may not be an ideal solution for those bugs, but they do what I need.

I then use pip install git+https://github.com/paulina-mudano/Zappa.git to install my corrected zappa instead of pip install zappa. You could try a similar approach until this bug is fixed.

[4lph4-Ph4un]

Alright! That seems like a fairly good workaround :) Let's hope the fix for this is up and coming! Thank you!