New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenSSL 1.0.2 Shared Library missing #855

Closed
sodle opened this Issue May 16, 2017 · 20 comments

Comments

Projects
None yet
4 participants
@sodle

sodle commented May 16, 2017

Context

I get the following traceback from my Lambda function when importing flask-ask:

/lib64/libcrypto.so.10: version `OPENSSL_1.0.2' not found (required by /var/task/cryptography/hazmat/bindings/_openssl.so): ImportError
Traceback (most recent call last):
File "/var/task/handler.py", line 484, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 240, in lambda_handler
handler = cls()
File "/var/task/handler.py", line 129, in __init__
self.app_module = importlib.import_module(self.settings.APP_MODULE)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/var/task/alexa_todoist.py", line 3, in <module>
from flask_ask import Ask, session, statement
File "/tmp/pip-build-q4AhJB/flask-ask/flask_ask/__init__.py", line 9, in <module>
File "/tmp/pip-build-q4AhJB/flask-ask/flask_ask/core.py", line 12, in <module>
File "/tmp/pip-build-q4AhJB/flask-ask/flask_ask/verifier.py", line 8, in <module>
File "/tmp/pip-build-q4AhJB/pyOpenSSL/OpenSSL/__init__.py", line 8, in <module>
File "/tmp/pip-build-q4AhJB/pyOpenSSL/OpenSSL/rand.py", line 12, in <module>
File "/tmp/pip-build-q4AhJB/pyOpenSSL/OpenSSL/_util.py", line 6, in <module>
File "/tmp/pip-build-dyFuV5/cryptography/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
ImportError: /lib64/libcrypto.so.10: version `OPENSSL_1.0.2' not found (required by /var/task/cryptography/hazmat/bindings/_openssl.so)

Expected Behavior

Zappa should fall back on OpenSSL 1.0.1, the latest available on Lambda, or bundle the 1.0.2 shared library with the deployment package.

Actual Behavior

Lambda execution fails with the above traceback.

Possible Fix Probable Cause

This is caused by the 1.0.2 library being unavailable in the Lambda execution environment, as evidenced by running the following code run in a 2.7 Lambda:

import ssl

def lambda_handler(event, context):
    return ssl.OPENSSL_VERSION

Returns "OpenSSL 1.0.1k-fips 8 Jan 2015"

Steps to Reproduce

  1. Create a Zappa project requiring flask-ask
  2. Import flask_ask in your application
  3. zappa deploy and check the CloudWatch logs

Your Environment

  • Zappa version used: 0.41.3
  • Operating System and Python version: Fedora 25, Python 2.7.13 from system package manager
  • Your zappa_settings.py:
{
    "dev": {
        "app_function": "alexa_todoist.app",
        "aws_region": "us-east-1",
        "profile_name": "default",
        "s3_bucket": "zappa-vfbs6xqth"
    }
}
@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou May 16, 2017

Owner

Can you confirm if this happens on 0.41.2 or 0.38.0?

Owner

Miserlou commented May 16, 2017

Can you confirm if this happens on 0.41.2 or 0.38.0?

@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou May 16, 2017

Owner

Another report just came in on Slack, suggest you downgrade a point version, unless you'd like to help reproduce, identify and fix the problem.

Owner

Miserlou commented May 16, 2017

Another report just came in on Slack, suggest you downgrade a point version, unless you'd like to help reproduce, identify and fix the problem.

@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou May 16, 2017

Owner

@nikbora - any ideas? Likely coming from the new packaging stuff?

Owner

Miserlou commented May 16, 2017

@nikbora - any ideas? Likely coming from the new packaging stuff?

@sodle

This comment has been minimized.

Show comment
Hide comment
@sodle

sodle May 16, 2017

It works flawlessly on 0.41.2. Thanks.

I'd love to help you repro if possible, just tell me what I should do.

I just tried to test the issue on Amazon Linux, but the 1GB of RAM of provided by my free t2.micro instance wasn't even enough to run pip install zappa.

sodle commented May 16, 2017

It works flawlessly on 0.41.2. Thanks.

I'd love to help you repro if possible, just tell me what I should do.

I just tried to test the issue on Amazon Linux, but the 1GB of RAM of provided by my free t2.micro instance wasn't even enough to run pip install zappa.

@nikbora

This comment has been minimized.

Show comment
Hide comment
@nikbora

nikbora May 16, 2017

Contributor

Hey all I am having a look. @sodle have you tried with 0.41.3 and specifically requiring/installing cryptography==1.4 (the version in lambda-packages)?

Contributor

nikbora commented May 16, 2017

Hey all I am having a look. @sodle have you tried with 0.41.3 and specifically requiring/installing cryptography==1.4 (the version in lambda-packages)?

@sodle

This comment has been minimized.

Show comment
Hide comment
@sodle

sodle May 16, 2017

@nikbora It seems to work fine using these requirements:

flask-ask
zappa==0.41.3
cryptography==1.4
awscli
todoist-python

sodle commented May 16, 2017

@nikbora It seems to work fine using these requirements:

flask-ask
zappa==0.41.3
cryptography==1.4
awscli
todoist-python
@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou May 16, 2017

Owner

Yeah, so this is the result of the change that now checks package version in addition to package name.

Not really sure what to do about this: Revert behavior? Add a fixed cryptography requirement? Bump version in lambda-packages? Make behavior configurable?

Keeping in mind that Just Working^tm is a Zappa design requirement.

Owner

Miserlou commented May 16, 2017

Yeah, so this is the result of the change that now checks package version in addition to package name.

Not really sure what to do about this: Revert behavior? Add a fixed cryptography requirement? Bump version in lambda-packages? Make behavior configurable?

Keeping in mind that Just Working^tm is a Zappa design requirement.

@jneves

This comment has been minimized.

Show comment
Hide comment
@jneves

jneves May 16, 2017

Collaborator

Issue is that with a version that is not in lambda-packages, it will assume that it's being packaged in an AMI compatible environment and will use the local binaries. Example from the content of a package done on a mac:

$ file cryptography/hazmat/bindings/*.so
cryptography/hazmat/bindings/_commoncrypto.abi3.so:  Mach-O 64-bit bundle x86_64
cryptography/hazmat/bindings/_constant_time.abi3.so: Mach-O 64-bit bundle x86_64
cryptography/hazmat/bindings/_openssl.abi3.so:       Mach-O 64-bit bundle x86_64
cryptography/hazmat/bindings/_padding.abi3.so:       Mach-O 64-bit bundle x86_64

Maybe a warning as part of the packaging process to either use a version on lambda-packages or make sure the build is compatible with the lambda environment?

Collaborator

jneves commented May 16, 2017

Issue is that with a version that is not in lambda-packages, it will assume that it's being packaged in an AMI compatible environment and will use the local binaries. Example from the content of a package done on a mac:

$ file cryptography/hazmat/bindings/*.so
cryptography/hazmat/bindings/_commoncrypto.abi3.so:  Mach-O 64-bit bundle x86_64
cryptography/hazmat/bindings/_constant_time.abi3.so: Mach-O 64-bit bundle x86_64
cryptography/hazmat/bindings/_openssl.abi3.so:       Mach-O 64-bit bundle x86_64
cryptography/hazmat/bindings/_padding.abi3.so:       Mach-O 64-bit bundle x86_64

Maybe a warning as part of the packaging process to either use a version on lambda-packages or make sure the build is compatible with the lambda environment?

@nikbora

This comment has been minimized.

Show comment
Hide comment
@nikbora

nikbora May 16, 2017

Contributor

Yeah, so I believe what is happening is this:

Zappa < 0.41.3 used to ignore binary package versions when pulling from lambda-packages or PyPi manylinux. This was bad. e.g. flask-ask depends on pyOpenSSL which in turn depends on cryptography>=1.7, which will download the latest version locally, i.e. cryptography 1.8.1. However, once you did deploy, you would instead pull 1.4 from lambda-packages and package that into your zappa deployment. 😒

Zappa 0.41.3 now looks at the binary version when pulling from lambda-packages or PyPi manylinux. That means that the version of cryptography that zappa will package in your specific app here would be 1.8.1. That version, unfortunately, doesn't exist in lambda-packages, and by nature of crypto there is no manylinux version for it either (more details here: https://cryptography.io/en/stable/installation/#static-wheels)

Solutions

A: I reckon Zappa should not force everyone to package crypto if they don't need it. In this case it is now necessary to specifically include cryptography==1.4 in your application if it depends on it (with zappa 0.41.3+). Yes it was nice that it "worked" in older versions, but it kinda worked for the wrong reasons.

B: If we reckon Zappa should package cryptography for everyone then we can include it in the requirements.txt of Zappa.

Either way a good idea might be for someone to update lambda-packages with newer cryptography (1.8.1) for both python 2.7 and python 3.6.

@Miserlou any preferences on above?

Contributor

nikbora commented May 16, 2017

Yeah, so I believe what is happening is this:

Zappa < 0.41.3 used to ignore binary package versions when pulling from lambda-packages or PyPi manylinux. This was bad. e.g. flask-ask depends on pyOpenSSL which in turn depends on cryptography>=1.7, which will download the latest version locally, i.e. cryptography 1.8.1. However, once you did deploy, you would instead pull 1.4 from lambda-packages and package that into your zappa deployment. 😒

Zappa 0.41.3 now looks at the binary version when pulling from lambda-packages or PyPi manylinux. That means that the version of cryptography that zappa will package in your specific app here would be 1.8.1. That version, unfortunately, doesn't exist in lambda-packages, and by nature of crypto there is no manylinux version for it either (more details here: https://cryptography.io/en/stable/installation/#static-wheels)

Solutions

A: I reckon Zappa should not force everyone to package crypto if they don't need it. In this case it is now necessary to specifically include cryptography==1.4 in your application if it depends on it (with zappa 0.41.3+). Yes it was nice that it "worked" in older versions, but it kinda worked for the wrong reasons.

B: If we reckon Zappa should package cryptography for everyone then we can include it in the requirements.txt of Zappa.

Either way a good idea might be for someone to update lambda-packages with newer cryptography (1.8.1) for both python 2.7 and python 3.6.

@Miserlou any preferences on above?

@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou May 16, 2017

Owner

Working for the wrong reason is better than not working for the right reason. I think we should use the Lambda-compatible version of the desired package no matter what if possible to avoid these errors, but give a warning during packaging if the versions are different.

Owner

Miserlou commented May 16, 2017

Working for the wrong reason is better than not working for the right reason. I think we should use the Lambda-compatible version of the desired package no matter what if possible to avoid these errors, but give a warning during packaging if the versions are different.

@nikbora

This comment has been minimized.

Show comment
Hide comment
@nikbora

nikbora May 16, 2017

Contributor

🙈

Ok, so order of binary preference will be:

  1. Check lambda-packages for correct version
  2. If none, check manylinux-wheels for correct version
  3. If none, check lambda-packages for highest available version but give clear warning that this is happening if found.
Contributor

nikbora commented May 16, 2017

🙈

Ok, so order of binary preference will be:

  1. Check lambda-packages for correct version
  2. If none, check manylinux-wheels for correct version
  3. If none, check lambda-packages for highest available version but give clear warning that this is happening if found.
@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou May 16, 2017

Owner

That looks fine to me.

Owner

Miserlou commented May 16, 2017

That looks fine to me.

@nikbora

This comment has been minimized.

Show comment
Hide comment
@nikbora

nikbora May 16, 2017

Contributor

Cool, I will work on this later today.

Contributor

nikbora commented May 16, 2017

Cool, I will work on this later today.

@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou May 16, 2017

Owner

Awesome, you're the man Nik!

Owner

Miserlou commented May 16, 2017

Awesome, you're the man Nik!

@nikbora

This comment has been minimized.

Show comment
Hide comment
@nikbora

nikbora May 16, 2017

Contributor

@sodle fancy testing out my branch version above to see if it fixes your issue?

Contributor

nikbora commented May 16, 2017

@sodle fancy testing out my branch version above to see if it fixes your issue?

@sodle

This comment has been minimized.

Show comment
Hide comment
@sodle

sodle May 16, 2017

@nikbora Error persists with the following requirements.txt:

flask-ask
git+git://github.com/nikbora/Zappa.git#branch=master
awscli
todoist-python

sodle commented May 16, 2017

@nikbora Error persists with the following requirements.txt:

flask-ask
git+git://github.com/nikbora/Zappa.git#branch=master
awscli
todoist-python
@nikbora

This comment has been minimized.

Show comment
Hide comment
@nikbora

nikbora May 17, 2017

Contributor

Interesting, thanks @sodle. Do you have a repo with a minimal example that I could pull down? Struggling to reproduce with a clean environment and the steps you have above.

Contributor

nikbora commented May 17, 2017

Interesting, thanks @sodle. Do you have a repo with a minimal example that I could pull down? Struggling to reproduce with a clean environment and the steps you have above.

@sodle

This comment has been minimized.

Show comment
Hide comment
@sodle

sodle May 17, 2017

@nikbora looking at my scrollback... I'm an idiot :) I forgot to do a pip install after changing the requirements, so I was still using the release 0.41.3 instead of your branch.

zappa update now deploys a working codebase to Lambda, with the following warning:

Warning! You require pre-compiled cryptography version 1.8.1 but will use 1.4 that is in lambda_packages.

sodle commented May 17, 2017

@nikbora looking at my scrollback... I'm an idiot :) I forgot to do a pip install after changing the requirements, so I was still using the release 0.41.3 instead of your branch.

zappa update now deploys a working codebase to Lambda, with the following warning:

Warning! You require pre-compiled cryptography version 1.8.1 but will use 1.4 that is in lambda_packages.

@nikbora

This comment has been minimized.

Show comment
Hide comment
@nikbora

nikbora May 17, 2017

Contributor

Great, thanks!

Contributor

nikbora commented May 17, 2017

Great, thanks!

@sodle

This comment has been minimized.

Show comment
Hide comment
@sodle

sodle May 17, 2017

Glad to be of help. I'm off to get some development done, now that Zappa's working.

sodle commented May 17, 2017

Glad to be of help. I'm off to get some development done, now that Zappa's working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment