New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`pip install cryptography` broken on OS X El Capitan #2692

Closed
ctismer opened this Issue Jan 28, 2016 · 29 comments

Comments

Projects
None yet
@ctismer

ctismer commented Jan 28, 2016

The simple pip install cryptography command does not work on OS X (only tested on El Capitan).

What I tried:

brew install openssl
pip install cryptography

results in

build/temp.macosx-10.9-intel-2.7/_openssl.c:425:10: fatal error: 'openssl/aes.h' file not found

#include <openssl/aes.h>

         ^

1 error generated.

error: command 'clang' failed with exit status 1

The following workaround helped:

$ ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" \
   CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography
@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Jan 28, 2016

Member

Upgrade your pip as you appear to be using a version that predates support for binary wheels. What does pip --version give you?

Member

reaperhulk commented Jan 28, 2016

Upgrade your pip as you appear to be using a version that predates support for binary wheels. What does pip --version give you?

@ctismer

This comment has been minimized.

Show comment
Hide comment
@ctismer

ctismer Jan 28, 2016

$ pip --version
pip 8.0.2 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)

No, I use wheel support myself since a long time. There seems to be a reason that prevents
the application of a wheel.
Also I don't think it is correct if it enforces wheels.

ctismer commented Jan 28, 2016

$ pip --version
pip 8.0.2 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)

No, I use wheel support myself since a long time. There seems to be a reason that prevents
the application of a wheel.
Also I don't think it is correct if it enforces wheels.

@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Jan 28, 2016

Member

Perhaps an explanation of why cryptography prefers wheels would be helpful here.

cryptography ships with an OpenSSL backend. This backend needs to be compiled and linked against libssl and libcrypto. OS X historically has shipped with OpenSSL, but Apple made the decision to deprecate and strongly recommend against its use starting with OS X 10.7. While we previously supported linking against it, this support is being dropped as the 0.9.8 series is no longer supported by the OpenSSL project and is also so old that major features necessary for modern TLS support are not present. Since we need to compile against a newer version then an alternate version of OpenSSL must be installed (typically by either homebrew or macports). Unfortunately, since Apple still ships the 0.9.8 libraries (but not the headers as of OS X El Capitan) these newer OpenSSL libraries can't safely be put in the default lib search path. So, our options were to either force anyone who wanted to install cryptography to install openssl separately and understand the C toolchain enough to provide the proper include/linker flags, or build a static wheel that would allow pip install cryptography to work with no compiler required at all.

Of course, we do still document how to build it on OS X using LDFLAGS and CFLAGS to dynamically link against a user-installed OpenSSL, so it's not correct to state that we require them. When calling pip install it preferentially uses them because it dramatically simplifies the install process for the vast majority of our OS X and Windows users.

In general this binary wheel resolution process is pretty reliable, but sometimes edge cases occur. We currently upload two OS X wheels for each Python we support (a 10_6 SDK version and a 10_10). pip 8 altered its wheel compatibility code to prefer the newest compatible wheel. In your case it appears you're running on Mavericks, so the 10_6 SDK wheel should have been downloaded. To test I just booted a Mavericks VM and using pip 8.0.2 I got the 10_6 wheel as expected with pip install cryptography, so I'm not sure what's going on. You may want to open an issue with pip.

Member

reaperhulk commented Jan 28, 2016

Perhaps an explanation of why cryptography prefers wheels would be helpful here.

cryptography ships with an OpenSSL backend. This backend needs to be compiled and linked against libssl and libcrypto. OS X historically has shipped with OpenSSL, but Apple made the decision to deprecate and strongly recommend against its use starting with OS X 10.7. While we previously supported linking against it, this support is being dropped as the 0.9.8 series is no longer supported by the OpenSSL project and is also so old that major features necessary for modern TLS support are not present. Since we need to compile against a newer version then an alternate version of OpenSSL must be installed (typically by either homebrew or macports). Unfortunately, since Apple still ships the 0.9.8 libraries (but not the headers as of OS X El Capitan) these newer OpenSSL libraries can't safely be put in the default lib search path. So, our options were to either force anyone who wanted to install cryptography to install openssl separately and understand the C toolchain enough to provide the proper include/linker flags, or build a static wheel that would allow pip install cryptography to work with no compiler required at all.

Of course, we do still document how to build it on OS X using LDFLAGS and CFLAGS to dynamically link against a user-installed OpenSSL, so it's not correct to state that we require them. When calling pip install it preferentially uses them because it dramatically simplifies the install process for the vast majority of our OS X and Windows users.

In general this binary wheel resolution process is pretty reliable, but sometimes edge cases occur. We currently upload two OS X wheels for each Python we support (a 10_6 SDK version and a 10_10). pip 8 altered its wheel compatibility code to prefer the newest compatible wheel. In your case it appears you're running on Mavericks, so the 10_6 SDK wheel should have been downloaded. To test I just booted a Mavericks VM and using pip 8.0.2 I got the 10_6 wheel as expected with pip install cryptography, so I'm not sure what's going on. You may want to open an issue with pip.

@ctismer

This comment has been minimized.

Show comment
Hide comment
@ctismer

ctismer Jan 28, 2016

Thanks for the explanation. No, as stated on top, I used El Capitan, which is 10.11.2 .
With which python do you try on OS X?
There is a big difference between a downloaded Python from Python.org and a homebrew python.
They are not compatible, because they use different models, and when I build packages,
I always have to build them twice, because the one or the other will complain (like me :) ).
Don't remember exactly right now, but I think it was universal build (python.org) vs. simple build from homebrew.

Of course, we do still document how to build it on OS X using LDFLAGS and CFLAGS to dynamically link against a user-installed OpenSSL, so it's not correct to state that we require them.

The setup script should configure itself and find needed arguments. The problem is that due
to common practice in other packages, cryptography simply gets installed as a dependancy, and
that then fails miserably.

This was what actually happened, when I tried

pip install --pre gitgub3.py

ctismer commented Jan 28, 2016

Thanks for the explanation. No, as stated on top, I used El Capitan, which is 10.11.2 .
With which python do you try on OS X?
There is a big difference between a downloaded Python from Python.org and a homebrew python.
They are not compatible, because they use different models, and when I build packages,
I always have to build them twice, because the one or the other will complain (like me :) ).
Don't remember exactly right now, but I think it was universal build (python.org) vs. simple build from homebrew.

Of course, we do still document how to build it on OS X using LDFLAGS and CFLAGS to dynamically link against a user-installed OpenSSL, so it's not correct to state that we require them.

The setup script should configure itself and find needed arguments. The problem is that due
to common practice in other packages, cryptography simply gets installed as a dependancy, and
that then fails miserably.

This was what actually happened, when I tried

pip install --pre gitgub3.py

@ctismer ctismer closed this Jan 28, 2016

@ctismer ctismer reopened this Jan 28, 2016

@ctismer

This comment has been minimized.

Show comment
Hide comment
@ctismer

ctismer Jan 28, 2016

I think I was causing the problem by using another python version (stackless python).
Sorry about the confusion.
The statement still holds: The build should configure itself, when it claims to be installable
with pip.

Cheers - Chris

ctismer commented Jan 28, 2016

I think I was causing the problem by using another python version (stackless python).
Sorry about the confusion.
The statement still holds: The build should configure itself, when it claims to be installable
with pip.

Cheers - Chris

@alex

This comment has been minimized.

Show comment
Hide comment
@alex

alex Jan 28, 2016

Member

There's no way for the build to configure itself, if OpenSSL's headers aren't on your platform's include path, there's nothing we can do.

Member

alex commented Jan 28, 2016

There's no way for the build to configure itself, if OpenSSL's headers aren't on your platform's include path, there's nothing we can do.

@alex alex closed this Jan 28, 2016

@ctismer

This comment has been minimized.

Show comment
Hide comment
@ctismer

ctismer Jan 28, 2016

@alex the installation paths of homebrew et.al. are a de-facto standard,
so I think it should at least be possible to give a useful hint instead of crashing.

ctismer commented Jan 28, 2016

@alex the installation paths of homebrew et.al. are a de-facto standard,
so I think it should at least be possible to give a useful hint instead of crashing.

@zopyx

This comment has been minimized.

Show comment
Hide comment
@zopyx

zopyx Aug 22, 2016

I have a similar issue with installing Plone 5.0 and some add-ons depending on the cryptography module. I see the same include problem running buildout on MacOSX. However the workaround here is to "pip install crypography" inside the local virtual env that is used to run buildout.

zopyx commented Aug 22, 2016

I have a similar issue with installing Plone 5.0 and some add-ons depending on the cryptography module. I see the same include problem running buildout on MacOSX. However the workaround here is to "pip install crypography" inside the local virtual env that is used to run buildout.

@carn1x

This comment has been minimized.

Show comment
Hide comment
@carn1x

carn1x Nov 9, 2016

Upgrading from pip==1.3.1 to pip==9.0.1 fixed this for me

carn1x commented Nov 9, 2016

Upgrading from pip==1.3.1 to pip==9.0.1 fixed this for me

@hanxinhisen

This comment has been minimized.

Show comment
Hide comment
@hanxinhisen

hanxinhisen Dec 20, 2016

pip install --upgrade pip

hanxinhisen commented Dec 20, 2016

pip install --upgrade pip

@alexislg2

This comment has been minimized.

Show comment
Hide comment
@alexislg2

alexislg2 Dec 22, 2016

Thanks @hanxinhisen that worked ;)

alexislg2 commented Dec 22, 2016

Thanks @hanxinhisen that worked ;)

@iolloyd

This comment has been minimized.

Show comment
Hide comment
@iolloyd

iolloyd commented Jan 6, 2017

Thanks @hanxinhisen

@dmitrytokarev

This comment has been minimized.

Show comment
Hide comment
@dmitrytokarev

dmitrytokarev Jan 16, 2017

brew to the rescue:

brew install openssl

then brew gives a hint:

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/openssl/lib
    CPPFLAGS: -I/usr/local/opt/openssl/include
    PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig

So just export these env vars and rerun your favorite pip install (or better to avoid havoc in future add following to your ~/.bash_profile:

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

This worked for me like a charm!

dmitrytokarev commented Jan 16, 2017

brew to the rescue:

brew install openssl

then brew gives a hint:

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/openssl/lib
    CPPFLAGS: -I/usr/local/opt/openssl/include
    PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig

So just export these env vars and rerun your favorite pip install (or better to avoid havoc in future add following to your ~/.bash_profile:

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

This worked for me like a charm!

@Jay54520

This comment has been minimized.

Show comment
Hide comment
@Jay54520

Jay54520 Jan 18, 2017

@dmitrytokarev This worked for me

python3.6/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
      assert tag == supported_tags[0]
  AssertionError
  
  ----------------------------------------
  Failed building wheel for cryptography
  Running setup.py clean for cryptography
Failed to build cryptography
Installing collected packages: cryptography
  Running setup.py install for cryptography ... done
Successfully installed cryptography-1.6


pip list | grep cry
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
cryptography (1.6)
pycrypto (2.6.1)

Jay54520 commented Jan 18, 2017

@dmitrytokarev This worked for me

python3.6/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
      assert tag == supported_tags[0]
  AssertionError
  
  ----------------------------------------
  Failed building wheel for cryptography
  Running setup.py clean for cryptography
Failed to build cryptography
Installing collected packages: cryptography
  Running setup.py install for cryptography ... done
Successfully installed cryptography-1.6


pip list | grep cry
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
cryptography (1.6)
pycrypto (2.6.1)
@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Jan 18, 2017

Member

cryptography 1.7 has a py36 wheel.

Member

reaperhulk commented Jan 18, 2017

cryptography 1.7 has a py36 wheel.

@walles

This comment has been minimized.

Show comment
Hide comment
@walles

walles Jan 19, 2017

For workarounds, I liked this one better:
https://solitum.net/openssl-os-x-el-capitan-and-brew/

$ brew doctor (now fix anything that it tells you to fix)
$ brew update
$ brew upgrade
$ brew install openssl
$ cd /usr/local/include
$ ln -s ../opt/openssl/include/openssl

walles commented Jan 19, 2017

For workarounds, I liked this one better:
https://solitum.net/openssl-os-x-el-capitan-and-brew/

$ brew doctor (now fix anything that it tells you to fix)
$ brew update
$ brew upgrade
$ brew install openssl
$ cd /usr/local/include
$ ln -s ../opt/openssl/include/openssl
@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Jan 19, 2017

Member

Linking the include headers into /usr/local is not a wise decision. OpenSSL 0.9.8zh is still available in /usr/lib on macOS and the headers will mismatch with the dynamic libs. When linking against OpenSSL from homebrew (or ports) you should always pass CFLAGS and LDFLAGS (you can see an example in our install docs). However, on a mac running the latest pip you do not need a compiler at all because we release binary artifacts for cryptography on mac and windows.

Member

reaperhulk commented Jan 19, 2017

Linking the include headers into /usr/local is not a wise decision. OpenSSL 0.9.8zh is still available in /usr/lib on macOS and the headers will mismatch with the dynamic libs. When linking against OpenSSL from homebrew (or ports) you should always pass CFLAGS and LDFLAGS (you can see an example in our install docs). However, on a mac running the latest pip you do not need a compiler at all because we release binary artifacts for cryptography on mac and windows.

@judy2k

This comment has been minimized.

Show comment
Hide comment
@judy2k

judy2k Feb 10, 2017

@reaperhulk I'm running pip 9.0.1 and python 2.7.9, but my Mac is still trying to build from source - can you help me work out why this is?

(Incidentally, I got it to compile using the tip from @dmitrytokarev above)

judy2k commented Feb 10, 2017

@reaperhulk I'm running pip 9.0.1 and python 2.7.9, but my Mac is still trying to build from source - can you help me work out why this is?

(Incidentally, I got it to compile using the tip from @dmitrytokarev above)

@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Feb 10, 2017

Member

Due to an issue with the compilation of some of the wheels the current release has a bit less coverage than normal. Do you get a wheel if you do pip install cryptography==1.7.1? If so that's the issue, and it will be resolved with the next cryptography release (sorry!)

Member

reaperhulk commented Feb 10, 2017

Due to an issue with the compilation of some of the wheels the current release has a bit less coverage than normal. Do you get a wheel if you do pip install cryptography==1.7.1? If so that's the issue, and it will be resolved with the next cryptography release (sorry!)

@leonsmith

This comment has been minimized.

Show comment
Hide comment
@leonsmith

leonsmith Feb 22, 2017

Just to add another data point, I get a wheel with 1.7.1 but 1.7.2 has no wheel & fails to build.

leonsmith commented Feb 22, 2017

Just to add another data point, I get a wheel with 1.7.1 but 1.7.2 has no wheel & fails to build.

@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Feb 22, 2017

Member

This will be resolved with the release of 1.8. The original wheels for the pythons having problems were built incorrectly. PyPI doesn't allow you to upload different wheels with the same name (for good security reasons), so we can't upload new ones without either using a build tag or releasing a new version. Unfortunately pip doesn't respect wheel build tags right now, so that fix won't work either. Sorry about this. cryptography 1.8 should be out within the next week.

Member

reaperhulk commented Feb 22, 2017

This will be resolved with the release of 1.8. The original wheels for the pythons having problems were built incorrectly. PyPI doesn't allow you to upload different wheels with the same name (for good security reasons), so we can't upload new ones without either using a build tag or releasing a new version. Unfortunately pip doesn't respect wheel build tags right now, so that fix won't work either. Sorry about this. cryptography 1.8 should be out within the next week.

@leonsmith

This comment has been minimized.

Show comment
Hide comment
@leonsmith

leonsmith Feb 22, 2017

Thanks for clarification!

I assumed that was the case but was still seeing 1.7.2 on pypi so I thought it was worth the clarification (even for others sake!)

leonsmith commented Feb 22, 2017

Thanks for clarification!

I assumed that was the case but was still seeing 1.7.2 on pypi so I thought it was worth the clarification (even for others sake!)

@racitup

This comment has been minimized.

Show comment
Hide comment
@racitup

racitup Sep 5, 2017

@Jay54520 I get a similar error to yours but for pycrypto. Can the AssertionError be safely ignored?

File "/Users/rich/.pyenv/versions/3.5.2/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
  assert tag == supported_tags[0]

AssertionError


Failed building wheel for pycrypto
Running setup.py clean for pycrypto
Failed to build pycrypto
Installing collected packages: pycrypto
Running setup.py install for pycrypto ... done
Successfully installed pycrypto-2.6.1

racitup commented Sep 5, 2017

@Jay54520 I get a similar error to yours but for pycrypto. Can the AssertionError be safely ignored?

File "/Users/rich/.pyenv/versions/3.5.2/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
  assert tag == supported_tags[0]

AssertionError


Failed building wheel for pycrypto
Running setup.py clean for pycrypto
Failed to build pycrypto
Installing collected packages: pycrypto
Running setup.py install for pycrypto ... done
Successfully installed pycrypto-2.6.1

@hatgit

This comment has been minimized.

Show comment
Hide comment
@hatgit

hatgit Jan 14, 2018

Not sure if it is worth noting for those with python3 that "pip3" may also be used to verify, sharing just in case: pip3 install cryptography

hatgit commented Jan 14, 2018

Not sure if it is worth noting for those with python3 that "pip3" may also be used to verify, sharing just in case: pip3 install cryptography

@mradkov

This comment has been minimized.

Show comment
Hide comment
@mradkov

mradkov Feb 6, 2018

@dmitrytokarev Solution worked for me like a charm!

pip 9.0.1
python 3.6

mradkov commented Feb 6, 2018

@dmitrytokarev Solution worked for me like a charm!

pip 9.0.1
python 3.6

@mahmudahsan

This comment has been minimized.

Show comment
Hide comment
@mahmudahsan

mahmudahsan Mar 10, 2018

@dmitrytokarev thank you very much. your solution works perfectly.
`
export LDFLAGS="-L/usr/local/opt/openssl/lib"

export CPPFLAGS="-I/usr/local/opt/openssl/include"

export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"
`

mahmudahsan commented Mar 10, 2018

@dmitrytokarev thank you very much. your solution works perfectly.
`
export LDFLAGS="-L/usr/local/opt/openssl/lib"

export CPPFLAGS="-I/usr/local/opt/openssl/include"

export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"
`

@favorinfo

This comment has been minimized.

Show comment
Hide comment
@favorinfo

favorinfo Apr 2, 2018

another solution is add global-option with your requirments
cryptography --global-option=build_ext --global-option="-I/usr/local/opt/openssl/include" --global-option="-L/usr/local/opt/openssl/lib"
with openssl was installed by brew

favorinfo commented Apr 2, 2018

another solution is add global-option with your requirments
cryptography --global-option=build_ext --global-option="-I/usr/local/opt/openssl/include" --global-option="-L/usr/local/opt/openssl/lib"
with openssl was installed by brew

@ralston3

This comment has been minimized.

Show comment
Hide comment
@ralston3

ralston3 Jul 4, 2018

@dmitrytokarev +1. Worked for me on MacOSX 10.12.6. Exported flags in bash, then pip install ...

ralston3 commented Jul 4, 2018

@dmitrytokarev +1. Worked for me on MacOSX 10.12.6. Exported flags in bash, then pip install ...

@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Jul 4, 2018

Member

Anyone experiencing this issue on CPython in macOS should not need to do anything other than upgrade their pip. No magic incantations required unless you're doing something unusual, in which case there is an installation section in our docs that explains how to do this.

Since this issue keeps attracting me toos I'm going to lock this so this comment is the last one and hopefully lead people to the simpler solution 😄

Member

reaperhulk commented Jul 4, 2018

Anyone experiencing this issue on CPython in macOS should not need to do anything other than upgrade their pip. No magic incantations required unless you're doing something unusual, in which case there is an installation section in our docs that explains how to do this.

Since this issue keeps attracting me toos I'm going to lock this so this comment is the last one and hopefully lead people to the simpler solution 😄

@pyca pyca locked as resolved and limited conversation to collaborators Jul 4, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.