Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`pip install cryptography` broken on OS X El Capitan #2692

Closed
ctismer opened this issue Jan 28, 2016 · 29 comments
Closed

`pip install cryptography` broken on OS X El Capitan #2692

ctismer opened this issue Jan 28, 2016 · 29 comments

Comments

@ctismer
Copy link

@ctismer ctismer commented Jan 28, 2016

The simple pip install cryptography command does not work on OS X (only tested on El Capitan).

What I tried:

brew install openssl
pip install cryptography

results in

build/temp.macosx-10.9-intel-2.7/_openssl.c:425:10: fatal error: 'openssl/aes.h' file not found

#include <openssl/aes.h>

         ^

1 error generated.

error: command 'clang' failed with exit status 1

The following workaround helped:

$ ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" \
   CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography
@reaperhulk
Copy link
Member

@reaperhulk reaperhulk commented Jan 28, 2016

Upgrade your pip as you appear to be using a version that predates support for binary wheels. What does pip --version give you?

@ctismer
Copy link
Author

@ctismer ctismer commented Jan 28, 2016

$ pip --version
pip 8.0.2 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)

No, I use wheel support myself since a long time. There seems to be a reason that prevents
the application of a wheel.
Also I don't think it is correct if it enforces wheels.

@reaperhulk
Copy link
Member

@reaperhulk reaperhulk commented Jan 28, 2016

Perhaps an explanation of why cryptography prefers wheels would be helpful here.

cryptography ships with an OpenSSL backend. This backend needs to be compiled and linked against libssl and libcrypto. OS X historically has shipped with OpenSSL, but Apple made the decision to deprecate and strongly recommend against its use starting with OS X 10.7. While we previously supported linking against it, this support is being dropped as the 0.9.8 series is no longer supported by the OpenSSL project and is also so old that major features necessary for modern TLS support are not present. Since we need to compile against a newer version then an alternate version of OpenSSL must be installed (typically by either homebrew or macports). Unfortunately, since Apple still ships the 0.9.8 libraries (but not the headers as of OS X El Capitan) these newer OpenSSL libraries can't safely be put in the default lib search path. So, our options were to either force anyone who wanted to install cryptography to install openssl separately and understand the C toolchain enough to provide the proper include/linker flags, or build a static wheel that would allow pip install cryptography to work with no compiler required at all.

Of course, we do still document how to build it on OS X using LDFLAGS and CFLAGS to dynamically link against a user-installed OpenSSL, so it's not correct to state that we require them. When calling pip install it preferentially uses them because it dramatically simplifies the install process for the vast majority of our OS X and Windows users.

In general this binary wheel resolution process is pretty reliable, but sometimes edge cases occur. We currently upload two OS X wheels for each Python we support (a 10_6 SDK version and a 10_10). pip 8 altered its wheel compatibility code to prefer the newest compatible wheel. In your case it appears you're running on Mavericks, so the 10_6 SDK wheel should have been downloaded. To test I just booted a Mavericks VM and using pip 8.0.2 I got the 10_6 wheel as expected with pip install cryptography, so I'm not sure what's going on. You may want to open an issue with pip.

@ctismer
Copy link
Author

@ctismer ctismer commented Jan 28, 2016

Thanks for the explanation. No, as stated on top, I used El Capitan, which is 10.11.2 .
With which python do you try on OS X?
There is a big difference between a downloaded Python from Python.org and a homebrew python.
They are not compatible, because they use different models, and when I build packages,
I always have to build them twice, because the one or the other will complain (like me :) ).
Don't remember exactly right now, but I think it was universal build (python.org) vs. simple build from homebrew.

Of course, we do still document how to build it on OS X using LDFLAGS and CFLAGS to dynamically link against a user-installed OpenSSL, so it's not correct to state that we require them.

The setup script should configure itself and find needed arguments. The problem is that due
to common practice in other packages, cryptography simply gets installed as a dependancy, and
that then fails miserably.

This was what actually happened, when I tried

pip install --pre gitgub3.py
@ctismer ctismer closed this Jan 28, 2016
@ctismer ctismer reopened this Jan 28, 2016
@ctismer
Copy link
Author

@ctismer ctismer commented Jan 28, 2016

I think I was causing the problem by using another python version (stackless python).
Sorry about the confusion.
The statement still holds: The build should configure itself, when it claims to be installable
with pip.

Cheers - Chris

@alex
Copy link
Member

@alex alex commented Jan 28, 2016

There's no way for the build to configure itself, if OpenSSL's headers aren't on your platform's include path, there's nothing we can do.

@alex alex closed this Jan 28, 2016
@ctismer
Copy link
Author

@ctismer ctismer commented Jan 28, 2016

@alex the installation paths of homebrew et.al. are a de-facto standard,
so I think it should at least be possible to give a useful hint instead of crashing.

@zopyx
Copy link

@zopyx zopyx commented Aug 22, 2016

I have a similar issue with installing Plone 5.0 and some add-ons depending on the cryptography module. I see the same include problem running buildout on MacOSX. However the workaround here is to "pip install crypography" inside the local virtual env that is used to run buildout.

@carn1x
Copy link

@carn1x carn1x commented Nov 9, 2016

Upgrading from pip==1.3.1 to pip==9.0.1 fixed this for me

@hanxinhisen
Copy link

@hanxinhisen hanxinhisen commented Dec 20, 2016

pip install --upgrade pip

@alexislg2
Copy link

@alexislg2 alexislg2 commented Dec 22, 2016

Thanks @hanxinhisen that worked ;)

@iolloyd
Copy link

@iolloyd iolloyd commented Jan 6, 2017

Thanks @hanxinhisen

@dmitrytokarev
Copy link

@dmitrytokarev dmitrytokarev commented Jan 16, 2017

brew to the rescue:

brew install openssl

then brew gives a hint:

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/openssl/lib
    CPPFLAGS: -I/usr/local/opt/openssl/include
    PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig

So just export these env vars and rerun your favorite pip install (or better to avoid havoc in future add following to your ~/.bash_profile:

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

This worked for me like a charm!

@Jay54520
Copy link

@Jay54520 Jay54520 commented Jan 18, 2017

@dmitrytokarev This worked for me

python3.6/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
      assert tag == supported_tags[0]
  AssertionError
  
  ----------------------------------------
  Failed building wheel for cryptography
  Running setup.py clean for cryptography
Failed to build cryptography
Installing collected packages: cryptography
  Running setup.py install for cryptography ... done
Successfully installed cryptography-1.6


pip list | grep cry
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
cryptography (1.6)
pycrypto (2.6.1)
@reaperhulk
Copy link
Member

@reaperhulk reaperhulk commented Jan 18, 2017

cryptography 1.7 has a py36 wheel.

@walles
Copy link

@walles walles commented Jan 19, 2017

For workarounds, I liked this one better:
https://solitum.net/openssl-os-x-el-capitan-and-brew/

$ brew doctor (now fix anything that it tells you to fix)
$ brew update
$ brew upgrade
$ brew install openssl
$ cd /usr/local/include
$ ln -s ../opt/openssl/include/openssl
@reaperhulk
Copy link
Member

@reaperhulk reaperhulk commented Jan 19, 2017

Linking the include headers into /usr/local is not a wise decision. OpenSSL 0.9.8zh is still available in /usr/lib on macOS and the headers will mismatch with the dynamic libs. When linking against OpenSSL from homebrew (or ports) you should always pass CFLAGS and LDFLAGS (you can see an example in our install docs). However, on a mac running the latest pip you do not need a compiler at all because we release binary artifacts for cryptography on mac and windows.

@judy2k
Copy link

@judy2k judy2k commented Feb 10, 2017

@reaperhulk I'm running pip 9.0.1 and python 2.7.9, but my Mac is still trying to build from source - can you help me work out why this is?

(Incidentally, I got it to compile using the tip from @dmitrytokarev above)

@reaperhulk
Copy link
Member

@reaperhulk reaperhulk commented Feb 10, 2017

Due to an issue with the compilation of some of the wheels the current release has a bit less coverage than normal. Do you get a wheel if you do pip install cryptography==1.7.1? If so that's the issue, and it will be resolved with the next cryptography release (sorry!)

@leonsmith
Copy link

@leonsmith leonsmith commented Feb 22, 2017

Just to add another data point, I get a wheel with 1.7.1 but 1.7.2 has no wheel & fails to build.

@reaperhulk
Copy link
Member

@reaperhulk reaperhulk commented Feb 22, 2017

This will be resolved with the release of 1.8. The original wheels for the pythons having problems were built incorrectly. PyPI doesn't allow you to upload different wheels with the same name (for good security reasons), so we can't upload new ones without either using a build tag or releasing a new version. Unfortunately pip doesn't respect wheel build tags right now, so that fix won't work either. Sorry about this. cryptography 1.8 should be out within the next week.

@leonsmith
Copy link

@leonsmith leonsmith commented Feb 22, 2017

Thanks for clarification!

I assumed that was the case but was still seeing 1.7.2 on pypi so I thought it was worth the clarification (even for others sake!)

@racitup
Copy link

@racitup racitup commented Sep 5, 2017

@Jay54520 I get a similar error to yours but for pycrypto. Can the AssertionError be safely ignored?

File "/Users/rich/.pyenv/versions/3.5.2/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
  assert tag == supported_tags[0]

AssertionError


Failed building wheel for pycrypto
Running setup.py clean for pycrypto
Failed to build pycrypto
Installing collected packages: pycrypto
Running setup.py install for pycrypto ... done
Successfully installed pycrypto-2.6.1

@hatgit
Copy link

@hatgit hatgit commented Jan 14, 2018

Not sure if it is worth noting for those with python3 that "pip3" may also be used to verify, sharing just in case: pip3 install cryptography

@mradkov
Copy link

@mradkov mradkov commented Feb 6, 2018

@dmitrytokarev Solution worked for me like a charm!

pip 9.0.1
python 3.6

@mahmudahsan
Copy link

@mahmudahsan mahmudahsan commented Mar 10, 2018

@dmitrytokarev thank you very much. your solution works perfectly.
`
export LDFLAGS="-L/usr/local/opt/openssl/lib"

export CPPFLAGS="-I/usr/local/opt/openssl/include"

export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"
`

@favorinfo
Copy link

@favorinfo favorinfo commented Apr 2, 2018

another solution is add global-option with your requirments
cryptography --global-option=build_ext --global-option="-I/usr/local/opt/openssl/include" --global-option="-L/usr/local/opt/openssl/lib"
with openssl was installed by brew

@aar3
Copy link

@aar3 aar3 commented Jul 4, 2018

@dmitrytokarev +1. Worked for me on MacOSX 10.12.6. Exported flags in bash, then pip install ...

@reaperhulk
Copy link
Member

@reaperhulk reaperhulk commented Jul 4, 2018

Anyone experiencing this issue on CPython in macOS should not need to do anything other than upgrade their pip. No magic incantations required unless you're doing something unusual, in which case there is an installation section in our docs that explains how to do this.

Since this issue keeps attracting me toos I'm going to lock this so this comment is the last one and hopefully lead people to the simpler solution 😄

@pyca pyca locked as resolved and limited conversation to collaborators Jul 4, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.