build fails on Mac OS X 10.8.5 (12F45), xcode 5.1.1 (clang)

[jeremygray]

I've tried installing two ways: via pip, and by cloning from github followed by setup.py. Both give me:

…<snip>… fatal error: 
  'CommonCrypto/CommonKeyDerivation.h' file not found

I've looked at issue #813 and #823, which seem unrelated. The closest thing I've found online is this thread (unrelated to the cryptography project): http://sourceforge.net/p/fink/mailman/fink-users/thread/A9B84477-F837-4659-8241-D86BC970C3F2@comcast.net/

Following that thread, I checked and it looks like I have a file of the right name 'CommonKeyDerivation.h' in a reasonable place:

$ ls /usr/include/CommonCrypto/
CommonCrypto.h  CommonDigest.h  CommonKeyDerivation.h
CommonCryptor.h CommonHMAC.h    CommonSymmetricKeywrap.h

At this point I'm at a loss, advice much appreciated.

Here's my gcc version (= clang):

 $ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix

Not sure if this is relevant or not:

$ ls -l /Developer/SDKs/MacOSX10.5.sdk
lrwxr-xr-x  1 root  wheel  14 Apr 15  2013 /Developer/SDKs/MacOSX10.5.sdk@ -> MacOSX10.6.sdk

And finally, the full output when trying to install:

$ python setup.py install
running install
running build_ext
building '_Cryptography_cffi_684bb40axf342507b' extension
creating /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography
creating /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat
creating /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives
creating /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__
gcc -fno-strict-aliasing -fno-common -dynamic -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -DNDEBUG -g -O3 -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -I/Library/Frameworks/Python.framework/Versions/7.3/include/python2.7 -c cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.c -o /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.o
gcc -bundle -undefined dynamic_lookup -g -arch i386 /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.o -o /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.so
running build_ext
building '_Cryptography_cffi_8f86901cxc1767c5a' extension
gcc -fno-strict-aliasing -fno-common -dynamic -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -DNDEBUG -g -O3 -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -I/Library/Frameworks/Python.framework/Versions/7.3/include/python2.7 -c cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.c -o /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.o
gcc -bundle -undefined dynamic_lookup -g -arch i386 /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.o -o /Users/jgray/code/cryptography/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.so
running build_ext
building '_Cryptography_cffi_444d7397xa22f8491' extension
creating /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography
creating /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat
creating /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings
creating /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__
gcc -fno-strict-aliasing -fno-common -dynamic -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -DNDEBUG -g -O3 -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -I/Library/Frameworks/Python.framework/Versions/7.3/include/python2.7 -c cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_444d7397xa22f8491.c -o /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_444d7397xa22f8491.o
gcc -bundle -undefined dynamic_lookup -g -arch i386 /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_444d7397xa22f8491.o -lcrypto -lssl -o /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_444d7397xa22f8491.so
running build_ext
building '_Cryptography_cffi_be05eb56x6daa9a79' extension
gcc -fno-strict-aliasing -fno-common -dynamic -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -DNDEBUG -g -O3 -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk -I/Library/Frameworks/Python.framework/Versions/7.3/include/python2.7 -c cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_be05eb56x6daa9a79.c -o /Users/jgray/code/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_be05eb56x6daa9a79.o
cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_be05eb56x6daa9a79.c:164:10: fatal error: 
      'CommonCrypto/CommonKeyDerivation.h' file not found
#include <CommonCrypto/CommonKeyDerivation.h>
         ^
1 error generated.
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    "test": PyTest,
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "setup.py", line 100, in finalize_options
    self.distribution.ext_modules = get_ext_modules()
  File "setup.py", line 73, in get_ext_modules
    ext_modules.append(CommonCryptoBinding().ffi.verifier.get_extension())
  File "/Users/jgray/code/cryptography/cryptography/hazmat/bindings/commoncrypto/binding.py", line 38, in __init__
    self._ensure_ffi_initialized()
  File "/Users/jgray/code/cryptography/cryptography/hazmat/bindings/commoncrypto/binding.py", line 47, in _ensure_ffi_initialized
    modules=cls._modules,
  File "/Users/jgray/code/cryptography/cryptography/hazmat/bindings/utils.py", line 80, in build_ffi
    extra_link_args=extra_link_args,
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/cffi/api.py", line 341, in verify
    lib = self.verifier.load_library()
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/cffi/verifier.py", line 74, in load_library
    self._compile_module()
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/cffi/verifier.py", line 139, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/cffi/ffiplatform.py", line 25, in compile
    outputfilename = _build(tmpdir, ext)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/cffi/ffiplatform.py", line 50, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: CompileError: command 'gcc' failed with exit status 1

[reaperhulk]

Without being able to replicate this I suspect that this relates to the isysroot argument. Try running this command and see what you get CFLAGS="-I/usr/include" pip install cryptography

[jeremygray]

This definitely helps (pip reports success). However, I can't import from
within python

>>> from cryptography.fernet import Fernet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named cryptography.fernet

And same for just import cryptopgraphy

My site-packages has a new directory, with contents:

$ ls cryptography/
_Cryptography_cffi_48bbf0ebx93c91939.so*

_Cryptography_cffi_8f86901cxc1767c5a.so*
_Cryptography_cffi_684bb40axf342507b.so*
_Cryptography_cffi_be05eb56x6daa9a79.so*

--Jeremy

On Sun, May 4, 2014 at 10:11 AM, Paul Kehrer notifications@github.comwrote:

Without being able to replicate this I suspect that this relates to the
isysroot argument. Try running this command and see what you get CFLAGS="-I/usr/include"
pip install cryptography

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1013#issuecomment-42133849
.

[reaperhulk]

So it built the modules but didn't install the actual python code... What version of pip are you running? Could you upgrade to the latest and try again?

[jeremygray]

success! thanks much.

I did hit a small bump along the way, but its a pip 1.5.5 thing rather than cryptography thing. In case its useful to someone, here are the details:

I had pip 1.2.1, so I just upgraded to pip 1.5.5, uninstalled cryptography, and tried again. The resulting error from pip was surprising, and happens for all the packages I've tried (not just cryptography). I can clearly see stuff if I point a browser at https://pypi.python.org/simple/cryptography/ . And I get same error if I try with sudo:

$ CFLAGS="-I/usr/include" pip install cryptography Downloading/unpacking cryptography
Cannot fetch index base URL https://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement cryptography
Cleaning up...
No distributions at all found for cryptography
Storing debug log for failure in /Users/jgray/.pip/pip.log

So I manually downloaded the tarball for v0.3 and did:

$ CFLAGS="-I/usr/include" pip install ~/Downloads/cryptography-0.3.tar.gz 

pip reports success and I can import cryptography from within python.

[dstufft]

I bet if you do pip install -vvv cryptography you'll get some sort of SSL error.

[jeremygray]

yes indeed. certificate verify failed, seemingly due to the more recent version of pip using an old / bad version of openssl (?!)

$ pip install -vvv cryptography --upgrade
Getting page https://pypi.python.org/simple/cryptography/
Could not fetch URL https://pypi.python.org/simple/cryptography/: connection error: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

from looking at your comments from a year ago here pypa/pip#829 I get

$  python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 0.9.7l 28 Sep 2006

my default (Mac) version of openssl is more recent (OpenSSL 0.9.8y 5 Feb 2013)

[reaperhulk]

Where did you get your current Python from? I assume it came from some distribution?

[jeremygray]

EPD canopy:

Enthought Python Distribution -- www.enthought.com
Version: 7.3-2 (32-bit)
Python 2.7.3 |EPD 7.3-2 (32-bit)| (default, Apr 12 2012, 11:28:34)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin

I've been using 32-bit python due to some libs needed for a package that I contribute to. Hopefully possible to go to 64-bit python soon.

[Ivoz]

@jeremygray Latest Enthought 32-bit Canopy Express Python

[jeremygray]

thanks everyone--very helpful.

@Ivoz I'm not sure what you are suggesting -- do a fresh re-install of everything canopy-related? The most recent version of pip I am seeing through the canopy package manager is v1.0.2

[Ivoz]

That, or find out if enthought has a way of upgrading OpenSSL for their previous releases. pip 1.0 is now 5 minor releases behind :(

[jeremygray]

Thanks much, very sensible. If anyone with the same issue ever reads this far, this old SO posting looks relevant http://stackoverflow.com/questions/15441224/can-i-relink-enthought-python-to-new-version-of-openssl-on-mac-os-x