Python package fails to link to openssl on a fresh install of OSX mavericks

[jcftang]

The python package fails to link against openssl (both built with hashdist) see output of build at https://gist.github.com/jcftang/3d84d9dd19903f5017c6

-- old -- not so descriptive of correct issue.

The current python package does not depend on lex and yacc, on a clean system python will fail to compile if they are not present.

Python should either depend on these two packages as well somehow or else hashdist should force the user to provide their own lex and yacc.

I'm trying to verify this behaviour on a fresh OSX based system with a fresh install of homebrew.

[certik]

Can you post the error? I didn't know Python depended on lex and yacc.

[jcftang]

I think I chased down the problem... but I am getting

[python] import hashlib
[python] ERROR:root:code for hash md5 was not found.
[python] Traceback (most recent call last):
[python]   File "/Users/jtang/.hashdist/bld/python/kkqkwipk2du5/lib/python2.7/hashlib.py", line 139, in 
[python]     globals()[__func_name] = __get_hash(__func_name)
[python]   File "/Users/jtang/.hashdist/bld/python/kkqkwipk2du5/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
[python]     raise ValueError('unsupported hash type ' + name)
[python] ValueError: unsupported hash type md5
[python] ERROR:root:code for hash sha1 was not found.

which is unrelated to my initial problem

[certik]

That's missing openssl or some other crypto library. I am glad I've added these checks into the build.

[jcftang]

I've added openssl into the python builds_with variable and its currently still failing. +1 for the checks!

[certik]

Post the whole build log (into gist). There will be a line which tries to build hashlib and it fails (usually gcc fails to find some openssl library, or it fails to link or something). We need to know the reason it fails.

[jcftang]

Here's the gist, I will also rename the title and description of this issue.

https://gist.github.com/jcftang/3d84d9dd19903f5017c6

[jcftang]

I didn't see this but...

building '_hashlib' extension
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jtang/.hashdist/tmp/python-kevgacw3uovu-1/Mac/Include -I. -IInclude -I./Include -I/Users/jtang/.hashdist/bld/bzip2/l5bepmrkvfa3/include -I/Users/jtang/.hashdist/bld/gettext/ov2eejnidg7x/include -I/Users/jtang/.hashdist/bld/libiconv/mvylctw5tlei/include -I/Users/jtang/.hashdist/bld/libxml2/6j5za7w2icyt/include -I/Users/jtang/.hashdist/bld/ncurses/qetfh6mrggk3/include -I/Users/jtang/.hashdist/bld/openssl/2r5b2wt3hrvk/include -I/Users/jtang/.hashdist/bld/readline/xpkuojccucms/include -I/Users/jtang/.hashdist/bld/sqlite/bytl5vd6rmlh/include -I/Users/jtang/.hashdist/bld/zlib/gx4hx7ykbdoy/include -I/usr/local/include -I/Users/jtang/.hashdist/tmp/python-kevgacw3uovu-1/Include -I/Users/jtang/.hashdist/tmp/python-kevgacw3uovu-1 -c /Users/jtang/.hashdist/tmp/python-kevgacw3uovu-1/Modules/_hashopenssl.c -o build/temp.macosx-10.9-x86_64-2.7/Users/jtang/.hashdist/tmp/python-kevgacw3uovu-1/Modules/_hashopenssl.o
gcc -bundle -undefined dynamic_lookup -L/Users/jtang/.hashdist/bld/bzip2/l5bepmrkvfa3/lib -L/Users/jtang/.hashdist/bld/gettext/ov2eejnidg7x/lib -L/Users/jtang/.hashdist/bld/launcher/h3gb4lb7cpvj/lib -L/Users/jtang/.hashdist/bld/libiconv/mvylctw5tlei/lib -L/Users/jtang/.hashdist/bld/libxml2/6j5za7w2icyt/lib -L/Users/jtang/.hashdist/bld/ncurses/qetfh6mrggk3/lib -L/Users/jtang/.hashdist/bld/openssl/2r5b2wt3hrvk/lib -L/Users/jtang/.hashdist/bld/pkg-config/we3j2ko72srm/lib -L/Users/jtang/.hashdist/bld/readline/xpkuojccucms/lib -L/Users/jtang/.hashdist/bld/sqlite/bytl5vd6rmlh/lib -L/Users/jtang/.hashdist/bld/zlib/gx4hx7ykbdoy/lib build/temp.macosx-10.9-x86_64-2.7/Users/jtang/.hashdist/tmp/python-kevgacw3uovu-1/Modules/_hashopenssl.o -L/Users/jtang/.hashdist/bld/bzip2/l5bepmrkvfa3/lib -L/Users/jtang/.hashdist/bld/gettext/ov2eejnidg7x/lib -L/Users/jtang/.hashdist/bld/libiconv/mvylctw5tlei/lib -L/Users/jtang/.hashdist/bld/libxml2/6j5za7w2icyt/lib -L/Users/jtang/.hashdist/bld/ncurses/qetfh6mrggk3/lib -L/Users/jtang/.hashdist/bld/openssl/2r5b2wt3hrvk/lib -L/Users/jtang/.hashdist/bld/readline/xpkuojccucms/lib -L/Users/jtang/.hashdist/bld/sqlite/bytl5vd6rmlh/lib -L/Users/jtang/.hashdist/bld/zlib/gx4hx7ykbdoy/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.macosx-10.9-x86_64-2.7/_hashlib.so
ld: warning: directory not found for option '-L/Users/jtang/.hashdist/bld/launcher/h3gb4lb7cpvj/lib'
ld: warning: directory not found for option '-L/Users/jtang/.hashdist/bld/pkg-config/we3j2ko72srm/lib'
ld: warning: ld: warning: ignoring file /Users/jtang/.hashdist/bld/openssl/2r5b2wt3hrvk/lib/libssl.dylib, file was built for i386 which is not the architecture being linked (x86_64): /Users/jtang/.hashdist/bld/openssl/2r5b2wt3hrvk/lib/libssl.dylibignoring file /Users/jtang/.hashdist/bld/openssl/2r5b2wt3hrvk/lib/libcrypto.dylib, file was built for i386 which is not the architecture being linked (x86_64): /Users/jtang/.hashdist/bld/openssl/2r5b2wt3hrvk/lib/libcrypto.dylib
 

32bit vs 64bit issues

[certik]

Also _ssl fails (https://gist.github.com/jcftang/3d84d9dd19903f5017c6#file-gistfile1-txt-L1003).

[certik]

Somehow you built openssl in 32 bit mode?

[jcftang]

I pulled in openssl as I have done previously on other systems, it's a fairly stock system. I did not modify the openssl package at all. @certik any recommendations of 'forcing' 64bit only builds with openssl?

[certik]

How do you build any library in 32bit mode? I don't understand how this can happen.

[certik]

This has some info:

http://stackoverflow.com/questions/5220435/file-was-built-for-i386-which-is-not-the-architecture-being-linked-x86-64-whil

I think wrong gcc flags are used for openssl. Can you post full log of openssl? You can grab it from the artifact directory, it's gzipped (on my computer it is /scratch/bld2/openssl/kuye7ap7sdck/build.log.gz). Instead of gcc -march=i386, one has to use gcc -march=native I guess. My openssl log doesn't contain any march flags, but I guess we should be able to see from your log what is wrong.

[jcftang]

I just checked bison to compare

MacBook-Air:bld jtang$ file bison/ikzk2jl3bke6/bin/bison 
bison/ikzk2jl3bke6/bin/bison: Mach-O 64-bit executable x86_64
MacBook-Air:bld jtang$ file openssl/vhdxvikhmhas/bin/openssl 
openssl/vhdxvikhmhas/bin/openssl: Mach-O executable i386

Here's the gist for the build log https://gist.github.com/jcftang/debc71bd259f9debbb57

[certik]

You can see, that the openssl log contains -arch i386 in gcc calls. So this needs to be changed.
It's the CFLAG line near the top, which contains this incorrect flag.

[ahmadia]

Just another note, it's really important that you set a consistent OSX_DEVELOPMENT_TARGET when building on OS X. The presence of the arch flags is also troubling. There's an ARCHFLAGS environment variable that may be getting set. Honestly, you should probably look at what homebrew is doing and just reuse that.

[jcftang]

@ahmadia I've extended my profile with the homebrew.yaml file so the OSX_DEVELOPMENT_TARGET should be set, openssl is the only one that is misbehaving. I've not needed to mess with homebrew's settings. I will dig around.

[ahmadia]

I mean refer to this: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/openssl.rb

OpenSSL does not appear to be completely trivial to install on OS X. I'm not very interested in Universal installs so you can probably safely skip that.

[certik]

Looks like you need to specify the 32 vs 64 bit arch explicitly on OS X, as done in the openssl.rb.

[ahmadia]

+1 to Ondřej's observation.

[jcftang]

sigh this is messy on OSX

[jcftang]

I guess these are the two relevant 'arch' options for OSX

x86_64 => darwin64-x86_64-cc enable-ec_nistp-64_gcc_128
i386 => darwin-i386-cc

The openssl build should either be parameterised or a decision could just be made to settle for a 64bit build across the board (doing this could rule out arm based systems which aren't quite 64bit yet)

I modified my openssl build with this

./Configure --prefix=$ARTIFACT shared zlib-dynamic -I$ZLIB_DIR/include darwin64-x86_64-cc enable-ec_nistp-64_gcc_128

and it links correctly with python now.

[ahmadia]

The openssl build should either be parameterised or a decision could just be made to settle for a 64bit build across the board (doing this could rule out arm based systems which aren't quite 64bit yet)

You might want to just hard-code those decision on the Darwin platform. OpenSSL is low-level enough that I'm more worried about getting it installed than its particular configuration.

[jcftang]

Would you accept patch like this?

diff --git a/darwin.yaml b/darwin.yaml
index 60ca931..dc62163 100644
--- a/darwin.yaml
+++ b/darwin.yaml
@@ -6,6 +6,8 @@ parameters:
   PATH: /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
   PROLOGUE: |
     export MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion | sed "s/\(10.[0-9]\).*/\1/")
+    export DARWIN_OPENSSL_ARCH="darwin64-x86_64-cc enable-ec_nistp-64_gcc_128"
+    #export DARWIN_OPENSSL_ARCH="darwin-i386-cc"
 
 packages:
   blas:
diff --git a/pkgs/openssl/openssl.yaml b/pkgs/openssl/openssl.yaml
index 3c6fbb1..401f0bd 100644
--- a/pkgs/openssl/openssl.yaml
+++ b/pkgs/openssl/openssl.yaml
@@ -1,7 +1,7 @@
 extends: [autotools_package]
 
 dependencies:
-  build: [zlib]
+  build: [zlib, pkg-config]
 
 sources:
   - url: https://www.openssl.org/source/openssl-1.0.1g.tar.gz
@@ -26,4 +26,10 @@ build_stages:
   mode: replace
   bash: |
     #./config --prefix=$ARTIFACT shared zlib-dynamic -I$ZLIB_DIR/include -Wl,-rpath=$ARTIFACT/lib -Wl,-rpath=$ZLIB_DIR/lib
-    ./config --prefix=$ARTIFACT shared zlib-dynamic -I$ZLIB_DIR/include
+    ./Configure --prefix=$ARTIFACT shared zlib-dynamic -I$ZLIB_DIR/include $DARWIN_OPENSSL_ARCH
+
+when_build_dependency:
+- prepend_path: PKG_CONFIG_PATH
+  value: '${ARTIFACT}/lib/pkgconfig'
+- prepend_path: PATH
+  value: '${ARTIFACT}/bin'

I can remove the pkg-config stuff if necessary.

[ahmadia]

+    export DARWIN_OPENSSL_ARCH="darwin64-x86_64-cc enable-ec_nistp-64_gcc_128"

Would be rejected :) That should be a parameter that the OpenSSL package exposes as an environment variable. The MACOSX_DEPLOYMENT_TARGET environment variable is special because we want all builds to use it globally.

[jcftang]

Heh, you should show me with what you mean with a patch ;)

[ahmadia]

But then how would you learn? :)

[jcftang]

Upon thinking about, what should the parameter be? build_arch, target_arch any preferences?

[certik]

why not just "arch"?

[ahmadia]

It's specific to OpenSSL, so it should be darwin_openssl_arch. There might be a global arch parameter that the profile sets.

Setting a parameter is optional in my mind, I would hardcode for Darwin in openssl.yaml

[jcftang]

diff --git a/osx.yaml b/osx.yaml
index 86861c5..c05bcd5 100644
--- a/osx.yaml
+++ b/osx.yaml
@@ -2,6 +2,7 @@ extends:
  - file: common.yaml
 
 parameters:
+  darwin_openssl_arch: 'darwin64-x86_64-cc enable-ec_nistp-64_gcc_128'
   platform: Darwin
   PATH: /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
   PROLOGUE: |
diff --git a/pkgs/openssl/openssl.yaml b/pkgs/openssl/openssl.yaml
index 3c6fbb1..4bb4cc7 100644
--- a/pkgs/openssl/openssl.yaml
+++ b/pkgs/openssl/openssl.yaml
@@ -26,4 +26,4 @@ build_stages:
   mode: replace
   bash: |
     #./config --prefix=$ARTIFACT shared zlib-dynamic -I$ZLIB_DIR/include -Wl,-rpath=$ARTIFACT/lib -Wl,-rpath=$ZLIB_DIR/lib
-    ./config --prefix=$ARTIFACT shared zlib-dynamic -I$ZLIB_DIR/include
+    ./Configure --prefix=$ARTIFACT shared zlib-dynamic -I$ZLIB_DIR/include {{darwin_openssl_arch}}

something liek the above? I just spent the last 30mins editting darwin.yaml instead of osx.yaml (using homebrew.yaml in my case) and I wasn't actually passing anything around :P

[ahmadia]

@jcftang - that looks right. Is that a custom build stage for OS X?

[jcftang]

Here's the diff

https://github.com/jcftang/hashstack2/tree/wip-osx-openssl-tweaks

I'm pretty sure I'm not doing the right thing here, i would assume since darwin_openssl_arch wont be defined elsewhere it will just be blank?

[jcftang]

I think this is more or else resolved for now, closing this issue. see c87698a