New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.7.0 :: The Python ssl extension was not compiled. Missing the OpenSSL lib? #1184

Open
rmerkushin opened this Issue Jul 4, 2018 · 35 comments

Comments

Projects
None yet
@rmerkushin
Copy link

rmerkushin commented Jul 4, 2018

Platform: macOS High Sierra 10.13.5
Pyenv version: Install by git clone
Python: 3.7.0
OpenSSL version: 1.0.2o_2 (installed from brew)

tricks below doesn't works for me:

CFLAGS="-I$(brew --prefix openssl)/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib" \
pyenv install -v 3.7.0
@Packets-n-Python

This comment has been minimized.

Copy link

Packets-n-Python commented Jul 6, 2018

Same with Ubuntu 14

CFLAGS=-I/usr/include/openssl
LDFLAGS=-L/usr/local/lib
pyenv install 3.7.0
Downloading Python-3.7.0.tar.xz...
-> https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
Installing Python-3.7.0...
ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?

Please consult to the Wiki page to fix the problem.
https://github.com/pyenv/pyenv/wiki/Common-build-problems

BUILD FAILED (Ubuntu 14.04 using python-build 1.2.5-4-gc057a80)

Inspect or clean up the working tree at /tmp/python-build.20180706171040.46966
Results logged to /tmp/python-build.20180706171040.46966.log

Last 10 log lines:
install|*) ensurepip="" ;;
esac;
./python -E -m ensurepip
$ensurepip --root=/ ;
fi
Looking in links: /tmp/tmpds7w9_se
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-10.0.1 setuptools-39.0.1

@gamesbrainiac

This comment has been minimized.

Copy link

gamesbrainiac commented Jul 7, 2018

I have the same issue. I also tried the following:

brew install libffi

Since it seems to do the trick on Ubuntu. This however does not work on macOS.

@mhart

This comment has been minimized.

Copy link

mhart commented Jul 10, 2018

Seeing the same thing on Amazon Linux (ie, Fedora essentially)

@ladyrick

This comment has been minimized.

Copy link

ladyrick commented Jul 22, 2018

still doesn't work yet.
have to use python 3.6.6

@lgw4

This comment has been minimized.

Copy link

lgw4 commented Jul 23, 2018

I can successfully install CPython 2.7.15, 3.6.6, and 3.7.0 using pyenv v1.2.5. on macOS 10.13.6. I (briefly) searched for differences between v1.2.5 and v1.2.6 but didn't see anything obvious that would cause this problem.

@jhunkeler jhunkeler referenced this issue Jul 29, 2018

Open

Link Python against local OpenSSL #1191

3 of 5 tasks complete
@lukebakken

This comment has been minimized.

Copy link

lukebakken commented Jul 31, 2018

I ran into this issue with openssl 1.0.2o_2 installed. The following allowed me to build and install Python 3.7.0 via pyenv:

brew install 'openssl@1.1'
CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl@1.1)" pyenv install 3.7.0
@lgw4

This comment has been minimized.

Copy link

lgw4 commented Jul 31, 2018

@lukebakken I also got it to work with OpenSSL 1.0.2o_2 by passing the same CONFIGURE_OPTS but for the openssl package:

CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" pyenv install 3.7.0

I wish I knew why this wasn't needed under pyenv 1.2.5.

@lukebakken

This comment has been minimized.

Copy link

lukebakken commented Jul 31, 2018

@lgw4 I'll give that a try just for grins. It's interesting that 1.0.2 worked. Prior to installing OpenSSL 1.1 the test for OpenSSL failed with a missing symbol in the dynamic library (SSL_new, I think), which I would have expected to be #ifdef'd for compatibility with older OpenSSL versions 🤷‍♂️

@lgw4

This comment has been minimized.

Copy link

lgw4 commented Jul 31, 2018

@lukebakken FWIW, I've also got the following environment variables set:

CFLAGS=-I/usr/local/opt/openssl/include
CPPFLAGS=-I/usr/local/opt/openssl/include
LDFLAGS=-L/usr/local/opt/openssl/lib
@lukebakken

This comment has been minimized.

Copy link

lukebakken commented Jul 31, 2018

I uninstalled openssl@1.1 and confirmed only the one OpenSSL version is installed:

$ ll /usr/local/opt/openssl*
lrwxr-xr-x  1 lbakken  admin  26 Jul 31 08:31 /usr/local/opt/openssl -> ../Cellar/openssl/1.0.2o_2
lrwxr-xr-x  1 lbakken  admin  26 Jul 31 08:31 /usr/local/opt/openssl@1.0 -> ../Cellar/openssl/1.0.2o_2

None of the environment variables mentioned here are exported, and I ran the installation as follows:

CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" pyenv install 3.7.0

Installation succeeds, and you can see the correct compiler and linker arguments in config.log during the feature testing phase:

configure:16986: checking whether compiling and linking against OpenSSL works
Trying link with OPENSSL_LDFLAGS=-L/usr/local/opt/openssl/lib; OPENSSL_LIBS=-lssl -lcrypto; OPENSSL_INCLUDES=-I/usr/local/opt/openssl/include
configure:17008: clang -o conftest  -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/Users/lbakken/.pyenv/versions/3.7.0/include  -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl/lib -L/Users/lbakken/.pyenv/versions/3.7.0/lib  -L/usr/local/opt/openssl/lib conftest.c -lssl -lcrypto -ldl  -framework CoreFoundation >&5
configure:17008: $? = 0
configure:17010: result: yes

So it looks like, for Python 3.7.0 on OS X at least, the --with-openssl argument to configure must be used. Note that I did try the fix proposed in #1191 but it didn't work, probably because I don't believe the pyenv-local OpenSSL was built (?).

@jhunkeler

This comment has been minimized.

Copy link

jhunkeler commented Aug 1, 2018

#1191 is only partially correct (for now). The system I wrote that patch for was running OSX 10.11 so I didn't see the next edge case coming...

On OSX 10.13 it proceeds to compile OpenSSL 1.0* as you'd except, however it fails to produce a library containing the SSL_new symbol. Python cannot link against it, so everything falls apart at that point.

After a little tinkering I managed to build OpenSSL, albeit the stable release, and it appears to work fine when compiled against Python 3.7.0.

(PS - 3.6.6 fails on 10.13 in the same way. If I switch its build script to use the stable release of OpenSSL it does seem to fix the problem there too. Hmm.)

index 1ff0ac29..2d5552d1 100755
--- a/plugins/python-build/bin/python-build
+++ b/plugins/python-build/bin/python-build
@@ -1504,6 +1504,9 @@ use_homebrew_openssl() {
 }

 build_package_mac_openssl() {
+  OPENSSL_VERSION_LTS='1.0.*'
+  OPENSSL_VERSION_STABLE='1.1.*'
+
   # Install to a subdirectory since we don't want shims for bin/openssl.
   OPENSSL_PREFIX_PATH="${PREFIX_PATH}/openssl"

@@ -1520,12 +1523,25 @@ build_package_mac_openssl() {
   # Hint OpenSSL that we prefer a 64-bit build.
   export KERNEL_BITS="64"
   OPENSSL_CONFIGURE="${OPENSSL_CONFIGURE:-./config}"
+  OPENSSL_OPTIONS="zlib-dynamic shared"

   local nokerberos
-  [[ "$1" != openssl-1.0.* ]] || nokerberos=1
+  [[ "$1" != openssl-${OPENSSL_VERSION_LTS} ]] || nokerberos=1
+
+  local openssl_stable
+  [[ "$1" == openssl-${OPENSSL_VERSION_STABLE} ]] && openssl_stable=1
+
+  # Set additional flags based on LTS versus Stable
+  if [[ ! $openssl_stable ]]; then
+    OPENSSL_OPTIONS="${OPENSSL_OPTIONS} no-ssl2 no-ssl3"
+    [[ $nokerberos ]] && OPENSSL_OPTIONS="${OPENSSL_OPTIONS} no-krb5"
+  else
+    # Deprecated: SSL2, KRB5
+    OPENSSL_OPTIONS="${OPENSSL_OPTIONS} no-ssl3"
+  fi

   # Compile a shared lib with zlib dynamically linked, no kerberos.
-  package_option openssl configure --openssldir="$OPENSSLDIR" zlib-dynamic no-ssl2 no-ssl3 no-krb5 shared
+  package_option openssl configure --openssldir="$OPENSSLDIR" ${OPENSSL_OPTIONS}

   # Default MAKE_OPTS are -j 2 which can confuse the build. Thankfully, make
   # gives precedence to the last -j option, so we can override that.
diff --git a/plugins/python-build/share/python-build/3.7.0 b/plugins/python-build/share/python-build/3.7.0
index f65d2a73..11a8ce7f 100644
--- a/plugins/python-build/share/python-build/3.7.0
+++ b/plugins/python-build/share/python-build/3.7.0
@@ -1,5 +1,5 @@
 #require_gcc
-install_package "openssl-1.0.2k" "https://www.openssl.org/source/openssl-1.0.2k.tar.gz#6b3977c61f2aedf0f96367dcfb5c6e578cf37e7b8d913b4ecb6643c3cb88d8c0" mac_openssl --if has_broken_mac_openssl
+install_package "openssl-1.1.0h" "https://www.openssl.org/source/openssl-1.1.0h.tar.gz#5835626cde9e99656585fc7aaa2302a73a7e1340bf8c14fd635a62c66802a517" mac_openssl --if has_broken_mac_openssl
 install_package "readline-6.3" "https://ftpmirror.gnu.org/readline/readline-6.3.tar.gz#56ba6071b9462f980c5a72ab0023893b65ba6debb4eeb475d7a563dc65cafd43" standard --if has_broken_mac_readline
 if has_tar_xz_support; then
   install_package "Python-3.7.0" "https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz#0382996d1ee6aafe59763426cf0139ffebe36984474d0ec4126dd1c40a8b3549" ldflags_dirs standard verify_py37 ensurepip```
@lgw4

This comment has been minimized.

Copy link

lgw4 commented Aug 1, 2018

@jhunkeler Awesome! However, had you switched to https://www.openssl.org/source/openssl-1.0.2o.tar.gz, I bet the LTS would succeed, too.

@jhunkeler

This comment has been minimized.

Copy link

jhunkeler commented Aug 1, 2018

D'oh! Thank you! Yes, the latest LTS works fine 👍

diff --git a/plugins/python-build/share/python-build/3.7.0 b/plugins/python-build/share/python-build/3.7.0
index f65d2a73..173f3249 100644
--- a/plugins/python-build/share/python-build/3.7.0
+++ b/plugins/python-build/share/python-build/3.7.0
@@ -1,5 +1,5 @@
 #require_gcc
-install_package "openssl-1.0.2k" "https://www.openssl.org/source/openssl-1.0.2k.tar.gz#6b3977c61f2aedf0f96367dcfb5c6e578cf37e7b8d913b4ecb6643c3cb88d8c0" mac_openssl --if has_broken_mac_openssl
+install_package "openssl-1.0.2o" "https://www.openssl.org/source/openssl-1.0.2o.tar.gz#ec3f5c9714ba0fd45cb4e087301eb1336c317e0d20b575a125050470e8089e4d" mac_openssl --if has_broken_mac_openssl
 install_package "readline-6.3" "https://ftpmirror.gnu.org/readline/readline-6.3.tar.gz#56ba6071b9462f980c5a72ab0023893b65ba6debb4eeb475d7a563dc65cafd43" standard --if has_broken_mac_readline
 if has_tar_xz_support; then
   install_package "Python-3.7.0" "https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz#0382996d1ee6aafe59763426cf0139ffebe36984474d0ec4126dd1c40a8b3549" ldflags_dirs standard verify_py37 ensurepip```
@rmerkushin

This comment has been minimized.

Copy link
Author

rmerkushin commented Aug 2, 2018

Yep, this working for me in zsh:

CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" pyenv install 3.7.0

But not working in fish :(

set CONFIGURE_OPTS "--with-openssl="(brew --prefix openssl); and pyenv install 3.7.0
@lgw4

This comment has been minimized.

Copy link

lgw4 commented Aug 2, 2018

@rmerkushin I think your fish syntax is off a bit.

env CONFIGURE_OPTS "--with-openssl="(brew --prefix openssl)"" pyenv install 3.7.0

I don’t use fish so I can’t test this.

@rmerkushin

This comment has been minimized.

Copy link
Author

rmerkushin commented Aug 3, 2018

@lgw4 nope, it's not working. I think it's python installation script not properly working with fish :(

@zyong812

This comment has been minimized.

Copy link

zyong812 commented Aug 7, 2018

use brew install pyenv instead.
It solved the same problem for me.

@kzidane

This comment has been minimized.

Copy link

kzidane commented Aug 8, 2018

@Packets-n-Python did you end up finding a solution to install 3.7 on Ubuntu 14.04?

@tonal

This comment has been minimized.

Copy link

tonal commented Aug 15, 2018

@rmerkushin

This comment has been minimized.

Copy link
Author

rmerkushin commented Aug 18, 2018

@zyong812 Do you export any variables before install Python 3.7? Which version of OpenSSL do you install?

@rmerkushin

This comment has been minimized.

Copy link
Author

rmerkushin commented Aug 18, 2018

Holy crap! I found a solution for fish!

git clone https://github.com/pyenv/pyenv.git ~/.pyenv
git clone https://github.com/pyenv/pyenv-virtualenv.git (pyenv root)/plugins/pyenv-virtualenv
brew install autoconf pkg-config openssl readline

My config.fish:

set -x PYENV_ROOT ~/.pyenv
set -x PATH ~/.pyenv/bin $PATH

if status --is-interactive
    source (pyenv init - | psub)
    source (pyenv virtualenv-init - | psub)
end

P.S.: No need extra environment variables

@paulswartz

This comment has been minimized.

Copy link

paulswartz commented Aug 23, 2018

For fish (@rmerkushin):

env CONFIGURE_OPTS=--with-openssl=(brew --prefix openssl@1.1) pyenv install 3.7.0
@EscapeLife

This comment has been minimized.

Copy link

EscapeLife commented Sep 9, 2018

MacOS system,please use brew install instead of Basic GitHub Checkout.

$ brew install pyenv

If you force do, can run this. Basic GitHub Checkout installation Python3.7.0 has bug.

$ brew install 'openssl@1.1'
$ CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl@1.1)" pyenv install 3.7.0

@rmerkushin l test can be install Python3.7.0 with pyenv normally,hope to help you!

@luchevoi

This comment has been minimized.

Copy link

luchevoi commented Oct 16, 2018

I've faced that issue in Debian 8.11 (jessie) with default "OpenSSL 1.0.1t 3 May 2016" version.
So, we have to make our custom openssl build.
For example we are logged in as user "test".

cd ~
mkdir openssl
cd openssl
wget "https://www.openssl.org/source/openssl-1.0.2p.tar.gz"
tar xf openssl-1.0.2p.tar.gz
cd openssl-1.0.2p

Latest OpenSSL 1.0.2 has been downloaded and extracted into "/home/test/openssl" directory. Now let's build it into the same directory. I took instructions from here.

./config --prefix=/home/test/openssl --openssldir=/home/test/openssl shared zlib
make
make test
make install

If compiling and testing passed well, you are become able to just link your fresh OpenSSL install to pyenv.

LD_RUN_PATH="/home/test/openssl/lib" LDFLAGS="-L/home/test/openssl/lib" CPPFLAGS="-I/home/test/openssl/include" CFLAGS="-I/home/test/openssl/include" CONFIGURE_OPTS="--with-openssl=/home/test/openssl" pyenv install 3.7.0

LD_RUN_PATH is a necessary option!

@astudnitskiy

This comment has been minimized.

Copy link

astudnitskiy commented Oct 28, 2018

On macOS with Homebrew, make sure that pkg-config is installed. Installing pyenv from Homebrew actually takes care of that. However, with git clone installation method, necessary dependences need to be installed manually.

I had the issue on one of my Macs, while on another it was building fine. I tried adding OpenSSL 1.0.2, 1.1, LibreSSL paths in CFLAGS and LDFLAGS - no change. Finally, comparing installed Homebrew packages and build logs helped to narrow this down to pkg-config. You actually don't need to add OpenSSL paths manually, pkg-config takes care of that once it's installed (unlike zlib and sqlite, by the way).

al45tair added a commit to al45tair/pyenv that referenced this issue Nov 14, 2018

@al45tair al45tair referenced this issue Nov 14, 2018

Open

Use the newly built OpenSSL (fixes #1184). #1241

5 of 5 tasks complete
@al45tair

This comment has been minimized.

Copy link

al45tair commented Nov 14, 2018

For Mac users, pyenv was supposed to work automatically by downloading and building a copy of OpenSSL specifically for the use of the Python version you're installing. Unfortunately, the code to do that neglected to actually tell the Python build process to use that copy of OpenSSL, which is why a lot of people in this thread advise you to install OpenSSL separately (e.g. using brew). But that shouldn't be necessary — pyenv has already built and installed a version of OpenSSL. The pull request I just submitted fixes things so that pyenv will actually tell Python to use the version it's just downloaded and built.

@laike9m

This comment has been minimized.

Copy link

laike9m commented Nov 23, 2018

For Mac users, pyenv was supposed to work automatically by downloading and building a copy of OpenSSL specifically for the use of the Python version you're installing. Unfortunately, the code to do that neglected to actually tell the Python build process to use that copy of OpenSSL, which is why a lot of people in this thread advise you to install OpenSSL separately (e.g. using brew). But that shouldn't be necessary — pyenv has already built and installed a version of OpenSSL. The pull request I just submitted fixes things so that pyenv will actually tell Python to use the version it's just downloaded and built.

pyenv-installer does not seem to be picking up the fix yet. So it's coming with the next release?

@al45tair

This comment has been minimized.

Copy link

al45tair commented Nov 23, 2018

pyenv-installer does not seem to be picking up the fix yet. So it's coming with the next release?

That's up to the pyenv maintainers. They might decide it isn't the right fix and that they want to do something different.

@laike9m

This comment has been minimized.

Copy link

laike9m commented Nov 23, 2018

pyenv-installer does not seem to be picking up the fix yet. So it's coming with the next release?

That's up to the pyenv maintainers. They might decide it isn't the right fix and that they want to do something different.

Hope it gets merged soon.

@imanhodjaev

This comment has been minimized.

Copy link

imanhodjaev commented Nov 29, 2018

For fish shell users you can here is fish version which worked for me

$ set -x CONFIGURE_OPTS --with-openssl=(brew --prefix openssl)
$ pyenv install 3.7.1
@SixExtreme

This comment has been minimized.

Copy link

SixExtreme commented Dec 11, 2018

My macos version is 10.14.2.
You should do something like below.

$ xcode-select --install
$ sudo installer -pkg \ 
    /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg \
    -target /
$ brew install readline xz openssl@1.1 pyenv
$ CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl@1.1)" pyenv install 3.7.1
@SpaceNet

This comment has been minimized.

Copy link

SpaceNet commented Dec 17, 2018

macOS Mojave 10.14.2

$ xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" to install updates

Software Update

$ sudo installer -pkg \ 
    /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg \
    -target /
$brew install pyenv
Updating Homebrew...
Error: pyenv 1.1.3 is already installed
To upgrade to 1.2.8_1, run `brew upgrade pyenv`
$ brew upgrade pyenv
・・・・・・
$ pyenv install 3.7.1
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.1.tar.xz...
-> https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
Installing Python-3.7.1...
python-build: use readline from homebrew
Installed Python-3.7.1 to /Users/user/.pyenv/versions/3.7.1
@deathbywedgie

This comment has been minimized.

Copy link

deathbywedgie commented Dec 17, 2018

I was going insane trying to figure this out (macOS High Sierra 10.13.6), and my issue also turned out to be that pkg-config needed to be installed. That really needs to be added to the Common Build Problems page!

@Almenon

This comment has been minimized.

Copy link

Almenon commented Dec 30, 2018

Same issue, pkg-config and openssl 1.1.0 installed, as well as all the other packages reccomended for ubuntu instance on the wiki.

CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl@1.1)" pyenv install 3.6.6

didn't work for me :(

I'm using Ubuntu 18.04.1 w/ ubuntu subsystem for windows

@cdeyoung

This comment has been minimized.

Copy link

cdeyoung commented Jan 29, 2019

For fish shell users you can here is fish version which worked for me

$ set -x CONFIGURE_OPTS --with-openssl=(brew --prefix openssl)
$ pyenv install 3.7.1

This worked for me. It feels like a hack, but at least Python is running.

Thanks, @imanhodjaev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment