OpenSSL issues

[tigerhawkvok]

Too many issues will kill our team's development velocity, drastically.
Make sure you have checked all steps below.

Description

• Platform information (e.g. Ubuntu Linux 16.04): Ubuntu 20.04.6 LTS on WSL
• OS architecture (e.g. amd64): amd64 on WSL
• pyenv version: pyenv 2.3.35
• Python version: 3.11.7
• C Compiler information (e.g. gcc 7.3): gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

I'm installing with, after some guidance from #950

sudo apt-get update && sudo add-apt-repository ppa:ubuntu-toolchain-r/test && sudo apt-get update && sudo apt-get install software-properties-common curl wget build-essential make git zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev libssl-dev ncurses-dev libffi-dev tk-dev libbz2-dev liblzma-dev libcrypt-dev libnis-dev libncursesw5-dev -y && sudo locale-gen en_US en_US.UTF-8 && sudo dpkg-reconfigure locales

if [! -e brew ]
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi

echo -e "$(whoami)\tsoft\tnofile\t$(ulimit -Sn)0" | sudo tee -a /etc/security/limits.conf
echo -e "*\tsoft\tnofile\t$(ulimit -Sn)0" | sudo tee -a /etc/security/limits.conf
ulimit -S -n 4096

# gawk and file-formula: https://github.com/orgs/Homebrew/discussions/3601#discussioncomment-4150308 
brew update && brew upgrade && brew install openssl@3 sqlite readline zlib libxcrypt libscrypt libgcrypt bcrypt libnsl ncurses
brew install gawk file-formula make
brew install pyenv

# https://github.com/pyenv/pyenv/issues/950#issuecomment-1538617794
cd $HOME && wget https://www.openssl.org/source/openssl-1.1.1s.tar.gz && tar xf openssl-1.1.1s.tar.gz && cd $HOME/openssl-1.1.1s && ./config && make -j && sudo make install

# Hits destinations:
# install libcrypto.so.1.1 -> /usr/local/lib/libcrypto.so.1.1
# install libssl.so.1.1 -> /usr/local/lib/libssl.so.1.1

cd $HOME

export CPATH=$CPATH:/usr/local/include/openssl/:$(brew --prefix openssl)/include/
export LIBRARY_PATH=$LIBRARY_PATH:$(brew --prefix gcc):$(brew --prefix openssl)/lib/:$(brew --prefix zlib)/lib:$(brew --prefix bzip2)/lib:$(brew --prefix readline)/lib:$(brew --prefix ncurses)/lib:/usr/local/lib/:/usr/lib/x86_64-linux-gnu/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:$(brew --prefix openssl)/lib:$(brew --prefix zlib)/lib:$(brew --prefix bzip2)/lib:$(brew --prefix readline)/lib:$(brew --prefix ncurses)/lib:$(openssl version -d)/lib:$(openssl version -d)
export LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix ncurses)/lib -L/usr/local/lib -L$(brew --prefix gcc)/lib -L$(pkg-config --libs openssl)"
export CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix zlib)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(brew --prefix ncurses)/include -I$(brew --prefix gcc)/include"
export CPPFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix zlib)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(brew --prefix ncurses)/include -I$(brew --prefix gcc) -L$(pkg-config --libs openssl)"


CONFIGURE_OPTS="-with-openssl=$(brew --prefix openssl)" PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@3 PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" PYTHON_CONFIGURE_OPTS='--enable-shared --with-openssl-rpath=auto' pyenv install -v 3.11.7

but despite explicitly building OpenSSL (and trying separately with both Homebrew and from source), I'm still getting SSL issues and weird glibc dependency issues

Build log: pybuild.txt
Config log: config.log

I also tried these flags to use the directory specified in openssl version -d, but still failed:

export OPENSSL_ETC_DIR=$(openssl version -d | cut -d ' ' -f 2 | sed 's/.\{1\}$//' | sed 's/^.\{1\}//')
export CPATH=$CPATH:/usr/local/include/openssl/:$(brew --prefix openssl)/include/:$OPENSSL_ETC_DIR
export LIBRARY_PATH=$LIBRARY_PATH:$(brew --prefix gcc):$(brew --prefix openssl)/lib/:$(brew --prefix zlib)/lib:$(brew --prefix bzip2)/lib:$(brew --prefix readline)/lib:$(brew --prefix ncurses)/lib:/usr/local/lib/:/usr/lib/x86_64-linux-gnu/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:$(brew --prefix openssl)/lib:$(brew --prefix zlib)/lib:$(brew --prefix bzip2)/lib:$(brew --prefix readline)/lib:$(brew --prefix ncurses)/lib:$OPENSSL_ETC_DIR/lib:$OPENSSL_ETC_DIR
export LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix ncurses)/lib -L/usr/local/lib -L$(brew --prefix gcc)/lib -L$(pkg-config --libs openssl)"
export CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix zlib)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(brew --prefix ncurses)/include -I$(brew --prefix gcc)/include"
export CPPFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix zlib)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(brew --prefix ncurses)/include -I$(brew --prefix gcc) -L$(pkg-config --libs openssl)"

[native-api]

LD_LIBRARY_PATH is incorrect:

LD_LIBRARY_PATH=':/usr/local/lib:/home/linuxbrew/.linuxbrew/opt/openssl@3/lib:/home/linuxbrew/.linuxbrew/opt/zlib/lib:/home/linuxbrew/.linuxbrew/opt/bzip2/lib:/home/linuxbrew/.linuxbrew/opt/readline/lib:/home/linuxbrew/.linuxbrew/opt/ncurses/lib:OPENSSLDIR: "/home/linuxbrew/.linuxbrew/etc/openssl@3"/lib:OPENSSLDIR: "/home/linuxbrew/.linuxbrew/etc/openssl@3"'

Need to also add $(brew --prefix)/include and $(brew --prefix)/libs to corresponding envvars, in order for compiler to find non-keg formulae. Alternatively, you can use Homebrew-provided GCC which is supposed to add those dirs to search path by default.

configure:23235: checking whether OpenSSL provides required ssl module APIs
configure:23267: gcc -pthread -o conftest <...>
/usr/bin/ld: /home/linuxbrew/.linuxbrew/opt/openssl@3/lib/libcrypto.so: undefined reference to `dladdr@GLIBC_2.34'
/usr/bin/ld: /home/linuxbrew/.linuxbrew/opt/openssl@3/lib/libcrypto.so: undefined reference to `dlclose@GLIBC_2.34'
/usr/bin/ld: /home/linuxbrew/.linuxbrew/opt/openssl@3/lib/libcrypto.so: undefined reference to `fstat@GLIBC_2.33'
/usr/bin/ld: /home/linuxbrew/.linuxbrew/opt/openssl@3/lib/libcrypto.so: undefined reference to `dlerror@GLIBC_2.34'

[native-api]

We plan to allow Pyenv to use Homebrew in Linux by default if Pyenv itself is installed via Homebrew.

[trevorpfiz]

#950 (comment)