Python build fails on M1 Apple Silicon with arm64 homebrew

[rjmoggach]

Referencing other issues, I've been trying unsuccessfully for a native compile of 3.9.0 on an M1 MacBook Air using the following:

(FYI rosetta works without any issue with this config - goal is to compile both)

.zshrc

# PYENV
ARCH=`arch`
if [[ "${ARCH}"  == "arm64" ]]; then
    export PYENV_ROOT="/Users/rob/.pyenv/arm64"
else
    export PYENV_ROOT="/Users/rob/.pyenv/rosetta"
fi
PYENV_BIN="$PYENV_ROOT/bin"
export PYENV_SHELL=zsh
# export PYENV_ROOT=$(pyenv root)
# export PYENV_VERSION=$(pyenv version-name)
export PYTHONPATH=$PYENV_ROOT/shims


SDK_PATH="$(xcrun --show-sdk-path)"
export CPATH="${SDK_PATH}/usr/include"
export CFLAGS="-I${SDK_PATH}/usr/include/sasl $CFLAGS"
export CFLAGS="-I${SDK_PATH}/usr/include $CFLAGS"
export LDFLAGS="-L${SDK_PATH}/usr/lib $LDFLAGS"

if [[ "${ARCH}"  == "arm64" ]]; then
    PREFIX="/opt/homebrew/opt"
else
    PREFIX="/usr/local/opt"
fi

ZLIB="${PREFIX}/zlib"
BZIP2="${PREFIX}/bzip2"
READLINE="${PREFIX}/readline"
SQLITE="${PREFIX}/sqlite"    
OPENSSL="${PREFIX}/openssl@1.1"
TCLTK="${PREFIX}/tcl-tk"
PGSQL="${PREFIX}/postgresql@10"
LIBS=('ZLIB' 'BZIP2' 'READLINE' 'SQLITE' 'OPENSSL' 'PGSQL' 'TCLTK')

for LIB in $LIBS; do

	BINDIR="${(P)LIB}/bin"
	if [ -d "${BINDIR}" ]; then
		export PATH="${BINDIR}:$PATH"
	fi

	LIBDIR="${(P)LIB}/lib"
	if [ -d "${LIBDIR}" ]; then
		export LDFLAGS="-L${LIBDIR} $LDFLAGS"
		export DYLD_LIBRARY_PATH="${LIBDIR}:$DYLD_LIBRARY_PATH"
		PKGCFGDIR="${LIBDIR}/pkgconfig"
		if [ -d "${PKGCFGDIR}" ]; then
			export PKG_CONFIG_PATH="${PKGCFGDIR} $PKG_CONFIG_PATH"
		fi
	fi

	INCDIR="${(P)LIB}/include"
	if [ -d "${INCDIR}" ]; then
		export CFLAGS="-I${INCDIR} $CFLAGS" 
	fi

done

export CPPFLAGS="${CFLAGS}"
export CXXFLAGS="${CFLAGS}"


export PYTHON_CONFIGURE_OPTS="--enable-framework"
export PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl) ${PYTHON_CONFIGURE_OPTS}"
export PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I$(brew --prefix tcl-tk)/include' ${PYTHON_CONFIGURE_OPTS}"
export PYTHON_CONFIGURE_OPTS="--with-tcltk-libs='-L$(brew --prefix tcl-tk)/lib -ltcl8.6 -ltk8.6' ${PYTHON_CONFIGURE_OPTS}"


eval "$(pyenv init -)"
# eval "$(pyenv virtualenv-init -)"

Result:

Here's the relevant output from pyenv install 3.8.6:

configure: error: Unexpected output of 'arch' on OSX

Environment

> env|grep 'FLAGS\|PATH'
PATH=/Users/rob/.pyenv/arm64/shims:/opt/homebrew/opt/tcl-tk/bin:/opt/homebrew/opt/openssl@1.1/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/opt/bzip2/bin:/Users/rob/bin:/Users/rob/.pyenv/arm64/bin:/Applications/Sublime Text.app/Contents/SharedSupport/bin:/Users/rob/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
PYTHONPATH=/Users/rob/.pyenv/arm64/shims
CPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
CFLAGS=-I/opt/homebrew/opt/tcl-tk/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/bzip2/include -I/opt/homebrew/opt/zlib/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sasl
LDFLAGS=-L/opt/homebrew/opt/tcl-tk/lib -L/opt/homebrew/opt/openssl@1.1/lib -L/opt/homebrew/opt/sqlite/lib -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/bzip2/lib -L/opt/homebrew/opt/zlib/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib
PKG_CONFIG_PATH=/opt/homebrew/opt/tcl-tk/lib/pkgconfig /opt/homebrew/opt/openssl@1.1/lib/pkgconfig /opt/homebrew/opt/sqlite/lib/pkgconfig /opt/homebrew/opt/readline/lib/pkgconfig /opt/homebrew/opt/zlib/lib/pkgconfig
CPPFLAGS=-I/opt/homebrew/opt/tcl-tk/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/bzip2/include -I/opt/homebrew/opt/zlib/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sasl
CXXFLAGS=-I/opt/homebrew/opt/tcl-tk/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/bzip2/include -I/opt/homebrew/opt/zlib/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sasl```

I know this is related to the compiler not recognizing `arm64` as architecture however the patches I've found don't solve the problem.

[rjmoggach]

It seems I was able to get 3.8.6 installed using a patch I found on a plone site - the ones in other issues here didn't work for me... Here's the code in question:

pyenv install --patch 3.8.6 <<(curl -sSL https://raw.githubusercontent.com/Homebrew/formula-patches/113aa84/python/3.8.3.patch\?full_index\=1)

That's a 3.8.3 patch but clearly it's not different enough to to affect 3.8.6. This does NOT work for 3.9.0 which is the next task.

[rjmoggach]

Here's the relevant patches in the homebrew formula-patches repo:

https://github.com/Homebrew/formula-patches/tree/master/python

[anton-petrov]

EDITED 11/01/2021:
Python 3.9.1 compiles successfully with native homebrew, and latest pyenv installed from github or homebrew packages.

[cclauss]

Can we please get a release of pyenv that includes Python 3.9.1?

It is the first CPython that defaults to 64-bit on Windows and natively supports Apple Silicon ARM CPUs.

[bengartner]

Can we please get a release of pyenv that includes Python 3.9.1?

There is an open ticket for this: #1760

[rjmoggach]

Python 3.9.1 compiles successfully with native homebrew, and latest pyenv installed from github.

@anton-petrov are you saying you had success by installing pyenv from github directly NOT via homebrew?

I'm trying to stay within released packages but happy to try that to verify -

[anton-petrov]

Python 3.9.1 compiles successfully with native homebrew, and latest pyenv installed from github.

@anton-petrov are you saying you had success by installing pyenv from github directly NOT via homebrew?

I'm trying to stay within released packages but happy to try that to verify -

Yes, directly from GitHub, that's right.

[anton-petrov]

It seems like it's time to give the rights to someone else, so the code changes quickly get into the main branch...

[sinwoobang]

Same problem here.

[sinwoobang]

I uninstalled native pyenv and am using only Rosetta-based pyenv.
This is something temporary but got rid of every stress of mine.

[mariusnita]

I just cloned pyenv and pyenv install 3.9.1 fails on M1 MacBook Air with this output:

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.1.tar.xz...
-> https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tar.xz
Installing Python-3.9.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.1 using python-build 1.2.22-12-gbb14f225)

Inspect or clean up the working tree at /var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542
Results logged to /var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542.log

Last 10 log lines:
/var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542 ~/Python-3.9.1
/var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542/Python-3.9.1 /var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542 ~/Python-3.9.1
checking build system type... Invalid configuration `arm64-apple-darwin20.2.0': machine `arm64-apple' not recognized
configure: error: /bin/sh ./config.sub arm64-apple-darwin20.2.0 failed
make: *** No targets specified and no makefile found.  Stop.

If I go into the build directory, I can build it successfully by configuring it like this:

./configure --build=aarch64-apple-darwin20.2.0

I can't tell where the root issue is but there is a discrepancy between Python assuming aarch64 and the system reporting arm64.

[cclauss]

@mariusnita OpenSSL v1.1.1 on M1 discussed in #1744

[mariusnita]

@cclauss My issue was actually related to using GNU coreutils from brew. When I removed GNU coreutils from my $PATH, Python built fine with the builtin macOS tools.

[cclauss]

Is this issue fixed in the current release? https://github.com/pyenv/pyenv/releases

[neilpanchal]

I am getting similar but not exact errors on Apple Silicon:

♦ neil / ➞  pyenv install 3.9.1
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.1.tar.xz...
-> https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tar.xz
Installing Python-3.9.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/3d/8b_gdtmd08q1mtthr1d3fybc0000gn/T/python-build.20210129223654.5823
Results logged to /var/folders/3d/8b_gdtmd08q1mtthr1d3fybc0000gn/T/python-build.20210129223654.5823.log

Last 10 log lines:
  "_libintl_textdomain", referenced from:
      _PyIntl_textdomain in libpython3.9.a(_localemodule.o)
      _PyIntl_textdomain in libpython3.9.a(_localemodule.o)
ld: symbol(s) not found for architecture arm64
ld: symbol(s) not found for architecture arm64
clang: clang: error: error: linker command failed with exit code 1 (use -v to see invocation)
linker command failed with exit code 1 (use -v to see invocation)
make: *** [Programs/_testembed] Error 1
make: *** Waiting for unfinished jobs....
make: *** [python.exe] Error 1