Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python build fails on M1 Apple Silicon with arm64 homebrew #1768

Closed
rjmoggach opened this issue Jan 4, 2021 · 97 comments
Closed

Python build fails on M1 Apple Silicon with arm64 homebrew #1768

rjmoggach opened this issue Jan 4, 2021 · 97 comments

Comments

@rjmoggach
Copy link

rjmoggach commented Jan 4, 2021

Referencing other issues, I've been trying unsuccessfully for a native compile of 3.9.0 on an M1 MacBook Air using the following:

(FYI rosetta works without any issue with this config - goal is to compile both)

.zshrc

# PYENV
ARCH=`arch`
if [[ "${ARCH}"  == "arm64" ]]; then
    export PYENV_ROOT="/Users/rob/.pyenv/arm64"
else
    export PYENV_ROOT="/Users/rob/.pyenv/rosetta"
fi
PYENV_BIN="$PYENV_ROOT/bin"
export PYENV_SHELL=zsh
# export PYENV_ROOT=$(pyenv root)
# export PYENV_VERSION=$(pyenv version-name)
export PYTHONPATH=$PYENV_ROOT/shims


SDK_PATH="$(xcrun --show-sdk-path)"
export CPATH="${SDK_PATH}/usr/include"
export CFLAGS="-I${SDK_PATH}/usr/include/sasl $CFLAGS"
export CFLAGS="-I${SDK_PATH}/usr/include $CFLAGS"
export LDFLAGS="-L${SDK_PATH}/usr/lib $LDFLAGS"

if [[ "${ARCH}"  == "arm64" ]]; then
    PREFIX="/opt/homebrew/opt"
else
    PREFIX="/usr/local/opt"
fi

ZLIB="${PREFIX}/zlib"
BZIP2="${PREFIX}/bzip2"
READLINE="${PREFIX}/readline"
SQLITE="${PREFIX}/sqlite"    
OPENSSL="${PREFIX}/openssl@1.1"
TCLTK="${PREFIX}/tcl-tk"
PGSQL="${PREFIX}/postgresql@10"
LIBS=('ZLIB' 'BZIP2' 'READLINE' 'SQLITE' 'OPENSSL' 'PGSQL' 'TCLTK')

for LIB in $LIBS; do

	BINDIR="${(P)LIB}/bin"
	if [ -d "${BINDIR}" ]; then
		export PATH="${BINDIR}:$PATH"
	fi

	LIBDIR="${(P)LIB}/lib"
	if [ -d "${LIBDIR}" ]; then
		export LDFLAGS="-L${LIBDIR} $LDFLAGS"
		export DYLD_LIBRARY_PATH="${LIBDIR}:$DYLD_LIBRARY_PATH"
		PKGCFGDIR="${LIBDIR}/pkgconfig"
		if [ -d "${PKGCFGDIR}" ]; then
			export PKG_CONFIG_PATH="${PKGCFGDIR} $PKG_CONFIG_PATH"
		fi
	fi

	INCDIR="${(P)LIB}/include"
	if [ -d "${INCDIR}" ]; then
		export CFLAGS="-I${INCDIR} $CFLAGS" 
	fi

done

export CPPFLAGS="${CFLAGS}"
export CXXFLAGS="${CFLAGS}"


export PYTHON_CONFIGURE_OPTS="--enable-framework"
export PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl) ${PYTHON_CONFIGURE_OPTS}"
export PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I$(brew --prefix tcl-tk)/include' ${PYTHON_CONFIGURE_OPTS}"
export PYTHON_CONFIGURE_OPTS="--with-tcltk-libs='-L$(brew --prefix tcl-tk)/lib -ltcl8.6 -ltk8.6' ${PYTHON_CONFIGURE_OPTS}"


eval "$(pyenv init -)"
# eval "$(pyenv virtualenv-init -)"

Result:

Here's the relevant output from pyenv install 3.8.6:

configure: error: Unexpected output of 'arch' on OSX

Environment

> env|grep 'FLAGS\|PATH'
PATH=/Users/rob/.pyenv/arm64/shims:/opt/homebrew/opt/tcl-tk/bin:/opt/homebrew/opt/openssl@1.1/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/opt/bzip2/bin:/Users/rob/bin:/Users/rob/.pyenv/arm64/bin:/Applications/Sublime Text.app/Contents/SharedSupport/bin:/Users/rob/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
PYTHONPATH=/Users/rob/.pyenv/arm64/shims
CPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
CFLAGS=-I/opt/homebrew/opt/tcl-tk/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/bzip2/include -I/opt/homebrew/opt/zlib/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sasl
LDFLAGS=-L/opt/homebrew/opt/tcl-tk/lib -L/opt/homebrew/opt/openssl@1.1/lib -L/opt/homebrew/opt/sqlite/lib -L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/bzip2/lib -L/opt/homebrew/opt/zlib/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib
PKG_CONFIG_PATH=/opt/homebrew/opt/tcl-tk/lib/pkgconfig /opt/homebrew/opt/openssl@1.1/lib/pkgconfig /opt/homebrew/opt/sqlite/lib/pkgconfig /opt/homebrew/opt/readline/lib/pkgconfig /opt/homebrew/opt/zlib/lib/pkgconfig
CPPFLAGS=-I/opt/homebrew/opt/tcl-tk/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/bzip2/include -I/opt/homebrew/opt/zlib/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sasl
CXXFLAGS=-I/opt/homebrew/opt/tcl-tk/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/bzip2/include -I/opt/homebrew/opt/zlib/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sasl```

I know this is related to the compiler not recognizing `arm64` as architecture however the patches I've found don't solve the problem.
@rjmoggach
Copy link
Author

It seems I was able to get 3.8.6 installed using a patch I found on a plone site - the ones in other issues here didn't work for me... Here's the code in question:

pyenv install --patch 3.8.6 <<(curl -sSL https://raw.githubusercontent.com/Homebrew/formula-patches/113aa84/python/3.8.3.patch\?full_index\=1)

That's a 3.8.3 patch but clearly it's not different enough to to affect 3.8.6. This does NOT work for 3.9.0 which is the next task.

@rjmoggach
Copy link
Author

Here's the relevant patches in the homebrew formula-patches repo:

https://github.com/Homebrew/formula-patches/tree/master/python

@anton-petrov
Copy link
Member

anton-petrov commented Jan 4, 2021

EDITED 11/01/2021:
Python 3.9.1 compiles successfully with native homebrew, and latest pyenv installed from github or homebrew packages.

@cclauss
Copy link
Contributor

cclauss commented Jan 4, 2021

Can we please get a release of pyenv that includes Python 3.9.1?

It is the first CPython that defaults to 64-bit on Windows and natively supports Apple Silicon ARM CPUs.

@bengartner
Copy link

bengartner commented Jan 4, 2021

Can we please get a release of pyenv that includes Python 3.9.1?

There is an open ticket for this: #1760

@rjmoggach
Copy link
Author

Python 3.9.1 compiles successfully with native homebrew, and latest pyenv installed from github.

@anton-petrov are you saying you had success by installing pyenv from github directly NOT via homebrew?

I'm trying to stay within released packages but happy to try that to verify -

@anton-petrov
Copy link
Member

Python 3.9.1 compiles successfully with native homebrew, and latest pyenv installed from github.

@anton-petrov are you saying you had success by installing pyenv from github directly NOT via homebrew?

I'm trying to stay within released packages but happy to try that to verify -

Yes, directly from GitHub, that's right.

@anton-petrov
Copy link
Member

It seems like it's time to give the rights to someone else, so the code changes quickly get into the main branch...

@sinwoobang
Copy link

Same problem here.

@sinwoobang
Copy link

I uninstalled native pyenv and am using only Rosetta-based pyenv.
This is something temporary but got rid of every stress of mine.

@mariusnita
Copy link

I just cloned pyenv and pyenv install 3.9.1 fails on M1 MacBook Air with this output:

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.1.tar.xz...
-> https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tar.xz
Installing Python-3.9.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.1 using python-build 1.2.22-12-gbb14f225)

Inspect or clean up the working tree at /var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542
Results logged to /var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542.log

Last 10 log lines:
/var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542 ~/Python-3.9.1
/var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542/Python-3.9.1 /var/folders/87/4qkbk8qx0pngsqr3k054m2640000gn/T/python-build.20210110005845.35542 ~/Python-3.9.1
checking build system type... Invalid configuration `arm64-apple-darwin20.2.0': machine `arm64-apple' not recognized
configure: error: /bin/sh ./config.sub arm64-apple-darwin20.2.0 failed
make: *** No targets specified and no makefile found.  Stop.

If I go into the build directory, I can build it successfully by configuring it like this:

./configure --build=aarch64-apple-darwin20.2.0

I can't tell where the root issue is but there is a discrepancy between Python assuming aarch64 and the system reporting arm64.

@cclauss
Copy link
Contributor

cclauss commented Jan 10, 2021

@mariusnita OpenSSL v1.1.1 on M1 discussed in #1744

@mariusnita
Copy link

@cclauss My issue was actually related to using GNU coreutils from brew. When I removed GNU coreutils from my $PATH, Python built fine with the builtin macOS tools.

@cclauss
Copy link
Contributor

cclauss commented Jan 27, 2021

Is this issue fixed in the current release? https://github.com/pyenv/pyenv/releases

@neilpanchal
Copy link

I am getting similar but not exact errors on Apple Silicon:

♦ neil / ➞  pyenv install 3.9.1
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.1.tar.xz...
-> https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tar.xz
Installing Python-3.9.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/3d/8b_gdtmd08q1mtthr1d3fybc0000gn/T/python-build.20210129223654.5823
Results logged to /var/folders/3d/8b_gdtmd08q1mtthr1d3fybc0000gn/T/python-build.20210129223654.5823.log

Last 10 log lines:
  "_libintl_textdomain", referenced from:
      _PyIntl_textdomain in libpython3.9.a(_localemodule.o)
      _PyIntl_textdomain in libpython3.9.a(_localemodule.o)
ld: symbol(s) not found for architecture arm64
ld: symbol(s) not found for architecture arm64
clang: clang: error: error: linker command failed with exit code 1 (use -v to see invocation)
linker command failed with exit code 1 (use -v to see invocation)
make: *** [Programs/_testembed] Error 1
make: *** Waiting for unfinished jobs....
make: *** [python.exe] Error 1

@cclauss
Copy link
Contributor

cclauss commented Jan 30, 2021

Which version of pyenv? brew upgrade pyenv

If you do get info on your terminal app, is Rosetta turned on or off?

@neilpanchal
Copy link

neilpanchal commented Jan 30, 2021

Which version of pyenv? If you do get info on your terminal app, is Rosetta turned on or off?

@cclauss

pyenv --version
pyenv 1.2.22

I am basically running iTerm + homebrew (/opt/homebrew pointing to current master). iTerm is running on Apple architecture according to Activity Monitor. I tried running get info command, it says no such command. If you meant, get info in the info panel, Cmd+I on iTerm shows nothing about which architecture its running. But hopefully, Activity Monitor is correct:

image

Also brew upgrade shows its latest 😃

brew upgrade pyenv
Warning: pyenv 1.2.22 already installed

@cclauss
Copy link
Contributor

cclauss commented Jan 30, 2021

Can you please post the the info panel, Cmd+I on iTerm

@neilpanchal
Copy link

@cclauss Ah, I see what you mean, is this helpful:

image

@cclauss
Copy link
Contributor

cclauss commented Jan 30, 2021

Cool. Open with Rosetta is not checked which aligns with Activity Monitor.

Can you please try brew upgrade OpenSSL ?

@neilpanchal
Copy link

brew upgrade OpenSSL
Warning: openssl 1.1.1i already installed

@cclauss
Copy link
Contributor

cclauss commented Jan 30, 2021

The 2018 in BUILD FAILED (OS X 11.1 using python-build 20180424) is bugging me.

Please try xcode-select --install

@neilpanchal
Copy link

@cclauss I wasn't able to fix it. Running xcode 12.4. I decided to uninstall pyenv for the moment to move forward since brew install python@3.9 works fine natively on Apple Silicon. Let me know if there is anything I can do to help, my guess is this is an upstream issue. Thank you.

@msanders
Copy link

pyenv install 3.9.1 works successfully for me on M1. The latest versions of previous major releases do not (2.7.18, 3.6.12, 3.7.9, 3.8.7).

$ pyenv --version
pyenv 1.2.22
$ pyenv install 3.9.1
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.1.tar.xz...
-> https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tar.xz
Installing Python-3.9.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.9.1 to /Users/mks/.pyenv/versions/3.9.1

$ pyenv install 3.8.7
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.7.tar.xz...
-> https://www.python.org/ftp/python/3.8.7/Python-3.8.7.tar.xz
Installing Python-3.8.7...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/7k/s1chvddx0kj2gjyblr2l8mh80000gn/T/python-build.20210131121321.78527
Results logged to /var/folders/7k/s1chvddx0kj2gjyblr2l8mh80000gn/T/python-build.20210131121321.78527.log

Last 10 log lines:
checking size of long double... 8
checking size of _Bool... 1
checking size of off_t... 8
checking whether to enable large file support... no
checking size of time_t... 8
checking for pthread_t... yes
checking size of pthread_t... 8
checking size of pthread_key_t... 8
checking whether pthread_key_t is compatible with int... no
configure: error: Unexpected output of 'arch' on OSX

@msanders
Copy link

Applying the latest Homebrew patches fixes the issue for me with 3.8 and 3.7.

$ pyenv install --patch 3.8.7 <<(curl -sSL https://raw.githubusercontent.com/Homebrew/formula-patches/9811be33170a8f31a684fae6955542e31eb4e61e/python/3.8.7.patch)"
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.7.tar.xz...
-> https://www.python.org/ftp/python/3.8.7/Python-3.8.7.tar.xz
Installing Python-3.8.7...
patching file Misc/NEWS.d/next/macOS/2020-05-18-02-43-11.bpo-34956.35IcGF.rst
patching file setup.py
patching file Mac/Tools/pythonw.c
patching file Misc/NEWS.d/next/macOS/2020-06-24-14-21-17.bpo-41101.z9hCsP.rst
patching file setup.py
patching file Doc/library/ctypes.rst
Hunk #1 succeeded at 1594 (offset 15 lines).
patching file Lib/test/test_bytes.py
patching file Lib/test/test_unicode.py
patching file Modules/_ctypes/_ctypes.c
patching file Modules/_ctypes/callproc.c
Hunk #4 succeeded at 1242 (offset 15 lines).
patching file Modules/_ctypes/ctypes.h
patching file Modules/_ctypes/callproc.c
patching file setup.py
patching file 'Misc/NEWS.d/next/Core and Builtins/2020-07-01-01-30-04.bpo-41100.hCrjJJ.rst'
patching file configure
Hunk #1 succeeded at 9289 (offset 25 lines).
patching file configure.ac
Hunk #1 succeeded at 2449 (offset 7 lines).
patching file 'Misc/NEWS.d/next/Core and Builtins/2020-06-30-00-02-42.bpo-41164.zbzWsP.rst'
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.7 to /Users/mks/.pyenv/versions/3.8.7
$ pyenv install --patch 3.7.9 <<(curl -sSL https://raw.githubusercontent.com/Homebrew/formula-patches/9811be33170a8f31a684fae6955542e31eb4e61e/python/3.7.9.patch)"
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.9.tar.xz...
-> https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
Installing Python-3.7.9...
patching file Lib/distutils/spawn.py
patching file Lib/distutils/tests/test_build_ext.py
patching file Lib/test/test_posix.py
patching file setup.py
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.7.9 to /Users/mks/.pyenv/versions/3.7.9

There doesn't seem to be one yet for 3.6 or 2.7.

@grawlinson
Copy link
Contributor

3.9.1 compiles for me with pyenv install 3.9.1.

2.7 reached end of life (EOL) last year on 2020-01-01, 3.6 will reach EOL by the end of this year. They shouldn't be used at all.

@egonbraun
Copy link

In my Macbook Air M1, I was able to install seamlessly 3.9.1 and 3.7.9 however 3.8.6 is also erroring out with configure: error: Unexpected output of 'arch' on OSX. I am also not using Rosetta for iTerm2.

@chandlerc
Copy link

FWIW, I'm hitting the same fundamental issue as @neilpanchal. Have done xcode-install and everything else. Still can't get it to work. Any further steps? I somewhat need pyenv, not just the homebrew installed python.

@gordonje
Copy link

gordonje commented Feb 7, 2021

@anton-petrov I have a fresh Macbook Air M1 with macOS 11.2. Just installed homebrew (which also installed Xcode Command Line tools).

Before I install pyenv via the pyenv-installer, which pre-reqs do I need to install?

From the "Suggested Build Environment" section of pyenv's wiki:

  • Mac OS X:

If you haven't done so, install Xcode Command Line Tools (xcode-select --install) and Homebrew. Then:

# optional, but recommended:
brew install openssl readline sqlite3 xz zlib

But then also from the "Common Build Problems":

  • macOS:
brew install readline xz

I already have:

  • Python 2.7.16 (in /usr/bin/python)
  • openssl (in /usr/bin/openssl)
  • sqlite3 (in /usr/bin/sqlite3)

I don't have:

  • readline
  • zlib

@cclauss
Copy link
Contributor

cclauss commented Feb 7, 2021

No need for pyenv-installer. Instead just brew install pyenv

Why use Python 2 which has been dead for more that a year? Python 3.9.1 is M1 native.

cgorshing added a commit to cgorshing/dotfiles that referenced this issue Nov 21, 2021
This isn't the same github issue I found, but close enough for future reference. Someone in the github issue I previously found mentioned 3.8.10 worked with no problem and was fine on my box.
pyenv/pyenv#1768
@jpmckinney
Copy link

jpmckinney commented Dec 9, 2021

I tried #1768 (comment):

arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
/usr/local/bin/brew install python@3.7

But it results in:

Error: Cannot install in Homebrew on ARM processor in Intel default prefix (/usr/local)!
Please create a new installation in /opt/homebrew using one of the
"Alternative Installs" from:
  https://docs.brew.sh/Installation
You can migrate your previously installed formula list with:
  brew bundle dump

The solution is that there needs to be arch -x86_64 before the brew install, like in #1768 (comment). The environment variables should also be set.

arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
eval "$(/usr/local/bin/brew shellenv)"
which brew  # should return /usr/local/bin/brew
arch -x86_64 brew install python@3.7

That said, I prefer the solution in #1768 (comment) to install Python versions via pyenv. So far I'm able to avoid the "ModuleNotFoundError: No module named '_ctypes'" error by installing 3.7 via arch -x86_64 as described in that comment. However, I'm still not able to install 3.6, as I discuss in #1184 (comment)

@scottstanfield
Copy link

@cclauss My issue was actually related to using GNU coreutils from brew. When I removed GNU coreutils from my $PATH, Python built fine with the builtin macOS tools.

I think the offending GNU util is uname. I kept all the GNU coreutils in my path but simply removed GNU uname and was able to pyenv install 3.9.1 finally.

where uname   # should show coreutils first, followed by /usr/bin/uname
rm $(which uname)

@cclauss
Copy link
Contributor

cclauss commented Jan 19, 2022

On an M1 Mac, brew --prefix should point to /opt/homebrew, not to /usr/local as it would on an Intel Mac.

@TaiSHiNet
Copy link

Can confirm that moving /opt/homebrew/opt/coreutils/libexec/gnubin/uname made it work for me as well

@cclauss
Copy link
Contributor

cclauss commented Mar 8, 2022

What does brew --prefix print?

@TaiSHiNet
Copy link

TaiSHiNet commented Mar 8, 2022

/opt/homebrew as it should on a Silicon Mac, but if you installed coreutils and set the path to point to /opt/homebrew/bin (or using the gnu-utils plugin for oh-my-zsh) it'll show like this:

/opt/homebrew/bin/guname
/usr/bin/uname

A very quick-n-dirty workaround, without having to remove a binary (that will get reinstalled next time you update) is putting the following on your .zshrc or .bashrc:

alias uname=/usr/bin/uname

It'll show the following afterwards:

uname: aliased to /usr/bin/uname
/opt/homebrew/bin/guname
/usr/bin/uname

@habib-rahman-cloud
Copy link

Hey guys anyone else having issues with Model:MacBook Pro (16-inch, 2021)

getting everything I need up and running.

For instance, I spent pretty much all day yesterday trying to fix the following error when trying to run unit tests on one of our GitHub repos (pyenv installed version of Python 3.8.12):

Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named '_ctypes'

@dontirun
Copy link

dontirun commented Apr 21, 2022

I successfully installed Python 3.6 on an Apple M1 MacBook Pro running Monterey using the following setup. There is probably some things in here that can be removed/refined... but it worked for me!

#Install Rosetta
/usr/sbin/softwareupdate --install-rosetta --agree-to-license

# Install x86_64 brew
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Set up x86_64 homebrew and pyenv and temporarily set aliases
alias brew86="arch -x86_64 /usr/local/bin/brew"
alias pyenv86="arch -x86_64 pyenv"

# Install required packages and flags for building this particular python version through emulation
brew86 install pyenv gcc libffi gettext
export CPPFLAGS="-I$(brew86 --prefix libffi)/include -I$(brew86 --prefix openssl)/include -I$(brew86 --prefix readline)/lib"
export CFLAGS="-I$(brew86 --prefix openssl)/include -I$(brew86 --prefix bzip2)/include -I$(brew86 --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include -Wno-implicit-function-declaration" 
export LDFLAGS="-L$(brew86 --prefix openssl)/lib -L$(brew86 --prefix readline)/lib -L$(brew86 --prefix zlib)/lib -L$(brew86 --prefix bzip2)/lib -L$(brew86 --prefix gettext)/lib -L$(brew86 --prefix libffi)/lib"

# Providing an incorrect openssl version forces a proper openssl version to be downloaded and linked during the build
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0

# Install Python 3.6
pyenv86 install --patch 3.6.15 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.15/Python-3.6.15/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

Note, the build succeeds but gives the following warning

WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?

running pyenv versions shows that 3.6.15 can be used normally by the system

@Anveshrithaa
Copy link

I successfully installed Python 3.6 on an Apple M1 MacBook Pro running Monterey using the following setup. There is probably some things in here that can be removed/refined... but it worked for me!

#Install Rosetta
/usr/sbin/softwareupdate --install-rosetta --agree-to-license

# Install x86_64 brew
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Set up x86_64 homebrew and pyenv and temporarily set aliases
alias brew86="arch -x86_64 /usr/local/bin/brew"
alias pyenv86="arch -x86_64 pyenv"

# Install required packages and flags for building this particular python version through emulation
brew86 install pyenv gcc libffi gettext
export CPPFLAGS="-I$(brew86 --prefix libffi)/include -I$(brew86 --prefix openssl)/include -I$(brew86 --prefix readline)/lib"
export CFLAGS="-I$(brew86 --prefix openssl)/include -I$(brew86 --prefix bzip2)/include -I$(brew86 --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include -Wno-implicit-function-declaration" 
export LDFLAGS="-L$(brew86 --prefix openssl)/lib -L$(brew86 --prefix readline)/lib -L$(brew86 --prefix zlib)/lib -L$(brew86 --prefix bzip2)/lib -L$(brew86 --prefix gettext)/lib -L$(brew86 --prefix libffi)/lib"

# Providing an incorrect openssl version forces a proper openssl version to be downloaded and linked during the build
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0

# Install Python 3.6
pyenv86 install --patch 3.6.15 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.15/Python-3.6.15/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

Note, the build succeeds but gives the following warning

WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?

running pyenv versions shows that 3.6.15 can be used normally by the system

Thank you so much! It saved my day.

@dja-tel
Copy link

dja-tel commented May 5, 2022

Hi @Anveshrithaa and @dontirun

Would you mind checking the following?
1 - Install the snappy library on the system and python@3.6 with brew86
2- Create a virtual environment with the pyton3.6 interpreter.
2- Install in the virtual environment python-snappy==0.5.4 with pip
4- Enter a python console in that environment and try to import snappy
5- try to use a snappy functionality, e.g. -> snappy.compress(file)

Thank you very much

@jesseinit
Copy link

I'm getting this error whenever I try to install any python version.

/var/folders/cv/3c0q0prj1ls8611s56lcm44r0000gp/T/python-build.20220506163951.73150 ~
/var/folders/cv/3c0q0prj1ls8611s56lcm44r0000gp/T/python-build.20220506163951.73150/Python-3.9.1 /var/folders/cv/3c0q0prj1ls8611s56lcm44r0000gp/T/python-build.20220506163951.73150 ~
checking build system type... arm-apple-darwin21.2.0
checking host system type... arm-apple-darwin21.2.0
checking for python3.9... no
checking for python3... python3
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang accepts -g... yes
checking for clang option to accept ISO C89... none needed
checking how to run the C preprocessor... clang -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for a sed that does not truncate output... /usr/bin/sed
checking for --with-cxx-main=<compiler>... no
checking for clang++... no
configure:

  By default, distutils will build C++ extension modules with "clang++".
  If this is not intended, then set CXX on the configure command line.
  
checking for the platform triplet based on compiler characteristics... darwin
configure: error: internal configure error for the platform triplet, please file a bug report
make: *** No targets specified and no makefile found.  Stop.

How do I fix it? I'm on Monterey v12.1 on the M1 Pro.

@cclauss
Copy link
Contributor

cclauss commented May 6, 2022

Is that macOS 12.1 or 12.3?!?

% sw_vers

ProductName:	macOS
ProductVersion:	12.3.1
BuildVersion:	21E258

@jesseinit
Copy link

Is that macOS 12.1 or 12.3?!?

% sw_vers

ProductName:	macOS
ProductVersion:	12.3.1
BuildVersion:	21E258

ProductName: macOS
ProductVersion: 12.1
BuildVersion: 21C52

@cclauss
Copy link
Contributor

cclauss commented May 7, 2022

https://stackoverflow.com/questions/71577626/issues-installing-python-3-8-10-on-macos-12-3-monterey

@vishal-wadhwa
Copy link

vishal-wadhwa commented Jan 2, 2023

Fix for 3.8.7 -

Okay. So within a mix of multiple issues. I was facing the following errors -

  1. When running pyenv install 3.8.7
Last 10 log lines:
checking size of _Bool... 1
checking size of off_t... 8
checking whether to enable large file support... no
checking size of time_t... 8
checking for pthread_t... yes
checking size of pthread_t... 8
checking size of pthread_key_t... 8
checking whether pthread_key_t is compatible with int... no
configure: error: Unexpected output of 'arch' on OSX
make: *** No targets specified and no makefile found.  Stop.
  1. When running pyenv install --patch 3.8.7 <<(curl -sSL https://raw.githubusercontent.com/Homebrew/formula-patches/9811be33170a8f31a684fae6955542e31eb4e61e/python/3.8.7.patch)
Last 10 log lines:
checking for --with-cxx-main=<compiler>... no
checking for clang++... no
configure:

  By default, distutils will build C++ extension modules with "clang++".
  If this is not intended, then set CXX on the configure command line.

checking for the platform triplet based on compiler characteristics... darwin
configure: error: internal configure error for the platform triplet, please file a bug report
make: *** No targets specified and no makefile found.  Stop.

This was despite the fact that I already have the latest pyenv.

Finally, the fix was to apply both the patches like this
Setup: Mac M1 (OS 12.6.2 Monterey)

$ curl -sSL 'https://raw.githubusercontent.com/Homebrew/formula-patches/master/python/3.8.7.patch' > 3.8.7.patch

$ cat /opt/homebrew/Cellar/pyenv/2.3.9/plugins/python-build/share/python-build/patches/3.8.7/Python-3.8.7/0001-bpo-45405-Prevent-internal-configure-error-when-runn.patch 3.8.7.patch | pyenv install --patch 3.8.7

@l0g1c-80m8
Copy link

l0g1c-80m8 commented Jan 25, 2023

I successfully installed Python 3.6 on an Apple M1 MacBook Pro running Monterey using the following setup. There is probably some things in here that can be removed/refined... but it worked for me!

#Install Rosetta
/usr/sbin/softwareupdate --install-rosetta --agree-to-license

# Install x86_64 brew
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Set up x86_64 homebrew and pyenv and temporarily set aliases
alias brew86="arch -x86_64 /usr/local/bin/brew"
alias pyenv86="arch -x86_64 pyenv"

# Install required packages and flags for building this particular python version through emulation
brew86 install pyenv gcc libffi gettext
export CPPFLAGS="-I$(brew86 --prefix libffi)/include -I$(brew86 --prefix openssl)/include -I$(brew86 --prefix readline)/lib"
export CFLAGS="-I$(brew86 --prefix openssl)/include -I$(brew86 --prefix bzip2)/include -I$(brew86 --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include -Wno-implicit-function-declaration" 
export LDFLAGS="-L$(brew86 --prefix openssl)/lib -L$(brew86 --prefix readline)/lib -L$(brew86 --prefix zlib)/lib -L$(brew86 --prefix bzip2)/lib -L$(brew86 --prefix gettext)/lib -L$(brew86 --prefix libffi)/lib"

# Providing an incorrect openssl version forces a proper openssl version to be downloaded and linked during the build
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0

# Install Python 3.6
pyenv86 install --patch 3.6.15 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.15/Python-3.6.15/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

Note, the build succeeds but gives the following warning

WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?

running pyenv versions shows that 3.6.15 can be used normally by the system

This alone didn't work for me while I was trying to install python 3.6.15 . I needed to do take the following additional steps before running the final installation command:

  • Add the alias lines to the .zshrc file so that you can freely use pyenv86 without having to alias them every time.
alias brew86="arch -x86_64 /usr/local/bin/brew"
alias pyenv86="arch -x86_64 pyenv"
  • Force the terminal to open with Rosetta. For this, right-click on the terminal app and click on get info, followed by checking the open with Rosetta 2 option. If you don't want your Terminal to open with Rosetta 2 always, you can make a copy of the Terminal app and only check the Rosetta 2 option on one of them.

After this, I ran the final installation command. There was an error with the installation saying I had a few directories missing:


Inspect or clean up the working tree at /var/folders/7_/6r9yw5rd783gt24frfyk39j80000gn/T/python-build.20230124183357.76209
Results logged to /var/folders/7_/6r9yw5rd783gt24frfyk39j80000gn/T/python-build.20230124183357.76209.log

Last 10 log lines:
ld: warning: directory not found for option '-L/usr/local/opt/openssl@3/lib'
ld: warning: directory not found for option '-L/usr/local/opt/readline/lib'
ld: warning: directory not found for option '-L/usr/local/opt/zlib/lib'
ld: warning: directory not found for option '-L/usr/local/opt/bzip2/lib'
ld: warning: directory not found for option '-L/usr/local/opt/gettext/lib'
ld: warning: directory not found for option '-L/usr/local/opt/libffi/lib'
ld: library not found for -lSystem
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [libcrypto.dylib] Error 1
make: *** [all] Error 2

To address this just install the missing packages in your brew x86 directory using the following command:

brew86 install readline zlib bzip2 gettext libffi

You can add or remove the packages as observed from the error message. Finally running the final installation command solved the issue:

pyenv86 install --patch 3.6.15 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.15/Python-3.6.15/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

@cclauss
Copy link
Contributor

cclauss commented Jan 25, 2023

@l0g1c-80m8 I would highly recommend that you run pyupgrade on your codebase and make whatever changes are required to run it on a supported version of Python. Python 3.6’s end-of-life was more than a year ago. In that time, all supported versions of Python have received high profile security updates but Python 3.6 and older have not.

There is also the issue that Python 3.9.1 was the first release to support Apple Silicon so while earlier versions may work, this is not supported by the Python core team. If someone knows of further backporting of this support then I would love to hear about it but it certainly would not cover Python 3.6.

@l0g1c-80m8
Copy link

@cclauss appreciate your suggestions and I completely agree with them. In this case however, I need python 3.6 for my assignments as the environment that will run my submissions is running python 3.6.

@omkarsk98
Copy link

omkarsk98 commented Feb 2, 2023

I successfully installed Python 3.6 on an Apple M1 MacBook Pro running Monterey using the following setup. There is probably some things in here that can be removed/refined... but it worked for me!

#Install Rosetta
/usr/sbin/softwareupdate --install-rosetta --agree-to-license

# Install x86_64 brew
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Set up x86_64 homebrew and pyenv and temporarily set aliases
alias brew86="arch -x86_64 /usr/local/bin/brew"
alias pyenv86="arch -x86_64 pyenv"

# Install required packages and flags for building this particular python version through emulation
brew86 install pyenv gcc libffi gettext
export CPPFLAGS="-I$(brew86 --prefix libffi)/include -I$(brew86 --prefix openssl)/include -I$(brew86 --prefix readline)/lib"
export CFLAGS="-I$(brew86 --prefix openssl)/include -I$(brew86 --prefix bzip2)/include -I$(brew86 --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include -Wno-implicit-function-declaration" 
export LDFLAGS="-L$(brew86 --prefix openssl)/lib -L$(brew86 --prefix readline)/lib -L$(brew86 --prefix zlib)/lib -L$(brew86 --prefix bzip2)/lib -L$(brew86 --prefix gettext)/lib -L$(brew86 --prefix libffi)/lib"

# Providing an incorrect openssl version forces a proper openssl version to be downloaded and linked during the build
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0

# Install Python 3.6
pyenv86 install --patch 3.6.15 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.15/Python-3.6.15/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

Note, the build succeeds but gives the following warning

WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?

running pyenv versions shows that 3.6.15 can be used normally by the system

This alone didn't work for me while I was trying to install python 3.6.15 . I needed to do take the following additional steps before running the final installation command:

* Add the alias lines to the `.zshrc` file so that you can freely use `pyenv86` without having to alias them every time.
alias brew86="arch -x86_64 /usr/local/bin/brew"
alias pyenv86="arch -x86_64 pyenv"
* Force the terminal to open with Rosetta. For this, right-click on the terminal app and click on get info, followed by checking the open with Rosetta 2 option. If you don't want your Terminal to open with Rosetta 2 always, you can make a copy of the Terminal app and only check the Rosetta 2 option on one of them.

After this, I ran the final installation command. There was an error with the installation saying I had a few directories missing:


Inspect or clean up the working tree at /var/folders/7_/6r9yw5rd783gt24frfyk39j80000gn/T/python-build.20230124183357.76209
Results logged to /var/folders/7_/6r9yw5rd783gt24frfyk39j80000gn/T/python-build.20230124183357.76209.log

Last 10 log lines:
ld: warning: directory not found for option '-L/usr/local/opt/openssl@3/lib'
ld: warning: directory not found for option '-L/usr/local/opt/readline/lib'
ld: warning: directory not found for option '-L/usr/local/opt/zlib/lib'
ld: warning: directory not found for option '-L/usr/local/opt/bzip2/lib'
ld: warning: directory not found for option '-L/usr/local/opt/gettext/lib'
ld: warning: directory not found for option '-L/usr/local/opt/libffi/lib'
ld: library not found for -lSystem
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [libcrypto.dylib] Error 1
make: *** [all] Error 2

To address this just install the missing packages in your brew x86 directory using the following command:

brew86 install readline zlib bzip2 gettext libffi

You can add or remove the packages as observed from the error message. Finally running the final installation command solved the issue:

pyenv86 install --patch 3.6.15 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.15/Python-3.6.15/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

Getting error as

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/kullya/.pyenv/versions/3.6.15/lib/python3.6/ssl.py", line 101, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?

@Tapudp
Copy link

Tapudp commented Feb 8, 2023

I successfully installed Python 3.6 on an Apple M1 MacBook Pro running Monterey using the following setup. There is probably some things in here that can be removed/refined... but it worked for me!

#Install Rosetta
/usr/sbin/softwareupdate --install-rosetta --agree-to-license

# Install x86_64 brew
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Set up x86_64 homebrew and pyenv and temporarily set aliases
alias brew86="arch -x86_64 /usr/local/bin/brew"
alias pyenv86="arch -x86_64 pyenv"

# Install required packages and flags for building this particular python version through emulation
brew86 install pyenv gcc libffi gettext
export CPPFLAGS="-I$(brew86 --prefix libffi)/include -I$(brew86 --prefix openssl)/include -I$(brew86 --prefix readline)/lib"
export CFLAGS="-I$(brew86 --prefix openssl)/include -I$(brew86 --prefix bzip2)/include -I$(brew86 --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include -Wno-implicit-function-declaration" 
export LDFLAGS="-L$(brew86 --prefix openssl)/lib -L$(brew86 --prefix readline)/lib -L$(brew86 --prefix zlib)/lib -L$(brew86 --prefix bzip2)/lib -L$(brew86 --prefix gettext)/lib -L$(brew86 --prefix libffi)/lib"

# Providing an incorrect openssl version forces a proper openssl version to be downloaded and linked during the build
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0

# Install Python 3.6
pyenv86 install --patch 3.6.15 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.15/Python-3.6.15/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

Note, the build succeeds but gives the following warning

WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?

running pyenv versions shows that 3.6.15 can be used normally by the system

Thank you so much! It saved my day.

I was facing this issue, while installing Python version 3.9/3.10 on my m1-mac with MacOS Ventura 13.1, and it kept throwing the same error about not being able to find module _ssl from OpenSSL. The system already had

$ openssl version 
LibreSSL 3.3.6

but this one following line from @dontirun 's message worked like a charm! Thank you! 🙇🏻‍♀️🙇♤

# Providing an incorrect openssl version forces a proper openssl version to be downloaded and linked during the build
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0

@l0g1c-80m8
Copy link

l0g1c-80m8 commented Feb 8, 2023

@omkarsk98 Pardon my late response to your issue.
Based on the above comment, it looks like you don't have any OpenSSL module installed or is not exported. I'd recommend installing the version used above and exporting the path and trying the installation again.

Also note that in the original workaround, we export an incorrect OpenSSL version to force the download of the correct version.

# Providing an incorrect openssl version forces a proper openssl version to be downloaded and linked during the build
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0

Since, you will be doing that manually now, this line will no longer be needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests