Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to build Python on macOS Big Sur with Xcode 12 beta #1643

Closed
Sjors opened this issue Jun 26, 2020 · 153 comments · Fixed by #2292
Closed

Unable to build Python on macOS Big Sur with Xcode 12 beta #1643

Sjors opened this issue Jun 26, 2020 · 153 comments · Fixed by #2292

Comments

@Sjors
Copy link

Sjors commented Jun 26, 2020

I installed macOS Big Sur beta 1, Xcode 12.0 beta (12A6159) on an old Mac to see what breaks. I'm able to install PyEnv via Homebrew, but I'm not able to build any Python version.

Apple clang version 12.0.0

% pyenv install 3.8.3
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 10.16 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/sl/5j1zvmtj0gb8qpgkklkf38dh0000gr/T/python-build.20200626170507.14625
Results logged to /var/folders/sl/5j1zvmtj0gb8qpgkklkf38dh0000gr/T/python-build.20200626170507.14625.log

Last 10 log lines:
./Modules/posixmodule.c:9141:12: note: forward declaration of 'struct sf_hdtr'
    struct sf_hdtr sf;
           ^
./Modules/posixmodule.c:9221:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
2 errors generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
1 warning generated.

pyenv_install.log

Note that MacOSX.sdk is symlinked to MacOSX10.16.sdk. Pointing the symlink to MacOSX10.15.sdk doesn't help.

I didn't try switching to Xcode 11 yet, though that will probably help, if it's indeed a clang problem.

Downloading and installing Python 3.8.3 from the website works fine (python3 executable).

@bjackson
Copy link

It doesn't build with Xcode 11.5 either.

@Sjors
Copy link
Author

Sjors commented Jun 27, 2020

@bjackson on macOS Catalina or the Big Sur beta?

@bjackson
Copy link

Big Sur beta

@bryanmobrien
Copy link

bryanmobrien commented Jun 28, 2020

Same here - Big Sur beta, Xcode 12.

pyenv install 3.8.3 fails as above.

Tried setting SDKROOT to 10.15 and 10.16, neither was successful.

./configure && make are successful (with cpython source from python.og) but the readline module does not get built. Wonder if this might have to do with readline?? (Def not an expert ;-)

Will update if I get it working.

@bryanmobrien
Copy link

So, still struggling with pyenv on BIg Sur (obv not a pressing issue for me). However, to add to this thread, I attempted to install python from homebrew. With:

brew install python --HEAD

I got the following error:

==> Pouring pkg-config-0.29.2_3.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/pkg-config/0.29.2_3: 11 files, 623.7KB
==> Installing python --HEAD
Error: An exception occurred within a child process:
  Utils::InreplaceError: inreplace failed
setup.py:
  expected replacement of "do_readline = self.compiler.find_library_file(lib_dirs, 'readline')" with "do_readline = '/usr/local/opt/readline/lib/libhistory.dylib'"

I tried to install vanilla python from homebrew (sans --HEAD) and homebrew python installed without error.

==> /usr/local/Cellar/python/3.7.7/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/
==> /usr/local/Cellar/python/3.7.7/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/
==> /usr/local/Cellar/python/3.7.7/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python/3.7.7: 4,006 files, 61.2MB

I really think (since homebrew install with --HEAD option and pyenv install both try to install from source, and both fail with readline error messages) there might be an issue with the readline module and Big Sur. I don't have the technical skills to absolutely determine this, but it is fishy. Wondering, @Sjors , if you are able to install python via homebrew (first attempt with --HEAD) and then attempt without --HEAD.

@Sjors
Copy link
Author

Sjors commented Jul 1, 2020

I tried the homebrew binary back then and it worked. IIRC having Homebrew build from source resulted in the same problem as PyEnv had. We should probably keep an eye on Homebrew/brew#7857

@palmerc
Copy link

palmerc commented Jul 1, 2020

I added #undef _POSIX_C_SOURCE right before #include <sys/socket.h> in Modules/posixmodule.c to get things compiling.

@bryanmobrien
Copy link

@palmerc , I'm showing my ignorance here. How to you add #undef to posixmodule.c if pyenv install downloads and compiles python in one step? I'm missing how to interrupt pyenv before it starts compiling the downloaded source code.

@palmerc
Copy link

palmerc commented Jul 2, 2020

@palmerc , I'm showing my ignorance here. How to you add #undef to posixmodule.c if pyenv install downloads and compiles python in one step? I'm missing how to interrupt pyenv before it starts compiling the downloaded source code.

No ignorance - this is obviously me hacking until I could get it to compile. Yes, you'll have to apply a patch to the source after it is installed. I'm not suggesting this is a legitimate solution, simply that the problem is related to the _POSIX_C_SOURCE #define.

What I did to get python on my beta build machine was to build it on a Catalina machine with Xcode 11 and then copy the whole .pyenv directory over (symlinks and all).

@PritamDutt
Copy link

PritamDutt commented Jul 8, 2020

I am also facing similar issue with posixmodule
I have tried following options unsuccessfully :(
Please note though I want to install Python 3.8.0, I have tried all 3.8 releases with same result

  1. Setting SDKROOT and MACOSX_DEPLOYMENT_TARGET

SDKROOT=/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
MACOSX_DEPLOYMENT_TARGET=10.16
PYTHON_CONFIGURE_OPTS="--enable-framework"
pyenv install 3.8.0

  1. Setting CFLAGS AND LDFLAGS

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib" pyenv install 3.8.0

% sw_vers 
ProductName:    macOS
ProductVersion: 10.16
BuildVersion:   20A4300b
% clang --version
Apple clang version 12.0.0 (clang-1200.0.22.7)
Target: x86_64-apple-darwin20.0.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
% pyenv install 3.8.0
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Installing Python-3.8.0...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 10.16 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/xw/srzcqq0j1lqfsm1f23wqp6380000gn/T/python-build.20200708180922.77980
Results logged to /var/folders/xw/srzcqq0j1lqfsm1f23wqp6380000gn/T/python-build.20200708180922.77980.log

Last 10 log lines:
           ^
./Modules/posixmodule.c:9084:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/_sre.c -o Modules/_sre.o
2 errors generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
1 warning generated.

pyenv_3_8_0_install.log

@PritamDutt
Copy link

I would like to share that ... I tried install Python using Homebrew it installed without any errors .. only downside in my case was that it I did not get 3.8.0..

Sharing the output of my session

% brew install python@3.8
Updating Homebrew...
==> Auto-updated Homebrew!
Updated Homebrew from b763d29df to d5123743b.
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
omake
==> Updated Formulae
exploitdb                                                   fastlane                                                    semgrep                                                     wildfly-as
==> Updated Casks
alt-tab                                                                         mos                                                                             qt-creator

Warning: You are using macOS 10.16.
We do not provide support for this pre-release version.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Discourse, Twitter or IRC. You are responsible for resolving any issues you
experience while you are running this pre-release version.

Warning: Building python@3.8 from source:
  The bottle needs the Apple Command Line Tools to be installed.
  You can install them, if desired, with:
    xcode-select --install

==> Downloading https://github.com/python/cpython/commit/8ea6353.patch?full_index=1
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/df/ed/bea598a87a8f7e21ac5bbf464102077c7102557c07db9ff4e207bd9f7806/setuptools-46.0.0.zip
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/75/28/521c6dc7fef23a68368efefdcd682f5b3d1d58c2b90b06dc1d0b805b51ae/wheel-0.34.2.tar.gz
######################################################################## 100.0%
==> Downloading https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
######################################################################## 100.0%
==> Patching
==> Applying 8ea6353.patch
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
==> ./configure --prefix=/usr/local/Cellar/python@3.8/3.8.3_1 --enable-ipv6 --datarootdir=/usr/local/Cellar/python@3.8/3.8.3_1/share --datadir=/usr/local/Cellar/python@3.8/3.8.3_1/share --enable-framework=/usr/local/Cellar/python@3.8/3.8.
==> make
==> make install PYTHONAPPSDIR=/usr/local/Cellar/python@3.8/3.8.3_1
==> make frameworkinstallextras PYTHONAPPSDIR=/usr/local/Cellar/python@3.8/3.8.3_1/share/python@3.8
==> /usr/local/Cellar/python@3.8/3.8.3_1/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python@3.8/3.8.3_1/bin --install-lib=/usr/local/lib/python3.8/site-packages --single-version-exte
==> /usr/local/Cellar/python@3.8/3.8.3_1/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python@3.8/3.8.3_1/bin --install-lib=/usr/local/lib/python3.8/site-packages --single-version-exte
==> /usr/local/Cellar/python@3.8/3.8.3_1/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python@3.8/3.8.3_1/bin --install-lib=/usr/local/lib/python3.8/site-packages --single-version-exte
==> Caveats
Python has been installed as
  /usr/local/opt/python@3.8/bin/python3

You can install Python packages with
  /usr/local/opt/python@3.8/bin/pip3 install <package>
They will install into the site-package directory
  /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python@3.8/3.8.3_1: 8,827 files, 125.1MB, built in 3 minutes 1 second
% brew info python@3.8
python@3.8: stable 3.8.3 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@3.8/3.8.3_1 (8,827 files, 125.1MB) *
  Built from source on 2020-07-08 at 20:59:27
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.8.rb
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, openssl@1.1 ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
  /usr/local/opt/python@3.8/bin/python3

You can install Python packages with
  /usr/local/opt/python@3.8/bin/pip3 install <package>
They will install into the site-package directory
  /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 372,347 (30 days), 1,149,638 (90 days), 1,723,074 (365 days)
install-on-request: 12,656 (30 days), 34,243 (90 days), 57,756 (365 days)
build-error: 0 (30 days)
% /usr/local/opt/python@3.8/bin/python3
Python 3.8.3 (default, Jul  8 2020, 20:58:19)
[Clang 12.0.0 (clang-1200.0.22.7)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

@PritamDutt
Copy link

PritamDutt commented Jul 8, 2020

Thank you HomeBrew team for all the hard work you did to find the patch that python team is brewing. I am now able to install Python 3.8.0 using pyenv, and this is how I did it.

I have tested installing versions 3.8.0 and 3.8.3. As

% CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.0 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Installing Python-3.8.0...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3394 (offset -32 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.0 to /Users/pritam/.pyenv/versions/3.8.0

% CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.3 to /Users/pritam/.pyenv/versions/3.8.3

% pyenv versions
* system (set by /Users/pritam/.pyenv/version)
  3.6.8
  3.6.9
  3.8.0
  3.8.3

@bryanmobrien
Copy link

@PritamDutt , thank you.
Trying your suggestions gives me the following warning (for 3.8.0):

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.0 to /Users/bobrien/.pyenv/versions/3.8.0

When I tried 3.8.3, again, pasting your script above:

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.3 to /Users/bobrien/.pyenv/versions/3.8.3

Checked homebrew, and it is installed, but not linked.

brew link bzip2

Had no effect. I removed 3.8.3 and reattempted

pyenv install 3.8.3

and got the same warning.

At least, I have python installed on Big Sur via pyenv.

@Al-Ferguson
Copy link

Al-Ferguson commented Jul 14, 2020

It took a while to get "pyenv install ..." to work under macOS Big Sur with Xcode 12 beta, but I have now build 3.8.3, 3.8-dev, 3.9.0b4 and 3.9-dev. I have also been able to build python via home-brew (i.e. brew install python@3.8)

Here is the relevant sections I have added to my ~/.zshrc and ~/.bash_profile:


# HOMEBREW Environment Variables
export HOMEBREW_INSTALL_CLEANUP=TRUE
export HOMEBREW_PREFIX=$(brew --prefix)

export PATH="$HOMEBREW_PREFIX/opt/tcl-tk/bin:$PATH"

# Use PyEnv to set Python Environment
export PYENV_SHELL=zsh
export PYENV_ROOT=$(pyenv root)
export PYENV_VERSION=$(pyenv version-name)
export PYTHONPATH=$PYENV_ROOT/shims

# PyEnv & HOMEBREW Build variables
PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I$HOMEBREW_PREFIX/opt/tcl-tk/include'"
export PYTHON_CONFIGURE_OPTS="$PYTHON_CONFIGURE_OPTS --with-tcltk-libs='-L$HOMEBREW_PREFIX/opt/tcl-tk/lib -ltcl8.6 -ltk8.6'"
export CFLAGS="-O2 -I$HOMEBREW_PREFIX/include"
CPPFLAGS="-I$HOMEBREW_PREFIX/opt/sqlite/include -I$HOMEBREW_PREFIX/opt/tcl-tk/include"
CPPFLAGS="$CPPFLAGS -I$HOMEBREW_PREFIX/opt/zlib/include"
CPPFLAGS="$CPPFLAGS -I$HOMEBREW_PREFIX/opt/bzip2/include"
export CPPFLAGS="$CPPFLAGS -I$HOMEBREW_PREFIX/opt/openssl@1.1/include"
LDFLAGS="-L$HOMEBREW_PREFIX/opt/sqlite/lib -L$HOMEBREW_PREFIX/opt/tcl-tk/lib"
LDFLAGS="$LDFLAGS -L$HOMEBREW_PREFIX/opt/zlib/lib"
LDFLAGS="$LDFLAGS -L$HOMEBREW_PREFIX/opt/bzip2/lib"
export LDFLAGS="$LDFLAGS -L$HOMEBREW_PREFIX/opt/openssl@1.1/lib -L$HOMEBREW_PREFIX/opt/readline/lib"
PKG_CONFIG_PATH="$HOMEBREW_PREFIX/opt/sqlite/lib/pkgconfig:$HOMEBREW_PREFIX/opt/tcl-tk/lib/pkgconfig"
PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOMEBREW_PREFIX/opt/zlib/lib/pkgconfig"
PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOMEBREW_PREFIX/opt/bzip2/lib/pkgconfig"
PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOMEBREW_PREFIX/opt/openssl@1.1/lib/pkgconfig"
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$PYENV_ROOT/versions/$PYENV_VERSION/lib/pkgconfig"

if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi

@andytriboletti
Copy link

andytriboletti commented Jul 15, 2020

I tried building using those inputs to my .zshrc and still got an error:

andytriboletti@andys-mac ~ % CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/1r/bdbwv9r53ls_80hpk1t6fvlm0000gn/T/python-build.20200715094722.77820
Results logged to /var/folders/1r/bdbwv9r53ls_80hpk1t6fvlm0000gn/T/python-build.20200715094722.77820.log

Last 10 log lines:
checking size of _Bool... 1
checking size of off_t... 8
checking whether to enable large file support... no
checking size of time_t... 8
checking for pthread_t... yes
checking size of pthread_t... 8
checking size of pthread_key_t... 8
checking whether pthread_key_t is compatible with int... no
configure: error: Unexpected output of 'arch' on OSX
make: *** No targets specified and no makefile found.  Stop.
andytriboletti@andys-mac ~ % arch
arm64

@PritamDutt
Copy link

I compiled it on i7 - i386 architecture.

Does brew install work for you?

@PritamDutt
Copy link

@PritamDutt , thank you.
Trying your suggestions gives me the following warning (for 3.8.0):

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.0 to /Users/bobrien/.pyenv/versions/3.8.0

When I tried 3.8.3, again, pasting your script above:

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.3 to /Users/bobrien/.pyenv/versions/3.8.3

Checked homebrew, and it is installed, but not linked.

brew link bzip2

Had no effect. I removed 3.8.3 and reattempted

pyenv install 3.8.3

and got the same warning.

At least, I have python installed on Big Sur via pyenv.

Here is the output of brew info bzip2 from my system...

Hope this can help!

% brew info bzip2
bzip2: stable 1.0.8 (bottled) [keg-only]
Freely available high-quality data compressor
https://sourceware.org/bzip2/
/usr/local/Cellar/bzip2/1.0.8 (26 files, 400.6KB)
  Poured from bottle on 2020-07-09 at 00:34:34
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/bzip2.rb
==> Caveats
bzip2 is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have bzip2 first in your PATH run:
  echo 'export PATH="/usr/local/opt/bzip2/bin:$PATH"' >> ~/.zshrc

For compilers to find bzip2 you may need to set:
  export LDFLAGS="-L/usr/local/opt/bzip2/lib"
  export CPPFLAGS="-I/usr/local/opt/bzip2/include"

==> Analytics
install: 1,045 (30 days), 3,612 (90 days), 16,828 (365 days)
install-on-request: 946 (30 days), 3,014 (90 days), 13,350 (365 days)
build-error: 0 (30 days)

@Al-Ferguson
Copy link

Al-Ferguson commented Jul 15, 2020

I saw something somewhere on make/cmake having issues under Big Sur & Xcode 12, so I did do a homebrew reinstall from source:
brew reinstall -s make cmake
As homebrew is does not yet support Big Sur binaries, this may make a difference.

FYI: I was just able to install Python 3.8.4 with my configuration (i.e. "pyenv install 3.8.4") under MacOS 11 & Xcode 12. And with home-brew's 3.8.4 roll out (via python@3.8), that build successfully as well.

If you could not tell from my ~/.zshrc above, I have installed the following key homebrew components before building python via "pyenv install ...":

  • OpenSSL@1.1
  • bzip2
  • readline
  • tcl-tk
  • zlib
  • sqlite

Also, while not having the following installed (and added to my ~/.zshrc) did not produce an error; output from the compile indicated the following could not be found, so support for them was not included in the python build:

  • _curses & _curses_panel
  • ossaudiodev
  • spwd

So if you need support for those, you will need to install them and add the to the ~/.zshrc.

@bryanmobrien
Copy link

@PritamDutt , thank you.
Trying your suggestions gives me the following warning (for 3.8.0):

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.0 to /Users/bobrien/.pyenv/versions/3.8.0

When I tried 3.8.3, again, pasting your script above:

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.3 to /Users/bobrien/.pyenv/versions/3.8.3

Checked homebrew, and it is installed, but not linked.

brew link bzip2

Had no effect. I removed 3.8.3 and reattempted

pyenv install 3.8.3

and got the same warning.
At least, I have python installed on Big Sur via pyenv.

Here is the output of brew info bzip2 from my system...

Hope this can help!

% brew info bzip2
bzip2: stable 1.0.8 (bottled) [keg-only]
Freely available high-quality data compressor
https://sourceware.org/bzip2/
/usr/local/Cellar/bzip2/1.0.8 (26 files, 400.6KB)
  Poured from bottle on 2020-07-09 at 00:34:34
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/bzip2.rb
==> Caveats
bzip2 is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have bzip2 first in your PATH run:
  echo 'export PATH="/usr/local/opt/bzip2/bin:$PATH"' >> ~/.zshrc

For compilers to find bzip2 you may need to set:
  export LDFLAGS="-L/usr/local/opt/bzip2/lib"
  export CPPFLAGS="-I/usr/local/opt/bzip2/include"

==> Analytics
install: 1,045 (30 days), 3,612 (90 days), 16,828 (365 days)
install-on-request: 946 (30 days), 3,014 (90 days), 13,350 (365 days)
build-error: 0 (30 days)

@PritamDutt ,
Thanks, stupid me - I did not have bzip2 from homebrew in my $PATH.

Thank you, again!

@htp
Copy link

htp commented Aug 31, 2020

This slightly-modified version of @PritamDutt's invocation allowed me to install a couple of different versions of Python (2.7.16, 3.7.7, and 3.8.3) without having to to brew link bzip2 or zlib:

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

I was also able to install Pythons 3.8.4 and 3.8.5 by omitting the patch, like this:

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib" \
pyenv install 3.8.5

@aaron-trout
Copy link

aaron-trout commented Sep 1, 2020

Thanks @htp! I managed to get this working for installing Python 3.5 using this script:

#!/bin/bash

set -ex

export CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include"
export LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib -L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

pyenv install --patch 3.5.9 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

I also did brew reinstall -s make cmake as someone else suggested earlier, not sure if that helped or not!

@rdbreak
Copy link

rdbreak commented Sep 5, 2020

Give this a try:

  1. Download and install the latest Xcode 12 beta
  2. Go to > Preferences > Locations > Command Line Tools > Choose Xcode 12.0.

Then set the following environment variables in your shell session. I created a shell script that does it all.

#!/bin/bash
echo "Installing Dependencies"
brew install zlib
brew install sqlite
brew install bzip2
brew install libiconv
brew install libzip
echo "Done!"
echo
echo -e "Setting Environment Variables"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/zlib/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/zlib/include"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/sqlite/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/sqlite/include"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/sqlite/lib/pkgconfig"
echo "Done!"
echo
echo "Installing Python 3.8.5"
pyenv install 3.8.5
echo "Done!"
exit 0

Then I was able to install Python. Script output below:

# bash fix.sh

Installing Dependencies
## BREW OUTPUT REMOVED ##
Done!

Setting Environment Variables
Done!

Installing Python 3.8.5
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to ~/.pyenv/versions/3.8.5
Done!

@henrychea
Copy link

Does anyone know if this issue is still present on the newest beta (macOS Big Sur beta 7)?

@rdbreak
Copy link

rdbreak commented Sep 19, 2020

Does anyone know if this issue is still present on the newest beta (macOS Big Sur beta 7)?

I used this fix on the Big Sur beta and it should work for all iterations of the beta.

@synchronizing
Copy link

synchronizing commented Sep 19, 2020

A unique issue may occur if one has installed and set the latest XCode SDK. pyenv fails on run due to invalid clang version:

  λ sudo xcode-select --switch /Applications/Xcode.app/
  λ clang --version
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
  λ ./pyenv_install_rdbreak_original_script.sh

Where ./pyenv_install_rdbreak_original_script.sh is the script posted above (here).

BUILD FAILED (OS X 11.0 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/4m/1v3nfl_x32z07msw_m6y_1m40000gn/T/python-build.20200919155415.61074
Results logged to /var/folders/4m/1v3nfl_x32z07msw_m6y_1m40000gn/T/python-build.20200919155415.61074.log

Last 10 log lines:
checking for python3.8... python3.8
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... no
configure: error: in `/var/folders/4m/1v3nfl_x32z07msw_m6y_1m40000gn/T/python-build.20200919155415.61074/Python-3.8.5':
configure: error: C compiler cannot create executables
See `config.log' for more details
make: *** No targets specified and no makefile found.  Stop.
Done!

The issue relies on the fact that Big Sur is now serving both versions 11.x and 10.x by default (for backwards compatibility) with automatic context switch (source, and a good article with a decent sitrep):

rfXPY

There are two viable solutions:

First Solution

Set your default SDK to the latest beta, which will automatically use context switching and utilize SDK 10.x with pyenv:

Download Command Line Tools for Xcode 12.2 beta 2 and then do the followowing:

  λ sudo xcode-select --switch /Library/Developer/CommandLineTools
  λ clang --version
Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Second Solution

The second solution involves simply updating @rdbreak's script above to include the new SYSTEM_VERSION_COMPAT=1 flag:

#!/bin/bash
echo "Installing Dependencies"
brew install zlib
brew install sqlite
brew install bzip2
brew install libiconv
brew install libzip
echo "Done!"
echo
echo -e "Setting Environment Variables"
export SYSTEM_VERSION_COMPAT=1
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/zlib/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/zlib/include"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/sqlite/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/sqlite/include"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/sqlite/lib/pkgconfig"
echo "Done!"
echo
echo "Installing Python 3.8.5"
pyenv install 3.8.5
echo "Done!"
exit 0

Either solution will work in the meantime.

Possible Solution (?)

Because of all the dependencies already installed on my computer, I am unable to test this from scratch to ensure it's working as expected -- however, it seems that:

SYSTEM_VERSION_COMPAT=1 pyenv install 3.8.5

Works just fine as long as brew dependencies are also installed correctly (without the need to export majority of the flags in the script above). If someone is willing to test this out, it would be appreciated -- my current assumption is that as long as zlib and readline is installed and exported correctly via brew install and brew link (following the instructions), the above one liner should work.

@henrychea
Copy link

henrychea commented Sep 25, 2020

Hello everyone, I'm trying to install python 3.6.x using @synchronizing script however this is the error I ran into

Last 10 log lines:
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl@1.1/include -I/Users/henrychea/.pyenv/versions/3.6.9/include  -I/usr/local/opt/zlib/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl@1.1/include -I/Users/henrychea/.pyenv/versions/3.6.9/include  -I/usr/local/opt/zlib/include -I/usr/local/opt/sqlite/include  -DPy_BUILD_CORE  -c ./Modules/errnomodule.c -o Modules/errnomodule.o
./Modules/posixmodule.c:8210:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
./Modules/posixmodule.c:10432:5: warning: code will never be executed [-Wunreachable-code]
    Py_FatalError("abort() called from Python code didn't abort!");
    ^~~~~~~~~~~~~
1 warning and 1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
Done!

Currently I'm on macOS 11.00 (20A5374i) Beta 8. This issue isn't present when installing python 3.8.x or python 3.7.x

@synchronizing
Copy link

synchronizing commented Sep 25, 2020

Currently I'm on macOS 11.00 (20A5374i) Beta 8. This issue isn't present when installing python 3.8.x or python 3.7.x

I was shown the available update yesterday, to beta 8, but I haven't gotten the chance to update yet. Would you mind reporting what your clang --version is? Also, are you installing all of the brew packages as mentioned in the script?

By the way, the script is originally from @rdbreak, so give his a shot as well and see if it helps.

@henrychea
Copy link

henrychea commented Sep 26, 2020

I was shown the available update yesterday, to beta 8, but I haven't gotten the chance to update yet. Would you mind reporting what your clang --version is? Also, are you installing all of the brew packages as mentioned in the script?

By the way, the script is originally from @rdbreak, so give his a shot as well and see if it helps.

I've given @rdbreak 's script a try as well but to no avail. I have all the dependencies from brew. Here is my clang version

Apple clang version 12.0.0 (clang-1200.0.32.4)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I can run 3.7.x or higher without issue but this is just to notify anyone using versions 3.6.x

@synchronizing
Copy link

I can run 3.7.x or higher without issue but this is just to notify anyone using versions 3.6.x

Updated to beta 8. Updating SDK to Command Line Tools for Xcode 12.2 beta 2 and running

sudo xcode-select --switch /Library/Developer/CommandLineTools
export SYSTEM_VERSION_COMPAT=1
pyenv install 3.6.0

fixed my issues.

@baspeti
Copy link

baspeti commented Mar 18, 2021

I tried building using those inputs to my .zshrc and still got an error:

andytriboletti@andys-mac ~ % CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/1r/bdbwv9r53ls_80hpk1t6fvlm0000gn/T/python-build.20200715094722.77820
Results logged to /var/folders/1r/bdbwv9r53ls_80hpk1t6fvlm0000gn/T/python-build.20200715094722.77820.log

Last 10 log lines:
checking size of _Bool... 1
checking size of off_t... 8
checking whether to enable large file support... no
checking size of time_t... 8
checking for pthread_t... yes
checking size of pthread_t... 8
checking size of pthread_key_t... 8
checking whether pthread_key_t is compatible with int... no
configure: error: Unexpected output of 'arch' on OSX
make: *** No targets specified and no makefile found.  Stop.
andytriboletti@andys-mac ~ % arch
arm64

I'm just writing down my experiences here, maybe it is going to be helpful to somebody else too. So I'm running on Big Sur on a Mac M1 chip and I had the exact same error output when I tried to install python 3.6.12. The key for me to add arch -x86_64 just before the pyenv install part of the command.

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" arch -x86_64 pyenv install --patch 3.6.12 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

@tomharvey
Copy link

@baspeti It looks like you’ve brew installed your pyenv? On an M1 I had much more success when pyenv was installed with the pyenv installer. With brew, i had to force the arch (as youve done) with almost everything brew related. I’d encourage you to try https://github.com/pyenv/pyenv-installer

but, I’m aware that things move fast and brew may be better with M1 now (or very soon)

@sbinlondon
Copy link

New (this week) M1 user here and have just set up:

  • Homebrew (native as of 3.0.0)
  • Pyenv 1.2.24 installed via brew
  • Python 3.9.2 installed via pyenv and set as the global version (apparently 3.9 and up is M1 native)

And everything seems to be working just fine.

For anyone like me who is also just learning about what arch means, you can check what your terminal is running on the M1 by typing arch. If it's running natively, you'll see arm64. If you see i386, it means your computer is emulating an Intel chip and anything you install from the terminal will not be installed natively. I didn't know any of this when I got my fresh Macbook on Monday and I've spent 2 days uninstalling everything and starting from a blank slate again and making sure I'm installing brew and other dev tools natively.

So if you install brew natively, then pyenv with brew, then Python 3 with pyenv, it should all work fine 👍

@bobweston
Copy link

bobweston commented Mar 26, 2021

New (this week) M1 user here and have just set up:

  • Homebrew (native as of 3.0.0)
  • Pyenv 1.2.24 installed via brew
  • Python 3.9.2 installed via pyenv and set as the global version (apparently 3.9 and up is M1 native)

And everything seems to be working just fine.

For anyone like me who is also just learning about what arch means, you can check what your terminal is running on the M1 by typing arch. If it's running natively, you'll see arm64. If you see i386, it means your computer is emulating an Intel chip and anything you install from the terminal will not be installed natively. I didn't know any of this when I got my fresh Macbook on Monday and I've spent 2 days uninstalling everything and starting from a blank slate again and making sure I'm installing brew and other dev tools natively.

So if you install brew natively, then pyenv with brew, then Python 3 with pyenv, it should all work fine 👍

I believe the problem is with versions of python earlier than 3.8.4.

@Anthonyive
Copy link

Anthonyive commented Apr 11, 2021

Well...I'm having this problem too...I'm using an intel mac.

~/test 14s
❯ pyenv install 3.8.8
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.8.tar.xz...
-> https://www.python.org/ftp/python/3.8.8/Python-3.8.8.tar.xz
error: failed to download Python-3.8.8.tar.xz

BUILD FAILED (OS X 11.2.3 using python-build 20180424)

I can download the file by just typing the link to the browser. wget and curl should be properly installed by brew.

@boabdilperez
Copy link

I didn't need to mess with CFLAGS, LDFLAGS or install anything in particular with brew, although I did already have openssl@1.1 and readline from brew. All that was required was:

$ pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

Environment:

  • macOS 11.2.2
  • xcode-select version 2384
  • pyenv 1.2.23

This worked for me! Thank goodness, I was about to lose my mind!

My Environment:
Intel MacBook Pro 16"
macOS 11.2.3
Apple clang version 12.0.0 (clang-1200.0.32.29)
xcode-select version 2384
pyenv ver. 1.2.26

@Anthonyive
Copy link

Well...I'm having this problem too...I'm using an intel mac.

~/test 14s
❯ pyenv install 3.8.8
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.8.tar.xz...
-> https://www.python.org/ftp/python/3.8.8/Python-3.8.8.tar.xz
error: failed to download Python-3.8.8.tar.xz

BUILD FAILED (OS X 11.2.3 using python-build 20180424)

I can download the file by just typing the link to the browser. wget and curl should be properly installed by brew.

Comment on my own comment. It actually works now. It seems like there was some downloading issue happening and I couldn't download the Python-3.8.8.tar.xz.

@native-api
Copy link
Member

Duplicate of #1737

@native-api native-api marked this as a duplicate of #1737 May 7, 2021
@M4RcB4
Copy link

M4RcB4 commented May 18, 2021

ISSUE:

I can't get pyenv install 3.8.2 to work without ending in error.

System Information:

  • OSX: 11.3.1(Big Sur)
  • pyenv: 1.2.27
  • zsh: 5.8

Error message received:
BUILD FAILED (OS X 11.3.1 using python-build 1.2.27-41-gddf952ab)
./Modules/posixmodule.c:9197:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = sendfile(in, out, offset, &sbytes, &sf, flags);
^
1 error generated.

make: *** [Modules/posixmodule.o] Error 1

SOLUTION!!!!:

Install with the --patch flag
pyenv install --patch 3.8.2 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

I would assume this is similar for other python version installs. Just swap the python install version and leave the patch URL the same.

BONUS
You may also want to include this(see below) in your zshrc. One of the many things it will do is set the python version to what is specified in the .python-version in the current directory. This is great for automating setting the version across different projects.

eval "$(pyenv init -)"
fi```

@InfinityCode777
Copy link

InfinityCode777 commented May 19, 2021

My environment

  • Big Sur 11.3.1
  • Xcode 12.5
  • Apple clang version 12.0.5 (clang-1205.0.22.9)
  • Target: x86_64-apple-darwin20.4.0
  • Thread model: posix

I tried a lot of options/combinations for LDFLAGS, CPPFLAGS and CFLAGS, none of them is working. I met the same error as M4RcB4 posted. Then I tried patch method (without setting any of LDFLAGS, CPPFLAGS and CFLAGS) and it just works like a charm!!!

Also, in case anyone encounters some pandas installation failed error (I saw a millions lines of errors and warnings spit out in my console) after you run pipenv install, you may try this line

export SYSTEM_VERSION_COMPAT=1

and it helped me install dependencies correctly (original answer from SO)

@fenngwd
Copy link

fenngwd commented Jun 1, 2021

New (this week) M1 user here and have just set up:

  • Homebrew (native as of 3.0.0)
  • Pyenv 1.2.24 installed via brew
  • Python 3.9.2 installed via pyenv and set as the global version (apparently 3.9 and up is M1 native)

And everything seems to be working just fine.

For anyone like me who is also just learning about what arch means, you can check what your terminal is running on the M1 by typing arch. If it's running natively, you'll see arm64. If you see i386, it means your computer is emulating an Intel chip and anything you install from the terminal will not be installed natively. I didn't know any of this when I got my fresh Macbook on Monday and I've spent 2 days uninstalling everything and starting from a blank slate again and making sure I'm installing brew and other dev tools natively.

So if you install brew natively, then pyenv with brew, then Python 3 with pyenv, it should all work fine 👍

No, in my case it doesn't work fine.

@fxcl
Copy link

fxcl commented Jun 1, 2021

CC=gcc pyenv install 3.9.1
or
CC=gcc asdf install python 3.9.1

@aktaseren
Copy link

Give this a try:

  1. Download and install the latest Xcode 12 beta
  2. Go to > Preferences > Locations > Command Line Tools > Choose Xcode 12.0.

Then set the following environment variables in your shell session. I created a shell script that does it all.

#!/bin/bash
echo "Installing Dependencies"
brew install zlib
brew install sqlite
brew install bzip2
brew install libiconv
brew install libzip
echo "Done!"
echo
echo -e "Setting Environment Variables"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/zlib/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/zlib/include"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/sqlite/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/sqlite/include"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/sqlite/lib/pkgconfig"
echo "Done!"
echo
echo "Installing Python 3.8.5"
pyenv install 3.8.5
echo "Done!"
exit 0

Then I was able to install Python. Script output below:

# bash fix.sh

Installing Dependencies
## BREW OUTPUT REMOVED ##
Done!

Setting Environment Variables
Done!

Installing Python 3.8.5
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to ~/.pyenv/versions/3.8.5
Done!

I was having the same issue. Only that solution worked in my case. Thanks.

@Sofia-shajahan
Copy link

Thanks @htp! I managed to get this working for installing Python 3.5 using this script:

#!/bin/bash

set -ex

export CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include"
export LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib -L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

pyenv install --patch 3.5.9 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

I also did brew reinstall -s make cmake as someone else suggested earlier, not sure if that helped or not!

Thank you so much it worked

@roelzkie15
Copy link

In my case I only need to:

$ arch -x86_64 pyenv install 3.8.5
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to /Users/roelzkie/.pyenv/versions/3.8.5

or

$ arch -x86_64 pyenv install 3.9.6
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.6.tar.xz...
-> https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tar.xz
Installing Python-3.9.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.9.6 to /Users/roelzkie/.pyenv/versions/3.9.6

I didn't test with any version but I guess it will work just like the above.

@andrii-furman
Copy link

I added #undef _POSIX_C_SOURCE right before #include <sys/socket.h> in Modules/posixmodule.c to get things compiling.

Hi, @palmerc. Can you reply to the full path to the posixmodule.c. Because I can't where the file is located from the stack trace.
Will appreciate your response.

@rorads
Copy link

rorads commented Nov 14, 2021

I didn't need to mess with CFLAGS, LDFLAGS or install anything in particular with brew, although I did already have openssl@1.1 and readline from brew. All that was required was:

$ pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

Environment:

  • macOS 11.2.2
  • xcode-select version 2384
  • pyenv 1.2.23

I had the same issue, effecting only some python versions (3.9.x fine, 3.7.x broken). Also tried with all the flags, and went down a long rabbit hole learning about BLAS and LAPACK. All in vain!

This solution worked for me too. Thanks to everyone for sharing solutions to fix.

  • macOS 11.6
  • xcode-select version 2384
  • pyenv 2.2.0

I don't believe I have anything out of date in my stack above, so it seems this issue is still live - should it be reopened?

@kazuki04
Copy link

kazuki04 commented Nov 19, 2021

I can install version 3.7.4

My environment:

  • M1 Mac
  • macOS 11.6
  • Apple clang version 12.0.5 (clang-1205.0.22.9)
  • Xcode 12.5
  • pyenv 2.2.0

Solution in my case

First step, install Rosetta. This may be useful

softwareupdate --install-rosetta

install x86 Homebrew

arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

and install required packages

arch -x86_64 brew install pyenv bzip2 zlib

Finally, pyenv install with patch

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix
bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun
--show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib
-L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew
--prefix bzip2)/lib" pyenv install --patch 3.7.4 < <(curl -sSL
https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

@paulyzhangSmartNews
Copy link

There are so many problems about installing python, so, the official had better find why and fix it.

@allen-q
Copy link

allen-q commented Oct 6, 2022

$ pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

I had a similar issue with Python 3.9.0. I tried to use the same patch but it failed with an error Hunk #1 FAILED at 3426.. Should I use a different patch for 3.9.0. If so, how can I find which patch commit to use for 3.9.0. Thank you if you could help!

@glebovdev
Copy link

I had the same issue when I was trying to pyenv install 3.9.7 under macOS Ventura 13.0.1.
I've tried almost every suggested solution here but had no luck.

The following steps helped me:

  1. make sure xcode-select -p gives the following:
    /Library/Developer/CommandLineTools

If it's different, then: sudo xcode-select -switch /Library/Developer/CommandLineTools

  1. cd ~/.pyenv/plugins/python-build && git pull
  2. pyenv install 3.9.7

I cannot exclude that some of the exported flags or installed (with brew) packages mentioned above also affected, but unfortunately, I can't reproduce which exactly and how.

@freefeynman123
Copy link

freefeynman123 commented Jan 16, 2023

I had the same issue when I was trying to pyenv install 3.9.7 under macOS Ventura 13.0.1. I've tried almost every suggested solution here but had no luck.

The following steps helped me:

  1. make sure xcode-select -p gives the following:
    /Library/Developer/CommandLineTools

If it's different, then: sudo xcode-select -switch /Library/Developer/CommandLineTools

  1. cd ~/.pyenv/plugins/python-build && git pull
  2. pyenv install 3.9.7

I cannot exclude that some of the exported flags or installed (with brew) packages mentioned above also affected, but unfortunately, I can't reproduce which exactly and how.

My environment:

MacBook Pro 2019, i7
macOS 13.1
Apple clang version 14.0.0 (clang-1400.0.29.202)
Xcode 14.2
pyenv 2.3.10

Thank you, that finally made it possible for me to install Python 3.8.9 without any additional flags or scripts (although installing packages suggested in the thread could have helped).

Unfortunately I still cannot install Python 3.6.4, which would be useful for older projects in my company. I tried every suggestion in the thread with different patching, flags, etc. but I'm still getting:

`BUILD FAILED (OS X 13.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/zh/_kmwgbrx5gx5jry3mqnw_xyw0000gn/T/python-build.20230116135504.38123
Results logged to /var/folders/zh/_kmwgbrx5gx5jry3mqnw_xyw0000gn/T/python-build.20230116135504.38123.log

Last 10 log lines:
checking for --with-cxx-main=... no
checking for clang++... no
configure:

By default, distutils will build C++ extension modules with "clang++".
If this is not intended, then set CXX on the configure command line.

checking for the platform triplet based on compiler characteristics... darwin
configure: error: internal configure error for the platform triplet, please file a bug report
make: *** No targets specified and no makefile found. Stop.`

@matheusft
Copy link

matheusft commented Jan 29, 2023

I can install version 3.7.4

My environment:

  • M1 Mac
  • macOS 11.6
  • Apple clang version 12.0.5 (clang-1205.0.22.9)
  • Xcode 12.5
  • pyenv 2.2.0

Solution in my case

First step, install Rosetta. This may be useful

softwareupdate --install-rosetta

install x86 Homebrew

arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

and install required packages

arch -x86_64 brew install pyenv bzip2 zlib

Finally, pyenv install with patch

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix
bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun
--show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib
-L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew
--prefix bzip2)/lib" pyenv install --patch 3.7.4 < <(curl -sSL
https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

I'm getting BUILD FAILED
Any thoughts?

I'm using macOS Ventura 13.1 | Apple M1 Pro

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.