setup.py: GCC detection is broken when cross-compiling with a German locale #82653

AlexGrund · 2019-10-14T12:28:35Z

BPO	38472
Nosy	@doko42, @vstinner, @miss-islington, @iritkatriel
PRs	bpo-38472: setup.py uses LC_ALL=C to check the C compiler #30929 [3.10] bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929) #30931 [3.9] bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929) #30932

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-10-14.12:28:34.957>
labels = ['build', '3.9']
title = 'setup.py: GCC detection is broken when cross-compiling with a German locale'
updated_at = <Date 2022-01-26.23:50:38.165>
user = 'https://bugs.python.org/AlexGrund'

bugs.python.org fields:

activity = <Date 2022-01-26.23:50:38.165>
actor = 'miss-islington'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2019-10-14.12:28:34.957>
creator = 'Alex Grund'
dependencies = []
files = []
hgrepos = []
issue_num = 38472
keywords = ['patch']
message_count = 11.0
messages = ['354629', '355072', '355074', '410831', '411388', '411801', '411805', '411806', '411807', '411812', '411813']
nosy_count = 5.0
nosy_names = ['doko', 'vstinner', 'miss-islington', 'Alex Grund', 'iritkatriel']
pr_nums = ['30929', '30931', '30932']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue38472'
versions = ['Python 3.9']

AlexGrund · 2019-10-14T12:28:34Z

setup.py runs <CC> -E -v - </dev/null 1>/dev/null to figure out include and library paths from the compiler in the function add_gcc_paths.

However sample output from the compiler is:

Es werden eingebaute Spezifikationen verwendet.
COLLECT_GCC=g++
Ziel: x86_64-pc-linux-gnu
Konfiguriert mit: ../configure --enable-languages=c,c++,fortran --enable-lto --enable-checking=release --disable-multilib --enable-shared=yes --enable-static=yes --enable-threads=posix --enable-plugins --enable-gold=default --enable-ld --with-plugin-ld=ld.gold --prefix=/sw/installed/GCCcore/9.1.0 --with-local-prefix=/sw/installed/GCCcore/9.1.0 --enable-bootstrap --with-isl=/dev/shm/easybuild-build/GCCcore/9.1.0/dummy-/gcc-9.1.0/stage2_stuff
Thread-Modell: posix
gcc-Version 9.1.0 (GCC)
COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
/software/haswell/GCCcore/9.1.0/bin/../libexec/gcc/x86_64-pc-linux-gnu/9.1.0/cc1 -E -quiet -v -iprefix /software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/ - -mtune=generic -march=x86-64
nicht vorhandenes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/include« wird ignoriert
doppeltes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/../../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include« wird ignoriert
doppeltes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/../../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include-fixed« wird ignoriert
nicht vorhandenes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/../../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/include« wird ignoriert
doppeltes Verzeichnis »/sw/installed/GCCcore/9.1.0/include« wird ignoriert
da es ein Nicht-Systemverzeichnis ist, das ein Systemverzeichnis dupliziert
Suche für »#include "..."« beginnt hier:
Suche für »#include <...>« beginnt hier:
/sw/installed/binutils/2.32-GCCcore-9.1.0/include
/sw/installed/zlib/1.2.11-GCCcore-9.1.0/include
/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include
/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include-fixed
/sw/installed/GCCcore/9.1.0/include
/usr/include
Ende der Suchliste.
COMPILER_PATH=/software/haswell/GCCcore/9.1.0/bin/../libexec/gcc/x86_64-pc-linux-gnu/9.1.0/:/software/haswell/GCCcore/9.1.0/bin/../libexec/gcc/
LIBRARY_PATH=/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/:/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/:/sw/installed/GCCcore/9.1.0/lib64/../lib64/:/sw/installed/GCCcore/9.1.0/lib/../lib64/:/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/sw/installed/binutils/2.32-GCCcore-9.1.0/lib/:/sw/installed/zlib/1.2.11-GCCcore-9.1.0/lib/:/sw/installed/GCCcore/9.1.0/lib64/:/sw/installed/GCCcore/9.1.0/lib/:/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'

So the correct matcher would be "gcc-Version", maybe in addition to "gcc version"

Due to this the setup fails to build e.g. bzip2 modules as the include and lib paths are passed in via CPATH and LIBRARY_PATH only (module system on HPC system)

AlexGrund · 2019-10-21T14:12:26Z

This seems to be a locale issue. So a solution would be to use LC_ALL=C before invoking the compiler (or the cross-platform equivalent)

vstinner · 2019-10-21T14:27:29Z

What are you trying to do? Compile Python? Cross-compile Python?

setup.py runs <CC> -E -v - </dev/null 1>/dev/null to figure out include and library paths from the compiler in the function add_gcc_paths.

Are you talking about the add_cross_compiling_paths() function which is only called when _PYTHON_HOST_PLATFORM environment variable is set?

So GCC says "gcc-Version 9.1.0 (GCC) " with your locale, but does it write "gcc version 9.1.0 ..." with LC_ALL=C?

iritkatriel · 2022-01-17T23:07:37Z

Alex, is this still an issue with current python versions (>= 3.9)? And can you followup on Victor's questions?

AlexGrund · 2022-01-23T17:38:18Z

Yes this is still an issue.

I'm trying to compile Python on an HPC system which uses modules (see e.g. LMod).

Yes with LC_ALL=C it does write "gcc version 9.1.0 ..."

vstinner · 2022-01-26T22:03:29Z

Ok, I reproduced the issue. I wrote PR 30929 to fix it.

The issue only occurs when cross-compiling Python with GCC and a German locale. I can reproduce the issue on Fedora 35:

$ LC_ALL=de_DE gcc -E -v
Es werden eingebaute Spezifikationen verwendet.
COLLECT_GCC=gcc
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Ziel: x86_64-redhat-linux
Konfiguriert mit: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-11.2.1-20211203/obj-x86_64-redhat-linux/isl-install --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread-Modell: posix
Unterst�tzte LTO-Kompressionsalgorithmen: zlib zstd
gcc-Version 11.2.1 20211203 (Red Hat 11.2.1-7) (GCC)

The last line starts with "gcc-Version 11.2.1". Whereas the last line starts with "gcc version 11.2.1" with the C locale:

$ LC_ALL=C gcc -E -v
Using built-in specs.
COLLECT_GCC=gcc
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-11.2.1-20211203/obj-x86_64-redhat-linux/isl-install --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.1 20211203 (Red Hat 11.2.1-7) (GCC)

vstinner · 2022-01-26T23:21:18Z

FYI distutils.unixccompiler has a private _is_gcc() function:

    def _is_gcc(self, compiler_name):
        # clang uses same syntax for rpath as gcc
        return any(name in compiler_name for name in ("gcc", "g++", "clang"))

It's called with:

    compiler = os.path.basename(sysconfig.get_config_var("CC"))

For example, on my Fedora 35, compiler is the string: 'gcc -pthread'. So _is_gcc() returns True.

vstinner · 2022-01-26T23:22:13Z

New changeset a9503ac by Victor Stinner in branch 'main':
bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929)
a9503ac

vstinner · 2022-01-26T23:23:49Z

The workaround for this bug is to build Python using the command:

    LC_ALL=C make

rather than running:

make

miss-islington · 2022-01-26T23:50:02Z

New changeset 171fdf2 by Miss Islington (bot) in branch '3.10':
bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929)
171fdf2

miss-islington · 2022-01-26T23:50:38Z

New changeset ff11eff by Miss Islington (bot) in branch '3.9':
bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929)
ff11eff

AlexGrund mannequin added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes build The build process and cross-build labels Oct 14, 2019

iritkatriel removed 3.7 (EOL) end of life 3.8 only security fixes labels Jan 23, 2022

vstinner changed the title ~~GCC detection in setup.py is broken~~ setup.py: GCC detection is broken when cross-compiling with a German locale Jan 26, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

iritkatriel closed this as completed May 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setup.py: GCC detection is broken when cross-compiling with a German locale #82653

setup.py: GCC detection is broken when cross-compiling with a German locale #82653

AlexGrund mannequin commented Oct 14, 2019

AlexGrund mannequin commented Oct 14, 2019

AlexGrund mannequin commented Oct 21, 2019

vstinner commented Oct 21, 2019

iritkatriel commented Jan 17, 2022

AlexGrund mannequin commented Jan 23, 2022

vstinner commented Jan 26, 2022

vstinner commented Jan 26, 2022

vstinner commented Jan 26, 2022

vstinner commented Jan 26, 2022

miss-islington commented Jan 26, 2022

miss-islington commented Jan 26, 2022

setup.py: GCC detection is broken when cross-compiling with a German locale #82653

setup.py: GCC detection is broken when cross-compiling with a German locale #82653

Comments

AlexGrund mannequin commented Oct 14, 2019

AlexGrund mannequin commented Oct 14, 2019

AlexGrund mannequin commented Oct 21, 2019

vstinner commented Oct 21, 2019

iritkatriel commented Jan 17, 2022

AlexGrund mannequin commented Jan 23, 2022

vstinner commented Jan 26, 2022

vstinner commented Jan 26, 2022

vstinner commented Jan 26, 2022

vstinner commented Jan 26, 2022

miss-islington commented Jan 26, 2022

miss-islington commented Jan 26, 2022