Build failure for 3.6 on Centos 5.11

[stevendaprano]

BPO	28092
Nosy	@ncoghlan, @tds333, @vstinner, @benjaminp, @ned-deily, @stevendaprano, @njsmith, @thewtex, @ambv, @vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-10-21.05:41:06.711>
created_at = <Date 2016-09-12.03:16:58.987>
labels = ['build']
title = 'Build failure for 3.6 on Centos 5.11'
updated_at = <Date 2016-10-21.05:41:06.709>
user = 'https://github.com/stevendaprano'

bugs.python.org fields:

activity = <Date 2016-10-21.05:41:06.709>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2016-10-21.05:41:06.711>
closer = 'benjamin.peterson'
components = ['Build']
creation = <Date 2016-09-12.03:16:58.987>
creator = 'steven.daprano'
dependencies = []
files = []
hgrepos = []
issue_num = 28092
keywords = []
message_count = 37.0
messages = ['275935', '275939', '275953', '275966', '275971', '275984', '275997', '276012', '276018', '276020', '276079', '276089', '276144', '276150', '276164', '278392', '278556', '278568', '278584', '278599', '278602', '278611', '278639', '278643', '278645', '278647', '278655', '278659', '278661', '278696', '278701', '278715', '278721', '279040', '279087', '279110', '279111']
nosy_count = 12.0
nosy_names = ['rgbecker', 'ncoghlan', 'tds333', 'vstinner', 'benjamin.peterson', 'ned.deily', 'steven.daprano', 'njs', 'thewtex', 'lukasz.langa', 'python-dev', 'martin.panter']
pr_nums = []
priority = None
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue28092'
versions = ['Python 3.6']

[stevendaprano]

On Centos 5.11, building fails with:

Python/dtrace_stubs.o: In function PyDTrace_LINE': /home/steve/python/python-dev/cpython/Include/pydtrace.h:28: multiple definition of PyDTrace_LINE'
Python/ceval.o:/home/steve/python/python-dev/cpython/Include/pydtrace.h:28: first defined here
Python/dtrace_stubs.o: In function PyDTrace_FUNCTION_ENTRY': /home/steve/python/python-dev/cpython/Include/pydtrace.h:29: multiple definition of PyDTrace_FUNCTION_ENTRY'
Python/ceval.o:/home/steve/python/python-dev/cpython/Include/pydtrace.h:29: first defined here
[... many, many more duplicate definitions ...]
collect2: ld returned 1 exit status
make: *** [Programs/_freeze_importlib] Error 1

I've tried the usual make distclean and hg update -C default tricks, and the error remains.

See also discussion here:

https://mail.python.org/pipermail/python-list/2016-September/713888.html

[vadmium]

This is like the OS X Tiger buildbot failure.
http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/11323/steps/compile/logs/stdio
/usr/bin/ld: multiple definitions of symbol _PyDTrace_FUNCTION_ENTRY
Python/ceval.o definition of _PyDTrace_FUNCTION_ENTRY in section (__TEXT,__text)
Python/dtrace_stubs.o definition of _PyDTrace_FUNCTION_ENTRY in section (__TEXT,__text)

My stab in the dark is that the compiler (GCC?) being used is interpreting “inline” functions differently to C99.

[ned-deily]

(The OS X Tiger buildbot uses a version of GCC 4.0.)

[vadmium]

Benjamin changed PEP-7 to allow static inline functions directly in Python 3.6. But later he added program-wide, linkable inline functions in the Python 3.6 code:

2f77a9f0b9d6: add plain “inline” to header file
63ae310b60ff: add “extern inline” stubs in a new file

It seems GCC does not support the C99 syntax for these kind of linkable inline functions until 4.3. Some possible fixes or workarounds:

1. Revert the offending functions back to macros

2. Clarify in PEP-7 that we intentionally use linkable C99 (extern) inline functions, therefore compilers like Steven’s and the buildbot’s aren’t supported for 3.6

3. Add some preprocessor magic based on __GNUC_STDC_INLINE__ to detect when the GCC 4.0 reversed meanings of “inline” and “extern inline” have to be used.

[benjaminp]

GCC 4.3 was released in March 2008, so I think we are within our rights to drop support for older toolchains.

[tds333]

This can be related to building manylinux1 binaries where the Centos 5.11 is used.
There can be a possibility to break manaylinux1 builds for Python 3.6, they are used to distribute packages for every Linux distribution.

https://www.python.org/dev/peps/pep-0513/

[ned-deily]

Do we know for sure that the manylinux1 builds are affected by this, i.e. can someone try building manylinux1 for 3.6?

On the one hand, it is clear that many core developers would like to use these newer features of C, features that have been available for many years. On the other, we may be breaking the "manylinux1" proposed standard and force a move to a newer one, as well as directly impacting some users (such as Steven and one of our buildbots) using old setups.

Nick, as BDFL-Delegate for PEP-513, what do you think? We need to resolve this issue before we can release b1.

[ncoghlan]

While manylinux1 requires that compliant binaries *run* when linked against the GCC 4.2 binaries, the actual recommended compiler version (and the one used in the reference Docker image) is 4.8.2: https://www.python.org/dev/peps/pep-0513/#compilation-of-compliant-wheels

That's the compiler version in Red Hat's Developer Toolset 2, the last version to support RHEL/CentOS 5.

Unfortunately, softwarecollections.org only goes back to RHEL/CentOS 6, so the devtoolset RPMs are available via Tru Huynh's personal account on the CentOS file server: https://people.centos.org/tru/devtools-2/readme

So if 3.6 compiles and runs on CentOS 5 with GCC 4.8.2, that's sufficiently compatible for manylinux1 purposes.

[ned-deily]

Thanks for the info, Nick. With that in mind, I"m removing this as a "release blocker" for now. Steven, would it be possible to upgrade gcc on your system? We'll have to deal with the Tiger buildbot separately.

[vstinner]

I created the issue bpo-28099 "Drop Mac OS X Tiger support in Python 3.6.

[ambv]

Benjamin, what's the rationale behind switching those to inline functions? Does it improve runtime performance or build speed? If not, I don't understand why the additional complexity.

[vstinner]

For the specific case of Dtrace, #define are just fine. But for
unicodeobject.h, it would be nice to be able to use inline functions.

[stevendaprano]

Ned, I know my system is old so I understand if 3.6 no longer supports
gcc 4.1. Tell me what minimum version I should use (4.8?) and I'll find
a way to upgrade or use another system.

Shouldn't the build system explicitly report that the compiler is too
old, and suggest a minimum version, rather than just dump some random
C-level error in the user's lap? There's a big gap between "able to run
configure/make" and "able to diagnose configure/make errors" where users
are currently not well provided for. It is very frustrating to deal with
these sorts of errors especially when most of the help you get from the
internet is "run make distclean and try again".

[ned-deily]

[discussion with Steven about compiler versions taken off-list]

[benjaminp]

On Mon, Sep 12, 2016, at 10:34, Łukasz Langa wrote:

Łukasz Langa added the comment:

Benjamin, what's the rationale behind switching those to inline
functions? Does it improve runtime performance or build speed? If not, I
don't understand why the additional complexity.

Macros for the stub case of dtrace are "fine" because they're so simple.
I mainly converted them to see if it would cause problems (e.g., this
bug report) before seeking further expansion.

Many of CPython's uglier macros could be replaced by inline funtions.
Inline functions provide the same performance benefit of macros while
improving type safety, debugability, and avoiding notorious macro quirks
like double evaluation.

[vadmium]

For replacing macros, I think “static inline” may be fine, even with older compilers. Maybe these PyDTrace_ functions could also be static inline, since they were originally macros. Or do they really need to be linkable inline functions?

[rgbecker]

Don't want to add too much noise, but this issue also affects the manylinux project build compiler (gcc 4.8.2).

[vstinner]

Don't want to add too much noise, but this issue also affects the manylinux project build compiler (gcc 4.8.2).

Would it be possible to upgrade the "manylinux" compiler (take a more recent GCC version)?

[njsmith]

Would it be possible to upgrade the "manylinux" compiler (take a more recent GCC version)?

No, it's possible :-(. 4.8.2 is the very most modern version of GCC you can use if you want to build binaries to run on CentOS/RHEL 5. (And "binaries should run on CentOS/RHEL 5" is the definition of manylinux1.)

I am a bit confused that gcc 4.8.2 is having trouble with cpython 3.6.0b2, though -- supposedly anything newer than gcc 4.3 should be fine with it. And yet. One possibility is that something funny is going on inside the build scripts Robin's using and that Python's ./configure is somehow finding and using the platform compiler (gcc 4.1) even though the first "gcc" in $PATH is 4.8.2, which would make this a false alarm.

[rgbecker]

I executed gcc --version (&cc --version) in the do_cpython_build function immediately prior to the make -j2 that builds python noth show 4.8.2.

I see the exact same errors as in the initial report. If the makefile or the configure is doing something special then I guess I have to work around that. A possibility is that the CFLAGS="-Wformat" in the environment or the configure argument --disable-shared is having some effect.

I have made very few changes to the build scripts.

[njsmith]

@rgbecker: Are you able to pull out the config.log generated by running python's ./configure script, and post that somewhere?

[vstinner]

"Don't want to add too much noise, but this issue also affects the manylinux project build compiler (gcc 4.8.2)."

Can you elaborate on these issues? Are you getting the same errors than the error described in the initial message with GCC 4.2? If not, you may open a new issue to track compilation issues of Python 3.6 on GCC 4.8.

[rgbecker]

Hi njs,

my manylinux diffs
https://www.reportlab.com/media/manylinux-diff.txt

full output of the docker command
docker build -f Dockerfile-x86_64 -t rl/manylinux-x86_64 . &> ~/tmp/ttt

https://www.reportlab.com/media/manylinux-docker-run-output.txt

the end showing the 3.6b2 config and the failure
https://www.reportlab.com/media/manylinux-docker-run-python-3.6b2.txt

[tds333]

Please check if you have enabled the compiler as default by enabling the devtoolset on CentOS 5.

I have compiled Python 3.6b2 on Ubuntu 14.04 with gcc 4.8.4 without any problems.
Therefore gcc 4.8.2 should not be that different.

Also keep in mind the default gcc for CentOS 5 will fail because it is to old.

[tds333]

Also verified on CentOS 6.8 with devtoolset2 installed, gcc 4.8.2
Python 3.6b2 builds fine, all unittest ok.
This is the same devtoolset as used on CentOS 5 manylinux1.
Have no CentOS 5 VM available to do more tests. But gcc 4.8 is able to build Python 3.6.

[rgbecker]

tds333, the config says that 4.8.2 is being used,

configure:3902: gcc --version >&5
gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

but perhaps the make is doing something else or the build goes wrong because shared is disabled.

[njsmith]

Yeah, the config.log there clearly shows that configure is using gcc 4.8.2 on the failed builds. Very odd.

Stepping back for a moment, is there any point in continuing to debug this? Given Benjamin's comment up-thread:

I mainly converted them to see if it would cause problems

My impression is that restricting inline functions to 'static inline' only is the best plan for 3.6 anyway, since AFAIK that really does work on all the compilers that people are likely to run into, while non-static inline has problems with the default compilers on both CentOS 5 and OS X.

[ned-deily]

"while non-static inline has problems with the default compilers on both CentOS 5 and OS X."

The changes introduced in 3.6 prevent compilation with gcc4.0 which was the default Apple-supplied compiler on Mac OS X 10.4 (Tiger). 3.6 currently compiles correctly with gcc4.2, the default on Mac OS X 10.5 (Leopard).

[njsmith]

3.6 currently compiles correctly with gcc4.2, the default on Mac OS X 10.5 (Leopard).

So to summarize our current understanding: gcc 4.3 added support for C99 inline, which explains why gcc 4.2 works and gcc 4.8 doesn't.

WTF is going on.

[benjaminp]

Any different results with CFLAGS="-fno-gnu89-inline"?

[rgbecker]

I'm not able to access my work computer, I'll try later today at home.

[rgbecker]

benjamin.peterson: I tried adding that option -fno-gnu89-inline conditionally for the 3.6.0b2 build, but it goes wrong in config with an error

configure:3913: $? = 4
configure:3933: checking whether the C compiler works
configure:3955: gcc -Wformat -fno-gnu89-inline conftest.c >&5
cc1: error: -fno-gnu89-inline is only supported in GNU99 or� C99 mode

The full docker output is here

https://www.reportlab.com/media/manylinux-docker-run-output-2.txt

[rgbecker]

After some searching I tried adding -std=gnu99 and the config goes through OK, but running make produces

[root@d3cce9786c2e Python-3.6.0b2]# make -j2
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wformat -fno-gnu89-inline -std=gnu99 -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I. -I./Include -DPy_BUILD_CORE -o Programs/python.o ./Programs/python.c
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wformat -fno-gnu89-inline -std=gnu99 -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I. -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
/tmp/cc3Mfloe.s: Assembler messages:
/tmp/cc3Mfloe.s:1575: Error: symbol stat64' is already defined /tmp/cc3Mfloe.s:1595: Error: symbol lstat64' is already defined
/tmp/cc3Mfloe.s:1615: Error: symbol fstat64' is already defined /tmp/cc3Mfloe.s:1635: Error: symbol fstatat64' is already defined
make: *** [Programs/python.o] Error 1
make: *** Waiting for unfinished jobs....
/tmp/ccOoXPte.s: Assembler messages:
/tmp/ccOoXPte.s:1575: Error: symbol stat64' is already defined /tmp/ccOoXPte.s:1595: Error: symbol lstat64' is already defined
/tmp/ccOoXPte.s:1615: Error: symbol fstat64' is already defined /tmp/ccOoXPte.s:1635: Error: symbol fstatat64' is already defined
make: *** [Parser/acceler.o] Error 1
[root@d3cce9786c2e Python-3.6.0b2]#

[rgbecker]

for what it's worth I tried reverse patching

https://hg.python.org/cpython/rev/63ae310b60ff/

and

https://hg.python.org/cpython/rev/2f77a9f0b9d6/

in the manylinux docker and the make then proceeds fine with one warning at the end

*** WARNING: renaming "_sqlite3" since importing it failed: build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so: undefined symbol: sqlite3_stmt_readonly

Following modules built successfully but were removed because they could not be imported:
_sqlite3

[ned-deily]

"*** WARNING: renaming "_sqlite3" since importing it failed: build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so: undefined symbol: sqlite3_stmt_readonly"

That's a different issue: most likely you are building with an old version of libsqlite3. See, for example: ghaering/pysqlite#85

[python-dev]

New changeset fd9a4bd16587 by Benjamin Peterson in branch '3.6':
mark dtrace stubs as static inline; remove stubs
https://hg.python.org/cpython/rev/fd9a4bd16587

[benjaminp]

I changed the dtrace stubs to static inline. Probably should reopen an investigation for 3.7. I would like to have exportable inlines.