configure --with-lto builds fail when CC=clang on Linux, requires gold linker

[gpshead]

BPO	28015
Nosy	@gpshead, @vstinner, @benjaminp, @ned-deily, @serge-sans-paille, @stratakis, @miss-islington
PRs	bpo-28015: Support LTO build with clang #9908 [3.7] bpo-28015: Support LTO build with clang (GH-9908) #10089 [3.7] bpo-28015: Support LTO build with clang (GH-9908) #10089 [3.6] bpo-28015: Support LTO build with clang (GH-9908) #10922 [3.6] bpo-28015: Support LTO build with clang (GH-9908) #10922 [3.6] bpo-28015: Support LTO build with clang (GH-9908) #10922

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-10-29.13:39:40.115>
created_at = <Date 2016-09-08.07:45:09.625>
labels = ['3.8', 'build', '3.7']
title = 'configure --with-lto builds fail when CC=clang on Linux, requires gold linker'
updated_at = <Date 2018-12-09.08:14:19.265>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2018-12-09.08:14:19.265>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-10-29.13:39:40.115>
closer = 'vstinner'
components = ['Build']
creation = <Date 2016-09-08.07:45:09.625>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 28015
keywords = ['patch']
message_count = 11.0
messages = ['274984', '327816', '327817', '328403', '328405', '328826', '329002', '329042', '329421', '331414', '331415']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'vstinner', 'benjamin.peterson', 'ned.deily', 'serge-sans-paille', 'cstratak', 'miss-islington']
pr_nums = ['9908', '10089', '10089', '10922', '10922', '10922']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue28015'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[gpshead]

The error message you will see when building may look something like:

clang -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fprofile-instr-generate -flto Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -lpthread -ldl -lutil -o Parser/pgen
/usr/bin/ld: /usr/lib/llvm-3.8/bin/../lib/LLVMgold.so: error loading plugin: /usr/lib/llvm-3.8/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:793: recipe for target 'Parser/pgen' failed
make[2]: *** [Parser/pgen] Error 1

as seen on ubuntu 16.04.

Related to https://bugs.launchpad.net/ubuntu/+source/llvm-toolchain-snapshot/+bug/1254970 which suggests we also need -B/usr/lib/gold-ld in our LDFLAGS and/or CFLAGS when using clang on Linux with LTO.

testing with it set in both still runs into errors; someone with clang LTO experience will need to untangle this.

(is this related to why --with-lto doesn't work on MacOS as well?)

[serge-sans-paille]

Looks like a package dependency issue: installing llvm-dev package should fix the problem. Or in that particular case llvm-3.8-dev.

[vstinner]

I confirm that it's currently broken. Test on Fedora 28, clang version 6.0.1 (tags/RELEASE_601/final):

$ ./configure --with-pydebug CC=clang --with-lto  && make
(...)
checking for x64 gcc inline assembler... yes
checking whether float word ordering is bigendian... unknown
configure: error: 

Unknown float word ordering. You need to manually preset
ax_cv_c_float_words_bigendian=no (or yes) according to your system.

[vstinner]

New changeset 5ad36f9 by Victor Stinner (serge-sans-paille) in branch 'master':
bpo-28015: Support LTO build with clang (GH-9908)
5ad36f9

[miss-islington]

New changeset 69a3f15 by Miss Islington (bot) in branch '3.7':
bpo-28015: Support LTO build with clang (GH-9908)
69a3f15

[vstinner]

The issue has been fixed in Python 3.7 and master.

If someone is volunteer to backport it to older version, please go ahead :-) But according to serge-sans-paille, it's non trivial and I'm not sure that it's worth it. Sorry, usually we focus on the master branch for "new features".
#9908 (comment)

So I close the issue.

Thanks serge-sans-paille for this nice enhancement!

[stratakis]

The issue with the 3.6 branch is https://bugs.python.org/issue31625 which hasn't been backported to 3.6. Would it make sense to backport it?

I have the backports ready locally but not sure if pushing https://bugs.python.org/issue31625 to 3.6 is desirable.

[benjaminp]

Probably that's fine, but can bpo-31625 be easily worked around by, e.g., using make AR=llvm-ar RANLIB=llvm-ranlib?

[stratakis]

The workaround indeed works.

[ned-deily]

New changeset f83ee47 by Ned Deily (stratakis) in branch '3.6':
bpo-28015: Support LTO build with clang (GH-9908) (GH-10922)
f83ee47

[ned-deily]

See also bpo-31354, bpo-35351, and bpo-35257.