R fails to build with LTO

[simon-king-jena]

I took sage-5.0.beta9, replaced the ppl package by the one from #12672, added the cloog-ppl package from #12666, replaced the glpk package by the one from #12703, added a new libelf spkg, added the mpc and gcc spkgs from #12369 and modified the gcc spkg such that the gcc is built with support of loop optimization (graphite) and lto.

Then, I tried to build Sage with rather high optimization:

king@mpc622:/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9$ echo $CFLAGS
-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing -flto
king@mpc622:/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9$ echo $CXXFLAGS
-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing -flto
king@mpc622:/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9$ echo $LDFLAGS
-flto

It was needed to temporarily remove -flto in order to build Singular (see #12738). Later, R failed to build - see this install log.

CC: @nexttime

Component: packages: standard

Keywords: R LTO r-project

Reviewer: Michael Orlitzky

Issue created by migration from https://trac.sagemath.org/ticket/12741

[simon-king-jena]

comment:1

I don't know R at all. In particular, in contrast to Singular at #12738, I did not look at the source code and have no idea how compiler or linker flags are dealt with in R.

What I find suspicious in the log is that only "gcc" is named in lines such as

gcc -std=gnu99 -Wl,--export-dynamic -fopenmp  -L/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9/local/lib/ -flto -o R.bin Rmain.o -L../../lib -lR 

Can we be sure that really the "correct" gcc, namely the one in SAGE_LOCAL/... created by the gcc spkg is used? Namely, my system gcc does not support LTO.

[nexttime]

comment:2

Just noticed that there's this suspicious line in your R spkg log:

checking command to parse /usr/bin/nm -B output from gcc object... failed

which indicates that the autotools the R spkg was made with are [too] old. (Note that I did get the same with vanilla GLPK upstream source.)

The error looks a bit strange though, but the log isn't really that verbose, at least at first glance.

Slightly unrelated: Looks like R builds its own zlib, which it shouldn't, since Sage provides it.

[nexttime]

comment:3

Replying to @nexttime:

Just noticed that there's this suspicious line in your R spkg log:

checking command to parse /usr/bin/nm -B output from gcc object... failed

which indicates that the autotools the R spkg was made with are [too] old. (Note that I did get the same with vanilla GLPK upstream source.)

Although I do have the same in my successful build log (with GCC 4.6.3 and -flto)... 8/

[nexttime]

comment:4

P.S.: Did you also try on another machine?

What's the version of your binutils (which also include ld, i.e., what does e.g. ld --version give)?

[simon-king-jena]

comment:5

Replying to @nexttime:

P.S.: Did you also try on another machine?

What's the version of your binutils (which also include ld, i.e., what does e.g. ld --version give)?

$ ld --version
GNU ld (GNU Binutils for Debian) 2.20.1-system.20100303
Copyright 2009 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

Please don't say that (after the gcc spkg) Sage also needs an ld spkg...

[nexttime]

comment:6

Replying to @simon-king-jena:

Please don't say that (after the gcc spkg) Sage also needs an ld spkg...

For LTO to fully work (or, better, be fully applicable), unfortunately yes, since -fuse-linker-plugin requires either gold or GNU ld version >= 2.21. B)

(But I've successfully built Sage with [just] -flto with exactly the same ld version as well.)

[simon-king-jena]

comment:7

Replying to @nexttime:

Replying to @simon-king-jena:

Please don't say that (after the gcc spkg) Sage also needs an ld spkg...

For LTO to fully work (or, better, be fully applicable), unfortunately yes, since -fuse-linker-plugin requires either gold or GNU ld version >= 2.21. B)

I don't understand: At what point do I need to provide -fuse-linker-plugin? It has not been part of my C(XX)FLAGS and has not been a configuration option to the gcc from the spkg, it I recall correctly.

[simon-king-jena]

comment:8

Replying to @simon-king-jena:

I don't understand: At what point do I need to provide -fuse-linker-plugin?

Sorry, now I recall your comments from #12703, where you say that some (not all) packages would need -fuse-linker-plugin added to LDFLAGS in order to work with -lto.

However, how problematic is that my ld is slightly too old? Would it refuse to work with -fuse-linker-plugin? Will it ignore that flag?

I guess Sage shouldn't also add another 60MB or so for a binutils spkg, isn't it?

[nexttime]

comment:9

Replying to @simon-king-jena:

Replying to @simon-king-jena:

I don't understand: At what point do I need to provide -fuse-linker-plugin?

Sorry, now I recall your comments from #12703, where you say that some (not all) packages would need -fuse-linker-plugin added to LDFLAGS in order to work with -lto.

Explicitly specifying -fuse-linker-plugin seems to only be necessary in case you updated binutils (or your ld, or used some older version during the GCC build) after building GCC.

From the GCC manual:

-fuse-linker-plugin

Enables the use of a linker plugin during link-time optimization. This option relies on the linker plugin support in linker that is available in gold or in GNU ld 2.21 or newer.

This option enables the extraction of object files with GIMPLE bytecode out of library archives. This improves the quality of optimization by exposing more code to the link-time optimizer. This information specifies what symbols can be accessed externally (by non-LTO object or during dynamic linking). Resulting code quality improvements on binaries (and shared libraries that use hidden visibility) are similar to `-fwhole-program`. See `-flto` for a description of the effect of this flag and how to use it.

This option is enabled by default when LTO support in GCC is enabled and GCC was configured for use with a linker supporting plugins (GNU ld 2.21 or newer or gold).

It's IMHO a minor issue anyway.

However, how problematic is that my ld is slightly too old? Would it refuse to work with -fuse-linker-plugin? Will it ignore that flag?

Nope, GCC will exit with an error if you specify -fuse-linker-plugin but the present linker doesn't support it.

I guess Sage shouldn't also add another 60MB or so for a binutils spkg, isn't it?

We need a standard LaTeX / TeXLive package, too. ;-)

No, seriously, we could of course also offer an optional binutils package, but that's not that important, although the GCC package may refuse to build on systems with too incapable versions of the programs provided by binutils.

[nexttime]

comment:10

I now get a funny Maxima build error:

In the last (link) step, ECL (or some Maxima build script) calls GCC with $LDFLAGS, once split into words ("-foo" "-bar" "-baz"), as it should, and in addition once with all of them passed as a single argument ("-foo -bar -baz"), which of course leads to an error (unrecognized command line option).

[simon-king-jena]

comment:11

Replying to @nexttime:

I now get a funny Maxima build error:

Interesting. That worked for me...

I broke my promise and did create a binutils-2.22.spkg, by the way. I hope that it will help solve my problems with -flto. Not that I really have problems - I just want to see that optimization does not make Sage slower (which it currently does!).

[nexttime]

comment:12

Replying to @nexttime:

I now get a funny Maxima build error:

In the last (link) step, ECL (or some Maxima build script) calls GCC with $LDFLAGS, once split into words ("-foo" "-bar" "-baz"), as it should, and in addition once with all of them passed as a single argument ("-foo -bar -baz"), which of course leads to an error (unrecognized command line option).

This is now (fixed at) #12759.

[nexttime]

comment:13

Simon, does using LTO still fail for you with GCC 4.7.1?

(According to the release notes, there've been major improvements, including bug fixes, to link-time optimization.)

Jeroen has provided an optional GCC 4.7.1 spkg at #13150, although that doesn't support using LTO out of the box I guess...

[simon-king-jena]

comment:14

Replying to @nexttime:

Simon, does using LTO still fail for you with GCC 4.7.1?

(According to the release notes, there've been major improvements, including bug fixes, to link-time optimization.)

Jeroen has provided an optional GCC 4.7.1 spkg at #13150, although that doesn't support using LTO out of the box I guess...

I didn't try (and I don't know if I will try soon, because it might very well be that I'll be off to holiday tomorrow).

[kcrisman]

Changed keywords from R LTO to R LTO r-project

[nexttime]

comment:16

For the record:

With FSF GCC 4.7.2, for me R still fails to build without LTO (i.e., with -O3), while it does build with -O3 -flto (which isn' t that surprising, but at least funny regarding this ticket).

Presumably some GCC optimizer bug, but hard to track down as R segfaults while byte-compiling. (This happens on Linux x86 and x86_64.)

[mkoeppe]

comment:21

outdated, should close

[orlitzky]

Reviewer: Michael Orlitzky