t/porting/libperl.t failure with GCC 12 and -flto #20518

ntyni · 2022-11-16T20:17:01Z

This is a bug report for perl from ntyni@debian.org,
generated with the help of perlbug 1.43 running under perl 5.37.6.

Description

As reported by Matthias Klose in https://bugs.debian.org/985884
t/porting/libperl.t fails when perl is built with link time optimization
(LTO).

Steps to Reproduce

On current Debian unstable, with gcc (Debian 12.2.0-9) 12.2.0
and perl v5.37.5-162-g52917b368f

$ ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make -j4 && make test
[...]
t/porting/libperl ................................................ # Failed test 3 - has data const symbols at porting/libperl.t line 322
# Failed test 4 - has PL_no_mem at porting/libperl.t line 323
FAILED at test 3
[...]
Failed 1 test out of 2502, 99.96% okay.
	porting/libperl.t


$ ./perl -Ilib t/porting/libperl.t 
# $^O = linux
# $Config{archname} = x86_64-linux
# $Config{cc} = cc
# libperl = ../libperl.a
# nm = /usr/bin/nm
# nm_style = gnu
# nm_opt = 
# command: "/usr/bin/nm  ../libperl.a 2>libperl564590 |"
ok 1 - has object util.o
ok 2 - has text Perl_croak in util.o
not ok 3 - has data const symbols
# Failed test 3 - has data const symbols at t/porting/libperl.t line 322
not ok 4 - has PL_no_mem
# Failed test 4 - has PL_no_mem at t/porting/libperl.t line 323
# nocommon = 0
ok 5 - has PL_hash_seed_w
ok 6 - has PL_ppaddr
ok 7 - has undefined symbols
ok 8 - uses chmod (doio.o)
ok 9 - uses dlopen (DynaLoader.o)
ok 10 - uses exp (pp.o)
ok 11 - uses getenv (DynaLoader.o locale.o perl.o perlio.o regcomp.o toke.o util.o)
ok 12 - uses sigaction (mg.o util.o)
ok 13 - uses socket (doio.o)
ok 14 - uses time (perl.o pp_sys.o util.o)
ok 15 - uses no atoi ()
ok 16 - uses no atol ()
ok 17 - uses no atoll ()
ok 18 - uses no fgets ()
ok 19 - uses no gets ()
ok 20 - uses no sprintf ()
ok 21 - uses no strcat ()
ok 22 - uses no strcpy ()
ok 23 - uses no strncat ()
ok 24 - uses no strncpy ()
ok 25 - uses no strtol ()
ok 26 - uses no strtoq ()
ok 27 - uses no strtoul ()
ok 28 - uses no system ()
ok 29 - uses no tmpfile ()
ok 30 - uses no vsprintf ()
ok 31 - no S_ exports
1..31

Ubuntu is currently disabling the failing tests FWIW.

Flags

category=core
severity=low

Perl configuration

Site configuration information for perl 5.37.6:

Configured by ntyni at Wed Nov 16 19:23:28 UTC 2022.

Summary of my perl5 (revision 5 version 37 subversion 6) configuration:
  Commit id: 52917b368fe204d0670f020d5f0f3ad9ec236e01
  Platform:
    osname=linux
    osvers=5.19.0-2-amd64
    archname=x86_64-linux
    uname='linux carme 5.19.0-2-amd64 #1 smp preempt_dynamic debian 5.19.11-1 (2022-09-24) x86_64 gnulinux '
    config_args='-des -Dusedevel -Dccflags=-flto -Dldflags=-flto'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cc'
    ccflags ='-flto -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-flto -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='12.2.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags ='-flto -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/x86_64-linux-gnu /usr/lib /usr/lib64
    libs=-lpthread -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -ldl -lm -lcrypt -lutil -lc
    libc=/lib/x86_64-linux-gnu/libc.so.6
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.36'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'


---
@INC for perl 5.37.6:
    lib
    /usr/local/lib/perl5/site_perl/5.37.6/x86_64-linux
    /usr/local/lib/perl5/site_perl/5.37.6
    /usr/local/lib/perl5/5.37.6/x86_64-linux
    /usr/local/lib/perl5/5.37.6

---
Environment for perl 5.37.6:
    HOME=/home/ntyni
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/usr/lib/libeatmydata
    LOGDIR (unset)
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

The text was updated successfully, but these errors were encountered:

ntyni · 2022-11-16T20:19:26Z

I haven't tested if this is GCC 12 specific. It was reported quite a while ago so it might well happen with earlier GCC versions too.

jkeenan · 2022-11-16T23:22:41Z

I was able to reproduce these failures on Linux (Debian 11) using gcc-10, the default C-compiler:

$ git describe
v5.37.5-171-gd47ed502d5
$ sh ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make test_porting
...
$ cd t;./perl harness -v porting/libperl.t; cd -

ok 1 - has object util.o
ok 2 - has text Perl_croak in util.o
not ok 3 - has data const symbols
# Failed test 3 - has data const symbols at porting/libperl.t line 322
not ok 4 - has PL_no_mem
# Failed test 4 - has PL_no_mem at porting/libperl.t line 323
ok 5 - has PL_hash_seed_w
...
ok 31 - no S_ exports
Failed 2/31 subtests 

Test Summary Report
-------------------
porting/libperl.t (Wstat: 0 Tests: 31 Failed: 2)
  Failed tests:  3-4
Files=1, Tests=31,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.05 cusr  0.05 csys =  0.12 CPU)
Result: FAIL

And, for what it's worth, I tried this with gcc10 as the C-compiler on FreeBSD-12. make failed on both blead and at v5.36.0. Example:

[perlmonger: perl] $ git describe
v5.37.5-171-gd47ed502d5
[perlmonger: perl] $ sh ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto -Dcc=gcc10 && make test_porting
...
Updating 'mktables.lst'
./miniperl -Ilib -MExtUtils::Miniperl -e 'writemain(\"perlmain.c", @ARGV)' DynaLoader 
gcc10 -c -DPERL_CORE -flto -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -std=c99 -O2 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings perlmain.c
gcc10 -o perl -flto -fstack-protector-strong -L/usr/local/lib  perlmain.o   libperl.a `cat ext.libs` -lpthread -ldl -lm -lcrypt -lutil -lc
/usr/local/bin/ld: /tmp//perl.Qqd7KG.ltrans0.ltrans.o: in function `xs_init':
<artificial>:(.text+0x6): undefined reference to `boot_DynaLoader'
/usr/local/bin/ld: <artificial>:(.text+0x10): undefined reference to `Perl_newXS'
/usr/local/bin/ld: /tmp//perl.Qqd7KG.ltrans0.ltrans.o: in function `main':
<artificial>:(.text.startup+0x23): undefined reference to `Perl_sys_init3'
/usr/local/bin/ld: <artificial>:(.text.startup+0x29): undefined reference to `PL_do_undump'
/usr/local/bin/ld: <artificial>:(.text.startup+0x31): undefined reference to `perl_alloc'
/usr/local/bin/ld: <artificial>:(.text.startup+0x49): undefined reference to `perl_construct'
/usr/local/bin/ld: <artificial>:(.text.startup+0x4f): undefined reference to `PL_perl_destruct_level'
/usr/local/bin/ld: <artificial>:(.text.startup+0x6c): undefined reference to `PL_exit_flags'
/usr/local/bin/ld: <artificial>:(.text.startup+0x72): undefined reference to `perl_parse'
/usr/local/bin/ld: <artificial>:(.text.startup+0x7d): undefined reference to `PL_sig_name'
/usr/local/bin/ld: <artificial>:(.text.startup+0x9a): undefined reference to `PL_sig_name'
/usr/local/bin/ld: <artificial>:(.text.startup+0xa5): undefined reference to `PL_sig_num'
/usr/local/bin/ld: <artificial>:(.text.startup+0xad): undefined reference to `Perl_rsignal_state'
/usr/local/bin/ld: <artificial>:(.text.startup+0xb4): undefined reference to `PL_csighandlerp'
/usr/local/bin/ld: <artificial>:(.text.startup+0xc0): undefined reference to `Perl_rsignal'
/usr/local/bin/ld: <artificial>:(.text.startup+0xd8): undefined reference to `perl_destruct'
/usr/local/bin/ld: <artificial>:(.text.startup+0xe7): undefined reference to `perl_free'
/usr/local/bin/ld: <artificial>:(.text.startup+0xec): undefined reference to `Perl_sys_term'
/usr/local/bin/ld: <artificial>:(.text.startup+0x100): undefined reference to `perl_run'
collect2: error: ld returned 1 exit status
*** Error code 1

Stop.
make: stopped in /usr/home/jkeenan/gitwork/perl

(I've never tried those 2 switches myself, so I don't know what to expect.)

jkeenan · 2022-11-16T23:26:12Z

This is a bug report for perl from ntyni@debian.org, generated with the help of perlbug 1.43 running under perl 5.37.6.

Description

As reported by Matthias Klose in https://bugs.debian.org/985884 t/porting/libperl.t fails when perl is built with link time optimization (LTO).

Steps to Reproduce

On current Debian unstable, with gcc (Debian 12.2.0-9) 12.2.0 and perl v5.37.5-162-g52917b368f
$ ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make -j4 && make test

Can you describe, in lay person's terms, what -flto is intended to accomplish? I can't recall seeing that used in any of our smoke-testing, nor can I recall previous bug reports with that.

jkeenan · 2022-11-17T00:02:02Z

On Linux (Debian 11), using gcc-10 as the C-compiler, I checked out various tags and built with this configuration:

sh ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto

As was the case with HEAD and v5.36.0, I got two failures in t/porting/libperl.t at each of v5.32.1 and v5.28.3. At v5.24.1 the build failed to complete.

So that leads me to ask: What evidence do you have that these configuration switches ever built and tested successfully?

gregoa · 2022-11-17T07:37:53Z

On Wed, 16 Nov 2022 15:26:22 -0800, James E Keenan wrote: Can you describe, in lay person's terms, what `-flto` is intended to accomplish? I can't recall seeing that used in any of our smoke-testing, nor can I recall previous bug reports with that.

I'm sure that @ntyni can explain this in a better way, but for starters, the announcement at <https://lists.debian.org/debian-devel/2022/06/msg00092.html> and the wiki link there might be helpful. Cheers, gregor

…

-- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- BOFH excuse #11: magnetic interference from money/credit cards

iabyn · 2022-11-17T12:10:59Z

On Wed, Nov 16, 2022 at 12:17:18PM -0800, Niko Tyni wrote: $ ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make -j4 && make test [...] t/porting/libperl ................................................ # Failed test 3 - has data const symbols at porting/libperl.t line 322 # Failed test 4 - has PL_no_mem at porting/libperl.t line 323 FAILED at test 3 [...] Failed 1 test out of 2502, 99.96% okay. porting/libperl.t not ok 3 - has data const symbols # Failed test 3 - has data const symbols at t/porting/libperl.t line 322 not ok 4 - has PL_no_mem # Failed test 4 - has PL_no_mem at t/porting/libperl.t line 323 # nocommon = 0

The proximate cause of these two failures is a lack of readonly symbols in libperl.a. In a normal build: $ /usr/bin/nm libperl.a | grep ' [Rr] ' 0000000000000e00 r array_passed_to_stat 00000000000003e0 r bodies_by_type 00000000000087c0 r custom_op_register_vtbl ... $ /usr/bin/nm libperl.a | grep PL_no_mem U PL_no_mem 0000000000000850 R PL_no_mem $ In a -flto build: $ /usr/bin/nm libperl.a | grep ' [Rr] ' $ $ /usr/bin/nm libperl.a | grep PL_no_mem U PL_no_mem 00000000 D PL_no_mem $ (Note that PL_no_mem has changed from R to D.) I don't know enough about LTO to know whether not having such symbols is to be expected and thus whether libperl.t should just skip those two tests when perl is configured with -flto.

…

-- More than any other time in history, mankind faces a crossroads. One path leads to despair and utter hopelessness. The other, to total extinction. Let us pray we have the wisdom to choose correctly. -- Woody Allen

jkeenan · 2022-11-17T14:11:21Z

On Wed, 16 Nov 2022 15:26:22 -0800, James E Keenan wrote: Can you describe, in lay person's terms, what -flto is intended to accomplish? I can't recall seeing that used in any of our smoke-testing, nor can I recall previous bug reports with that.
I'm sure that @ntyni can explain this in a better way, but for starters, the announcement at https://lists.debian.org/debian-devel/2022/06/msg00092.html and the wiki link there might be helpful. Cheers, gregor

@gregoa, thank you very much for those URLs. Here is my summary/paraphrase of what they say:

Link time optimizations are an optimization that helps with a single digit percent number optimizing both for smaller size, and better speed. ... The proposal is to turn on LTO by default on most 64bit release architectures. In test rebuilds, there were 373 packages (dd-list in the wiki page) found not to build with link time optimizations for various reasons. [Perl is one of those with test failures.] The idea is to fix as many of these as possible, and then change the packaging for the others to just turn off LTO in the package build.

So this ticket is only incidentally a report of test failures. It's actually a request for a new functionality, i.e., build perl in a way not previously requested so that Debian (and presumably other distributions) can move in a new direction. (I note this because the original posting led me to believe this was a regression in perl; it's not.)

Until reading this ticket I had never heard of link-time optimizations, so I don't know how we should proceed. @ntyni, @gregoa, @jmdh, could one of you write up a post for the perl5-porters mailing list about Debian's move to LTO? That list has much wider visibility than these GH issues and it would be good for our readership to learn more about this new initiative.

Thank you very much.
Jim Keenan

ntyni · 2022-11-17T20:07:42Z

So this ticket is only incidentally a report of test failures. It's actually a request for a new functionality, i.e., build perl in a way not previously requested so that Debian (and presumably other distributions) can move in a new direction. (I note this because the original posting led me to believe this was a regression in perl; it's not.)

Indeed. Apologies for the misunderstanding.

Until reading this ticket I had never heard of link-time optimizations, so I don't know how we should proceed. @ntyni, @gregoa, @jmdh, could one of you write up a post for the perl5-porters mailing list about Debian's move to LTO? That list has much wider visibility than these GH issues and it would be good for our readership to learn more about this new initiative.

Okay; mail sent.

Niko

Leont · 2022-11-18T02:38:01Z

I don't know enough about LTO to know whether not having such symbols is
to be expected and thus whether libperl.t should just skip those two tests
when perl is configured with -flto.

I think we should change the test. We should not make assumptions about in which section the compiler puts that variable.

jkeenan · 2022-11-21T15:03:20Z

I don't know enough about LTO to know whether not having such symbols is
to be expected and thus whether libperl.t should just skip those two tests
when perl is configured with -flto.

I think we should change the test. We should not make assumptions about in which section the compiler puts that variable.

Could you explain what these "sections" are? Also, would it be possible to get a p.r. for skipping the tests?

Leont · 2022-11-21T15:55:45Z

Could you explain what these "sections" are?

They're the different parts of the ELF binary format for executables and libraries. There is one section that is readonly and pre-initialized at compile time and the test expects a the variable to be put in there.

Apparently when using link-time optimization it shows up in the "initialized data section" (used for static writable variables) for the library, even if it ends up read-only in the resulting executable.

I don't know if this is a bug in nm or a side-effect of this decision being made at link time. In either case, we should either:

skip() if $ccflags =~ /-flto/
remove the test entirely

For lto builds with gcc and clang the PL_no_mem symbol as listed by nm is flagged as "D" (writable data) and "T" (text, aka code) respectively. Looking at the final generated executable PL_no_mem does end up in the .rodata (read only data) section, so it might be worth adding a separate test for that. Fixes Perl#20518

tonycoz · 2022-11-22T22:38:40Z

We should not make assumptions about in which section the compiler puts that variable.

I can see it being useful as a "does the toolchain work as we expect" test, but maybe not a test that we run in the released perl.

jkeenan · 2022-11-23T13:48:49Z

We should not make assumptions about in which section the compiler puts that variable.

I can see it being useful as a "does the toolchain work as we expect" test, but maybe not a test that we run in the released perl.

Do we currently have any porting tests in the core distribution which are for the purpose of verifying "that the toolchain works as we expect"? If not, how (and where) would we implement such a test?

tonycoz · 2022-11-23T22:26:41Z

libperl.t is the only one that I can see that specifically tests the build toolchain.

Several tests end up testing the underlying operating system, and have caused problems when the OS had bugs (the recent Solaris PERLIO=stdio bug, and DragonflyBSD error reporting for tty functions for example).

Leont · 2022-11-24T01:25:19Z

I can see it being useful as a "does the toolchain work as we expect" test, but maybe not a test that we run in the released perl.

I suspect the failing test is the sort of test that is much more likely to fail due to this sort of fragility than because of an actual toolchain failure. A lot of these things are much harder to check for reliably than one may think.

For LTO builds with gcc and clang the PL_no_mem symbol as listed by nm is flagged as "D" (writable data) and "T" (text, aka code) respectively. Looking at the final generated executable PL_no_mem does end up in the .rodata (read only data) section, so it might be worth adding a separate test for that. Fixes Perl#20518

For LTO builds with gcc and clang the PL_no_mem symbol as listed by nm is flagged as "D" (writable data) and "T" (text, aka code) respectively. Looking at the final generated executable PL_no_mem does end up in the .rodata (read only data) section, so it might be worth adding a separate test for that. Fixes #20518

For LTO builds with gcc and clang the PL_no_mem symbol as listed by nm is flagged as "D" (writable data) and "T" (text, aka code) respectively. Looking at the final generated executable PL_no_mem does end up in the .rodata (read only data) section, so it might be worth adding a separate test for that. Fixes Perl#20518

ntyni added the Needs Triage label Nov 16, 2022

jkeenan added Feature Request and removed Needs Triage labels Nov 17, 2022

tonycoz mentioned this issue Nov 27, 2022

skip checking categorization of libperl symbols for LTO builds #20553

Merged

tonycoz closed this as completed in #20553 Nov 28, 2022

daniel-pfeiffer mentioned this issue Aug 13, 2023

bignum >= 0.51 breaks BigInt >> #21370

Open

jdv mentioned this issue Jul 23, 2024

Use of uninitialized value $_[0] in defined operator #22423

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

t/porting/libperl.t failure with GCC 12 and -flto #20518

t/porting/libperl.t failure with GCC 12 and -flto #20518

ntyni commented Nov 16, 2022

ntyni commented Nov 16, 2022

jkeenan commented Nov 16, 2022

jkeenan commented Nov 16, 2022 •

edited

Loading

jkeenan commented Nov 17, 2022

gregoa commented Nov 17, 2022 via email

iabyn commented Nov 17, 2022 via email

jkeenan commented Nov 17, 2022

ntyni commented Nov 17, 2022 •

edited

Loading

Leont commented Nov 18, 2022

jkeenan commented Nov 21, 2022

Leont commented Nov 21, 2022

tonycoz commented Nov 22, 2022

jkeenan commented Nov 23, 2022

tonycoz commented Nov 23, 2022

Leont commented Nov 24, 2022 •

edited

Loading

t/porting/libperl.t failure with GCC 12 and -flto #20518

t/porting/libperl.t failure with GCC 12 and -flto #20518

Comments

ntyni commented Nov 16, 2022

ntyni commented Nov 16, 2022

jkeenan commented Nov 16, 2022

jkeenan commented Nov 16, 2022 • edited Loading

jkeenan commented Nov 17, 2022

gregoa commented Nov 17, 2022 via email

iabyn commented Nov 17, 2022 via email

jkeenan commented Nov 17, 2022

ntyni commented Nov 17, 2022 • edited Loading

Leont commented Nov 18, 2022

jkeenan commented Nov 21, 2022

Leont commented Nov 21, 2022

tonycoz commented Nov 22, 2022

jkeenan commented Nov 23, 2022

tonycoz commented Nov 23, 2022

Leont commented Nov 24, 2022 • edited Loading

jkeenan commented Nov 16, 2022 •

edited

Loading

ntyni commented Nov 17, 2022 •

edited

Loading

Leont commented Nov 24, 2022 •

edited

Loading