Building debug version on Windows 11 with gcc12: inlining failed in call to 'always_inline' 'S_amagic_i_ncmp'

[hakonhagland]

I am trying to build a debug version of perl from blead on Windows 11 using MinGW-w64 and gcc 12.2 from https://winlibs.com/ (using the MSVCRT runtime library).

After downloading WinLibs to C:\Winlibs64-Gcc12-msvcrt, I needed to rename gnu make binary to gmake.exe:

> copy C:\Winlibs64-Gcc12-msvcrt\mingw64\bin\mingw32-make.exe C:\Winlibs64-Gcc12-msvcrt\mingw64\bin\gmake.exe

This is gcc version 12.2.0, and gnu make version 4.3:

# make sure core.autocrlf in the git config is not set to true before checkout below:
c:\PerlWinLib> git clone https://github.com/Perl/perl5.git  
[...]

c:\PerlWinLib>cd perl5\win32

c:\PerlWinLib\perl5\win32>where gcc
C:\Winlibs64-Gcc12-msvcrt\mingw64\bin\gcc.exe

c:\PerlWinLib\perl5\win32>gcc --version
gcc (MinGW-W64 x86_64-ucrt-posix-seh, built by Brecht Sanders) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

C:\PerlWinLib\perl5\win32>where gmake
C:\Winlibs64-Gcc12-msvcrt\mingw64\bin\gmake.exe

c:\PerlWinLib\perl5\win32>gmake --version
GNU Make 4.3
Built for x86_64-w64-mingw32
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

To build perl I needed to set the environment variable CCHOME:

c:\PerlWinLib\perl5\win32>set CCHOME=C:\Winlibs64-Gcc12-msvcrt\mingw64

Now, I tried to run make like this:

> powershell "gmake CFG=Debug INST_TOP=c:\perl-debug 2>&1 | tee gmake.log"
# CCTYPE=GCC
# GCCBIN=gcc
# GCCVER=12.2.0
# GCCTARGET=x86_64-w64-mingw32
# GCCCROSS=
# WIN64=define
# ARCHITECTURE=x64
# ARCHNAME=MSWin32-x64-multi-thread
# MAKE=gmake
if not exist "mini" mkdir "mini"
copy config_H.gc config.h
        1 file(s) copied.
gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -Og -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -omini\toke.o  ..\toke.c
gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -Og -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -omini\regcomp.o  ..\regcomp.c

[...]

gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -Og -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -omini\perly.o  ..\perly.c
gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -Og -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -omini\pp_sort.o  ..\pp_sort.c
In function 'dynprep',
    inlined from 'S_sortsv_flags_impl' at ..\pp_sort.c:354:20,
    inlined from 'sortsv_amagic_i_ncmp' at ..\pp_sort.c:568:5:
..\pp_sort.c:1224:1: error: inlining failed in call to 'always_inline' 'S_amagic_i_ncmp': function not considered for inlining
 1224 | S_amagic_i_ncmp(pTHX_ SV *const a, SV *const b)
      | ^~~~~~~~~~~~~~~
..\pp_sort.c:184:14: note: called from here
  184 |     sense = (cmp(aTHX_ *b, *(b+1)) > 0);
      |              ^~~~~~~~~~~~~~~~~~~~~

[...]

gmake: *** [GNUmakefile:1421: mini\pp_sort.o] Error 1

The complete log is here: gmake.log

[sisyphus]

I haven't yet managed to reproduce that failure.
I'm wondering if that setting of the CCHOME environment variable stuck.
Could you re-run the build with CCHOME set via an argument given to 'gmake' - ie CCHOME=C:\Winlibs64-Gcc12-msvcrt\mingw64 ?
(Or, alternatively, edit the CCHOME specification in GNUmakefile.)

If it still throws that same error then I'll make some more attempts to reproduce it.

Cheers,
Rob

[hakonhagland]

Could you re-run the build with CCHOME set via an argument given to 'gmake'

@sisyphus Yes, good idea. But it still gives the same error:

>gmake realclean
>powershell "gmake CFG=Debug CCHOME=C:\Winlibs64-Gcc12-msvcrt\mingw64 INST_TOP=c:\perl-debug 2>&1 | tee gmake.log"

[...]

gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -Og -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -omini\pp_sort.o  ..\pp_sort.c
In function 'dynprep',
    inlined from 'S_sortsv_flags_impl' at ..\pp_sort.c:354:20,
    inlined from 'sortsv_amagic_i_ncmp' at ..\pp_sort.c:568:5:
..\pp_sort.c:1224:1: error: inlining failed in call to 'always_inline' 'S_amagic_i_ncmp': function not considered for inlining
 1224 | S_amagic_i_ncmp(pTHX_ SV *const a, SV *const b)
      | ^~~~~~~~~~~~~~~
..\pp_sort.c:184:14: note: called from here
  184 |     sense = (cmp(aTHX_ *b, *(b+1)) > 0);
      |              ^~~~~~~~~~~~~~~~~~~~~

[...]

[sisyphus]

OK ... thanks for checking; sorry that it was a bum steer.
I can see the same when (in the powershell console) I run:

> gmake CFG=Debug CCHOME=C:\winlibs-gcc-1220\mingw64 INST_TOP=c:\perl-debug 2>&1 | tee gmake.log

(AFAICT, that's essentially the same command as @hakonhagland ran, except that I've modified CCHOME to be correct for my environment.)

The make process begins with:

gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -Og  -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -omini\toke.o  ..\toke.c

and, after a while, terminates with the same error as reported by @hakonhagland.
The build that I had initially tried, had specified optimization to be -g -O2, not -g -Og, and that seems to account for the difference.

There's a recent (Aug 28) commit affecting win32/GNUmakefile, that's probably relevant here.
cc51139 changes the -DDEBUGGING optimization from -g -O2 to -g -Og.

I'm not very experienced in -DDEBUGGING builds.
Could someone (@khwilliamson, @xenu, @Leont ) comment on the validity of that alteration ?

The same commit implemented another change from -O2 to-Os
That part seems fine. It's just the change from -g -O2 to -g -Og that I'm wondering about.
It "feels" right .... but then it's apparently leading to this failure.

Cheers,
Rob

[sisyphus]

This issue seems to be specific to gcc version 12.
I see the same problem when I switch from gcc-12.2.0 to gcc-12.1.0, but gcc-11.3.0 is quite happy to accept the -g -Og argument.

Has a similar issue not been seen on Linux -DDEBUGGING builds that use gcc-12 ?

It seems that, on Windows at least, the "always_inline" attribute is unacceptable to gcc-12 when -Og optimization level is specified.
I don't know whose bug that is, or what the correct fix is.

The obvious workaround is to not specify -Og optimization level under those circumstances,
However, I don't know what affect that has on the resultant debugging capability.

@hakonhagland , the required optimization level can be specified on the command line, - which makes it fairly easy to experiment with different settings..
I find that specifying OPTIMIZE=-g or OPTIMIZE="-g -O1" or OPTIMIZE="-g -O2" as a command line argument all work fine.

I looked at https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html to see if I could find an optimization flag that could be added to -g -Og and make the build succeed.
I even experimented by including a couple of the flags listed there ... but no luck.

Cheers,
Rob

[hakonhagland]

I find that specifying OPTIMIZE=-g or OPTIMIZE="-g -O1" or OPTIMIZE="-g -O2" as a command line argument all work fine.

@sisyphus Interesting! I now tried with -O0 and the linker fails instead:

>gmake realclean
>powershell "gmake CFG=Debug OPTIMIZE=\"-g -O0\" CCHOME=C:\Winlibs64-Gcc12-msvcrt\mingw64 INST_TOP=c:\perl-debug 2>&1 | tee gmake.log"

[...]

gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -O0 -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_IS_MINIPERL -omini\win32thread.o  win32thread.c
gcc -c  -I.\include -I. -I.. -DWIN32 -DWIN64 -DDEBUGGING -DPERLDLL -DPERL_CORE -g -O0 -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_IS_MINIPERL -omini\fcrypt.o  fcrypt.c
g++ -mconsole -o ..\miniperl.exe  -g -L"c:\perl-debug\lib\CORE" -L"C:\Winlibs64-Gcc12-msvcrt\mingw64\lib" -L"C:\Winlibs64-Gcc12-msvcrt\mingw64\x86_64-w64-mingw32\lib" -L"C:\Winlibs64-Gcc12-msvcrt\mingw64\lib\gcc\x86_64-w64-mingw32\12.2.0" mini\toke.o mini\regcomp.o mini\regexec.o mini\op.o mini\sv.o mini\pp.o mini\pp_ctl.o mini\pp_sys.o mini\pp_pack.o mini\pp_hot.o mini\gv.o mini\perl.o mini\utf8.o mini\dump.o mini\hv.o mini\av.o mini\builtin.o mini\caretx.o mini\deb.o mini\doio.o mini\doop.o mini\dquote.o mini\globals.o mini\mro_core.o mini\locale.o mini\keywords.o mini\mathoms.o mini\mg.o mini\numeric.o mini\pad.o mini\peep.o mini\perly.o mini\pp_sort.o mini\reentr.o mini\run.o mini\scope.o mini\taint.o mini\time64.o mini\universal.o mini\util.o mini\miniperlmain.o mini\perlio.o mini\win32.o mini\win32sck.o mini\win32thread.o mini\fcrypt.o  -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
c:/winlibs64-gcc12-msvcrt/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
mini\regexec.o: in function `Perl_re_intuit_start':
C:\PerlWinLib\perl5-debug\win32/../regexec.c:885: undefined reference to `Perl_re_printf'
c:/winlibs64-gcc12-msvcrt/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
C:\PerlWinLib\perl5-debug\win32/../regexec.c:916: undefined reference to `Perl_re_printf'
c:/winlibs64-gcc12-msvcrt/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
C:\PerlWinLib\perl5-debug\win32/../regexec.c:939: undefined reference to `Perl_re_printf'
c:/winlibs64-gcc12-msvcrt/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
C:\PerlWinLib\perl5-debug\win32/../regexec.c:946: undefined reference to `Perl_re_printf'
c:/winlibs64-gcc12-msvcrt/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
C:\PerlWinLib\perl5-debug\win32/../regexec.c:994: undefined reference to `Perl_re_printf'
c:/winlibs64-gcc12-msvcrt/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
mini\regexec.o:C:\PerlWinLib\perl5-debug\win32/../regexec.c:1014: more undefined references to `Perl_re_printf' follow
collect2.exe: error: ld returned 1 exit status
gmake: *** [GNUmakefile:1207: ..\lib\buildcustomize.pl] Error 1

The complete log is here: gmake.log

[sisyphus]

I hadn't tried building with -g -O0 because I read somewhere that one shouldn't specify -O0 on debugging builds.

I can't reproduce the issue you're seeing.
The 'gmake' step works fine for me with 'CFG=Debug' and 'OPTIMIZE="-g -O0".
Maybe the source has become corrupted from some previous build ?
Normally a build on Windows will begin with the compilation of toke.c and regcomp.c, but those 2 compilations were apparently skipped - going by the build log that was provided.
@hakonhagland , could you try it again, using freshly derived source.

However, IMO, the important questions are "why does -g -Og not work, and what to do about it ?".
I don't think it matters all that much whether debugging perls can/cannot be built with optimization levels -g -O0 and -g -O1 optimizations. (But I could be wrong about that.)

It would be nice to hear from someone who knows about such things.

Cheers,
Rob

[hakonhagland]

Maybe the source has become corrupted from some previous build ?

@sisyphus Yes, that seems to be the case. But why did not gmake realclean work? and why did git status show no files changed (except for the generated gmake.log that I included here) after calling gmake realclean. Isn't that strange?

[sisyphus]

I always run 'make distclean' to clean up the perl source and 'make realclean' to clean up a module's source.
On Windows, you might get away with 'make realclean' to clean up the perl source. (I'm not sure, and too lazy to check.)

I think what probably happened in your case is that you started a new build after a successful cleanup but then, after toke.c and regcomp.c had been compiled, you killed that build (perhaps because you realized that you had not selected the 'gmake' arguments you intended).
Then you've gone straight to another run of 'gmake' (using different arguments), but neither toke.c nor regcomp.c were recompiled with the new arguments because their targets already existed.
At least, that's the sort of stuff-up that I sometimes make - and it's a scenario that fits with that log report.

I'm pretty weak on git, but I'm guessing that git status would report only on changes to files that git is tracking - and running 'gmake' should not alter the contents of any of those files AFAIK.

Cheers,
Rob

[hakonhagland]

you killed that build

@sisyphus Maybe yes! Anyway, after building successfully with -g -O0 (log files are here: gmake.log, gmake-test.log, gmake-install.log), I tried to install cpanm with this newly installed perl. However, this failed. See miyagawa/cpanminus#652.

[sisyphus]

Does the same problem arise if you try to install other modules using cpan, or is it just with App::Cpanminus ?

[hakonhagland]

Does the same problem arise if you try to install other modules using cpan

@sisyphus I think not. I tried with Path::Tiny and it installed many of the dependencies fine, but some tests failed for the module itself. See dagolden/Path-Tiny#258

[hakonhagland]

I tried to install cpanm with this newly installed perl. However, this failed
cpan : process_begin: CreateProcess(NULL, pl2bat.bat blib\script\cpanm, ...) failed.

It seems like pl2bat.bat was not installed. See also msys2/MINGW-packages#1463. Should I create a new issue for this here?

[sisyphus]

I think we should first determine 2 things:

1. why building with CFG=Debug OPTIMIZE="-g -Og" is failing
2. what to do about that

@hakonhagland, unfortunately neither you nor I have the answer to either question, and apparently everyone else is either disinterested or has nothing to contribute.
Or maybe it's just that this thread has become too long for people to wade through.

In a couple of days from now, perl-5.37.5 will be released.
I think we should concentrate our efforts on it (rather than on the everchanging git repo), probably close this issue and create a new one titled something like "[Win32] -DDEBUGGING perl-5.37.5 fails to build with -Og, using gcc-12".

In the meantime, if you really need a functional debugging perl I would suggest using an MSVCRT build of gcc-11.3.0 from Winlibs, and build with CFG=Debug OPTIMIZE="-g -Og".
From my (non-rigorous) testing, that should be fine, and would be a handy reference to have.
(That's one of the builds that I'll be testing when 5.37.5 devel release is made.)

Cheers,
Rob

PS If you do try that build with gcc-11.3.0 and strike trouble, please do report it here (in this issue), as it's something I'd like to be informed of.

[tonycoz]

Duplicate of #19776

[hakonhagland]

PS If you do try that build with gcc-11.3.0 and strike trouble, please do report it here

@sisyphus Yes I tried it now, and it installs fine here (building with CFG=Debug OPTIMIZE="-g -Og") : gmake.log, gmake-test.log, gmake-install.log. However, I get the same issue when trying to install cpanm with this perl. Do you get this issue also?

[sisyphus]

With gcc-11.3.0 (building with CFG=Debug OPTIMIZE=-g -Og") I find that the cpan utility is unusable.
For all modules that I try to install using cpan, the process terminates with something like:

.....
Configuring M/MI/MIYAGAWA/App-cpanminus-1.7046.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a gmake-style Makefile
Writing Makefile for App::cpanminus
Writing MYMETA.yml and MYMETA.json
  MIYAGAWA/App-cpanminus-1.7046.tar.gz
  D:\debugperl-5.37.4-d-1130\bin\perl.exe Makefile.PL -- OK
Running make for M/MI/MIYAGAWA/App-cpanminus-1.7046.tar.gz
The system cannot find the path specified.
  MIYAGAWA/App-cpanminus-1.7046.tar.gz
  C:\_32\make\bin\make.EXE -- NOT OK

Since all modules seem to be affected by this, I believe the bugto not be in App::cpanminus.
Furthermore, App-cpanminus-1.7046 builds, tests and installs fine if I do it manually - ie cd to the "/.cpan/build/App-cpanminus-1.7046-0" directory and run perl Makefile.PL followed by gmake test and then gmake install.
Having done that, App::cpanminus then functions correctly - so it's worth going to the trouble of installing it.

Since cpan -i ... is supposed to emulate that manual process, and since that manual process works fine, I suggest that the bug is in the cpan utility.

Cheers,
Rob

[sisyphus]

Duh ... 'cpan' was working across my various builds of perl on Windows 11, but now it's not working on any of them (except for cygwin).
This is a newly acquired box and I've apparently recently done something to clobber 'cpan', but not 'cpanm'.
Until I fix that, my current experiences with 'cpan' are irrelevant.

[hakonhagland]

It seems like pl2bat.bat was not installed.

I noticed something interesting. It seems that INST_TOP is not respected for some files when running gmake install. When installing the last version with gcc-11.3.0 and msvcrt, I used the following command line

> powershell "gmake CFG=Debug OPTIMIZE="-g -Og" INST_TOP=c:\perl-debug2 2>&1 | tee gmake.log"

Then I ran "gmake test", and then "gmake install" which should then install into directory "c:\perl-debug2" according to the above command line. But from the log file I see the following at line 2098:

[...]
xcopy /f /r /i /d /y "bin\*.bat" c:\perl\bin\
C:\PerlWinLib\perl5-debug3\win32\bin\exetype.bat -> C:\perl\bin\exetype.bat
C:\PerlWinLib\perl5-debug3\win32\bin\perlglob.bat -> C:\perl\bin\perlglob.bat
C:\PerlWinLib\perl5-debug3\win32\bin\pl2bat.bat -> C:\perl\bin\pl2bat.bat
C:\PerlWinLib\perl5-debug3\win32\bin\runperl.bat -> C:\perl\bin\runperl.bat
C:\PerlWinLib\perl5-debug3\win32\bin\search.bat -> C:\perl\bin\search.bat
5 File(s) copied
[...]

notice that it copies pl2bat.bat to C:\perl\bin instead of to C:\perl-debug2\bin as it should.

[sisyphus]

Then I ran "gmake test", and then "gmake install" which should then install into directory "c:\perl-debug2" according to the above command line.

For "gmake install", I think you'll need to run gmake INST_TOP=c:\perl-debug2 install".
I think (untested) that if you run "gmake install" without the INST_TOP setting, it will just revert to the default because that's what the GNUmakefile still specifies.

When I build perl with command line configuration options, I always pass the same arguments to "gmake test" and "gmake install" as I passed to "gmake".
That's probably overdoing it a bit, but it ensures that the "test" and "install" phases DWIM.

I'm guessing that's how stuff got into your c:\perl\bin.
I can't actually see how it could be anything other than that, but let me know if that was a bad guess and I'll look further into it.

The GNUmakefile sets:

INST_SCRIPT	= $(INST_TOP)$(INST_VER)\bin

and then runs:

$(XCOPY) "bin\*.bat" $(INST_SCRIPT)\$(NULL)

So you can see that those batch files will end up in whatever is specified by $(INST_TOP)$(INST_VER}\bin.

Cheers,
Rob

[hakonhagland]

if you run "gmake install" without the INST_TOP setting, it will just revert to the default because that's what the GNUmakefile still specifies

@sisyphus Yes, you are right. I tested this now and that seems to solve the issue with pl2bat.bat missing and I am now able to install cpanm for this debug perl.

[xenu]

I reported the bug in GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107931