[binutils] internal error aborting

[DaGiHu]

Description / Steps to reproduce the issue

Error occurs building a rather complex application linking to ffmpeg, cef, sdl2, ssp libraries.
This error happened overnight, just after updating packages with pacman. I update Msys2 on a almost daily basis.

The code is exactly the same that linked correctly a few days ago, before the pacman update.

Expected behavior

Linkage expected to finish and .exe being generated.

Actual behavior

This errors appears and build stops:
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: please report this bug
collect2.exe: error: ld returned 1 exit status

Verification

• I have verified that my MSYS2 is up-to-date before submitting the report (see https://www.msys2.org/docs/updating/)

Windows Version

MINGW64_NT-10.0-19045

MINGW environments affected

• MINGW64
• MINGW32
• UCRT64
• CLANG64
• CLANG32
• CLANGARM64

Are you willing to submit a PR?

No response

[Biswa96]

The exact steps to reproduce the issue would be helpful.

[DaGiHu]

I'm sorry being so concise. I'm building the application using a IDE. This is its output:

Despite it reports 0 errors, the exe file is not generated.
Thank you.

C:/Msys64/mingw64/bin/mingw32-make.exe -j8 SHELL=cmd.exe -e -f  Makefile
"----------Building project:[ MyApp - Debug ]----------"
mingw32-make[1]: Entering directory 'C:/_DEV_PRJ/dev_cl/MyApp'
C:/Msys64/mingw64/bin/g++.exe -o c:/MyApp/MyApp @"MyApp.txt" -L. -LC:/Msys64/home/ion/ffmpeg/ffmpeg64/lib -LC:/_DEV/CEF/build/libcef_dll_wrapper -LC:/_DEV/CEF/Release -LC:/Msys64/mingw64/lib  -lcef_dll_wrapper -lcef -lavcodec.dll -lavdevice.dll -lavfilter.dll -lavformat.dll -lavutil.dll -lpostproc.dll -lswresample.dll -lswscale.dll -lSDL2.dll -lssp  -v
Using built-in specs.
COLLECT_GCC=C:\Msys64\mingw64\bin\g++.exe
COLLECT_LTO_WRAPPER=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-12.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/include --libexecdir=/mingw64/lib --enable-bootstrap --enable-checking=release --with-arch=nocona --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++,jit --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts --enable-libstdcxx-time --disable-libstdcxx-pch --enable-lto --enable-libgomp --disable-multilib --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev10, Built by MSYS2 project' --with-bugurl=https://github.com/msys2/MINGW-packages/issues --with-gnu-as --with-gnu-ld --disable-libstdcxx-debug --with-boot-ldflags=-static-libstdc++ --with-stage1-ldflags=-static-libstdc++
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (Rev10, Built by MSYS2 project)
MAKEFLAGS=e -j8  -- $(MAKEOVERRIDES)
COMPILER_PATH=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/;C:/Msys64/mingw64/bin/../lib/gcc/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/
LIBRARY_PATH=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/;C:/Msys64/mingw64/bin/../lib/gcc/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib/../lib/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../
COLLECT_GCC_OPTIONS='-o' 'c:/MyApp/MyApp.exe' '-L.' '-LC:/Msys64/home/ion/ffmpeg/ffmpeg64/lib' '-LC:/_DEV/CEF/build/libcef_dll_wrapper' '-LC:/_DEV/CEF/Release' '-LC:/Msys64/mingw64/lib' '-v' '-shared-libgcc' '-mtune=generic' '-march=nocona' '-dumpdir' 'c:/MyApp/MyApp.'
 C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/collect2.exe -plugin C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/liblto_plugin.dll -plugin-opt=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/lto-wrapper.exe -plugin-opt=-fresolution=C:\Users\ion\AppData\Local\Temp\ccEPoTpb.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lkernel32 -m i386pep -Bdynamic -o c:/MyApp/MyApp.exe C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib/crt2.o C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/crtbegin.o @C:\Users\ion\AppData\Local\Temp\cc2xmwLc -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0 -LC:/Msys64/mingw64/bin/../lib/gcc -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib/../lib -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../.. @C:\Users\ion\AppData\Local\Temp\cc0cD2be -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lkernel32 -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lkernel32 C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib/default-manifest.o C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/crtend.o
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: please report this bug
collect2.exe: error: ld returned 1 exit status
mingw32-make[1]: *** [MyApp.mk:83: c:/MyApp/MyApp] Error 1
mingw32-make[1]: Leaving directory 'C:/_DEV_PRJ/dev_cl/MyApp'
mingw32-make: *** [Makefile:5: All] Error 2
=== build completed successfully (0 errors, 0 warnings) ===

[DaGiHu]

Let me add that ffmpeg and cef libraries are updated, build and maintained as local in Msys2 by myself. Those have worked for years in this project with zero issues.

[rouault]

I confirm there is a regression with mingw-w64-x86_64-binutils==2.40-1

Steps to reproduce from mingw64 installed from msys2-x86_64-20230127.exe installer:

pacman -Suy
pacman -S base-devel git mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake mingw-w64-x86_64-ccache \
            mingw-w64-x86_64-pcre mingw-w64-x86_64-xerces-c mingw-w64-x86_64-zstd mingw-w64-x86_64-libarchive \
            mingw-w64-x86_64-geos mingw-w64-x86_64-libspatialite mingw-w64-x86_64-proj \
            mingw-w64-x86_64-cgal mingw-w64-x86_64-libfreexl mingw-w64-x86_64-hdf5 mingw-w64-x86_64-netcdf mingw-w64-x86_64-poppler mingw-w64-x86_64-postgresql \
            mingw-w64-x86_64-libgeotiff mingw-w64-x86_64-libpng mingw-w64-x86_64-libtiff mingw-w64-x86_64-openjpeg \
            mingw-w64-x86_64-python-pip mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-pytest mingw-w64-x86_64-python-setuptools mingw-w64-x86_64-python-lxml
git clone https://github.com/OSGeo/gdal
cd gdal
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DGDAL_BUILD_OPTIONAL_DRIVERS=OFF -DOGR_BUILD_OPTIONAL_DRIVERS=OFF -DGDAL_USE_EXTERNAL_LIBS=OFF -DGDAL_USE_POPPLER=ON -DGDAL_ENABLE_DRIVER_PDF=ON -DBUILD_PYTHON_BINDINGS=ON
cmake --build . --target pytest_runner

errors out with

FAILED: autotest/pytest_runner.exe
cmd.exe /C "cd . && C:\dev\msys64\mingw64\bin\c++.exe -fvisibility=hidden   -O3 -DNDEBUG  autotest/C
MakeFiles/pytest_runner.dir/pytest_runner.cpp.obj -o autotest\pytest_runner.exe -Wl,--out-implib,aut
otest\libpytest_runner.dll.a -Wl,--major-image-version,0,--minor-image-version,0  C:/dev/miniconda3/
libs/python39.lib  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomd
lg32 -ladvapi32 && cd ."
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: please report this bug
collect2.exe: error: ld returned 1 exit status

Downgrading to mingw-w64-x86_64-binutils==2.39.2 makes things work again

This is the "minimum" reproducer I managed to get with GDAL. I also get a similar error when linking the libgdal-32.dll itself on a more feature full build, as can been seen in the log of https://github.com/OSGeo/gdal/actions/runs/4093595228/jobs/7059529472

Wondering if there's a relationship with upstream bug https://sourceware.org/bugzilla/show_bug.cgi?id=30079 although that one is about Debian's binutils-mingw-w64-x86-64 2.39.90.20230110-1+10.3

[Biswa96]

@rouault In your case, the program is trying to link with C:/dev/miniconda3/libs/python39.lib file. The library name does not look like from msys2 compiled python package. Is it possible to reproduce the issue with mingw python library?

Wondering if there's a relationship with upstream bug

In that upstream issue, author also provided a import library which also looks different than binutils generated import library. I can not recall the exact name right now. If you open any libXYZ.dll.a the format looks different between binutils and clang/msvc.

[rouault]

Is it possible to reproduce the issue with mingw python library?

@Biswa96 The key to reproduce the issue is apparently to link against a .lib not built with mingw64. This can be python39.lib from conda-forge, or the C:/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/SDK/Lib/x64/msodbcsql17.lib library from Microsoft ODBC Driver 17 for SQL Server (to be installed with https://go.microsoft.com/fwlink/?linkid=2223304 and then select the "ODBC driver for SQL Server SDK" optional component during installation)

Failed job https://github.com/rouault/gdal/actions/runs/4096319505/jobs/7063871277 links against msodbcsql17.lib (no Python involved during the linking phase, and this job would only use the one from mingw64), whereas successful job https://github.com/rouault/gdal/actions/runs/4096322267/jobs/7063875659 does not link against it (cf rouault/gdal@7c2c620 for the difference between the 2 job which is the disabling of linking against msodbcsql17.lib)

I can also reproduce this locally in my Windows VM with (after installing msodbcsql17.lib as detailed above):

cmake .. -DPython_ROOT_DIR=/c/dev/msys64/mingw64 -DCMAKE_BUILD_TYPE=Release -DGDAL_BUILD_OPTIONAL_DRIVERS=OFF -DOGR_BUILD_OPTIONAL_DRIVERS=OFF -DGDAL_USE_EXTERNAL_LIBS=OFF -DGDAL_USE_POPPLER=OFF -DGDAL_ENABLE_DRIVER_PDF=OFF -DGDAL_USE_MSSQL_ODBC=ON -DOGR_ENABLE_DRIVER_MSSQLSPATIAL=ON -DGDAL_USE_ODBC=ON
cmake --build . --target GDAL

leads to

[3/3] Linking CXX shared library libgdal-32.dll
FAILED: libgdal-32.dll libgdal.dll.a
cmd.exe /C "cd . && C:\dev\msys64\mingw64\bin\c++.exe -fvisibility=hidden -O3 -DNDEBUG  -Wl,--no-und
efined -shared -o libgdal-32.dll -Wl,--out-implib,libgdal.dll.a -Wl,--major-image-version,32,--minor
-image-version,3 @CMakeFiles\GDAL.rsp  && cd ."
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: please report this bug
collect2.exe: error: ld returned 1 exit status

On the previous configured build, downgrading to binutils 2.39-2 and rebuilding leads to a successful build:

pacman -U /c/dev/msys64/var/cache/pacman/pkg/mingw-w64-x86_64-binutils-2.39-2-any.pkg.tar.zst
cmake --build . --target GDAL

[2/2] Linking CXX shared library libgdal-32.dll

Or keeping 2.40-1 but configuring cmake with -DOGR_ENABLE_DRIVER_MSSQLSPATIAL=OFF to ignore msodbcsql17.lib

[Biswa96]

The key to reproduce the issue is apparently to link against a .lib

It is not a supported case. I am not familiar with the internals of binutils. I have talked with other developers. The gist is that the primary supported library format is different in binutils. That is, binutils does not properly handle new library format (here .lib ones) since many versions. If you want to link with that .lib file clang would be better choice now.

But I agree this is clearly an issue. And using binutils generated import library workarounds the issue.

[DaGiHu]

By reading your comments on the cause of the issue I've finally been able to compile my app circumventing the canonical .lib linkage I've used for years by linking directly to its associated .dll.

In any case I consider this a major issue because I reckon there must be a zillion users having .lib import libraries in their linking stage.

As of today this issue is already been treated as a bug in Sourceware's Bugzilla:
https://sourceware.org/bugzilla/show_bug.cgi?id=30079

Thanks a lot you for your hints and support.

[lazka]

I've bisected it, see https://sourceware.org/bugzilla/show_bug.cgi?id=30079#c1

[lazka]

A potential fix is in the repo, please try again now.

[DaGiHu]

It works with .lib files again. Thank you very much, Lazka.

[rouault]

I confirm this also fixes the linking issue of GDAL against msodbcsql17.lib

[lazka]

ok, thanks everyone for figuring this out and testing etc.

[pfusik]

Thanks a lot for fixing it! It was failing my build today.

[mstorsjo]

The key to reproduce the issue is apparently to link against a .lib

It is not a supported case. I am not familiar with the internals of binutils. I have talked with other developers. The gist is that the primary supported library format is different in binutils. That is, binutils does not properly handle new library format (here .lib ones) since many versions.

This statement gives the wrong picture IMO - the word "supported" has many different possible interpretations.

Yes, binutils uses a different import library format normally, and binutils' native format is the one that does get most testing. But binutils does contain some amount of code to handle linking against the MSVC (and LLVM) style import libraries. There have been multiple issues with this mode throughout the years - but effort has been spent on making it work better. Last I checked (before this regression - thanks @lazka for handling it! I just started looking into it, when I see that you've already handled it!) it worked for most of the common cases.

Bugs/regressions withstanding, the only missing feature I'm aware of, is handling of weak aliases. These aren't usually used in normal import libraries for user DLLs, but are heavily used in mingw-w64-crt. So binutils should be able to handle linking against third party DLLs with MSVC-style import libraries, but it doesn't work entirely on top of a mingw-w64 sysroot built with LLVM (i.e. the clang64 environment in msys2).

So, is this "supported"? I guess that depends on your definition of the word supported. Less tested, for sure - but since there actually has been effort to improve things (e.g. bminor/binutils-gdb@b53b12c and bminor/binutils-gdb@46bbc1c), I think it's important to give the right picture. The remaining barrier between "hopefully works" and "actually works in practice" is real world usage and track record of it. And we won't get that by just discouraging usage.