Golang support broken after v20.1.8

[roffe]

It's not possible to use llvm-mingw to build golang applications after v20.1.8

if upgraded past that CGO fails with a single line of output:

cgo: cannot parse $WORK\b293\_cgo_.o as ELF, Mach-O, PE or XCOFF

I'm on Windows 11 with Go 1.25.0

[mstorsjo]

Hmm, so it works with LLVM 20.1.8, but fails with LLVM 21.1.x?

In that case, it sounds to me like llvm/llvm-project@be5c96b could be a potential culprit; CC @cjacek.

[roffe]

@mstorsjo correct, sorry for bad wording. 20.1.8 works fine but 21.1.0 & 21.1.1 fails with the mentioned error

[mstorsjo]

To test the hypothesis that this is the culprit, I started two builds, using a version of LLVM before and after this change - see https://github.com/mstorsjo/llvm-mingw/actions/runs/17958759696 and https://github.com/mstorsjo/llvm-mingw/actions/runs/17958778529. In ~8 hours or so, those should have completed and should be able to download artifacts of these builds, to see if the guess was right, or if we need to try something more to pinpoint the issue.

[roffe]

Awesome. I'll test them when I get up tomorrow and report back

[mstorsjo]

Ok, those builds failed due to trying to build for arm64ec with an older version of LLVM that didn't support everything we need there; I made changes to skip arm64ec and restarted new builds at https://github.com/mstorsjo/llvm-mingw/actions/runs/17966127239 and https://github.com/mstorsjo/llvm-mingw/actions/runs/17966130372.

[roffe]

https://github.com/mstorsjo/llvm-mingw/actions/runs/17966127239 <-- this build works
https://github.com/mstorsjo/llvm-mingw/actions/runs/17966130372 <-- this one also works o.O
https://github.com/mstorsjo/llvm-mingw/actions/runs/17968649501 <-- 21.1.2 does not work, gives parse error

on all 3 above i downloaded "windows-ucrt-x86_64-toolchain"

i tried downloading 21.1.0 RC-1 (llvm-mingw-20250717-ucrt-x86_64.zip) it also gives the "cannot parse" error

[mstorsjo]

Ok, thanks for testing! So my guess was wrong - but we have a more narrow interval to bisect.

Thanks for indicating which package you use for testing; it's less effort to produce just that one rather than all of them.

I'll try to set up a way for building another set of packages to narrow the range down further. With a couple rounds of that we should have the breaking commit pinpointed.

[kleisauke]

Is this the same issue as golang/go#75219?

[roffe]

@kleisauke i just pulled down trunk of Go and built it from source

go version go1.26-devel_371c1d2fcb Wed Sep 24 06:21:53 2025 -0700 windows/amd64

clang --version
clang version 21.1.2 (https://github.com/llvm/llvm-project.git b708aea0bc7127adf4ec643660699c8bcdde1273)
Target: x86_64-w64-windows-gnu
Thread model: posix
InstalledDir: C:/llvm/bin
Configuration file: C:/llvm/bin/x86_64-w64-windows-gnu.cfg

Software compiles fine without CGO error now so the problem seems to be Go as you suspected.

@mstorsjo sorry for sending you on a ghost chase when it's clearly something in Go that was causing it.

Hopefully others that has had the same problem will find this issue and know what to do now.

I guess it will fix itself as well when the next Go release comes out :)

[mstorsjo]

Ok, so that makes it kinda plausible that this is related to golang/go#75219. There, I asked for a way to reproduce it; if I have a sample C file I can compile and inspect for some detail in the output, it should be quite straightforward for me to bisect it locally.

If you're still interested in pinpointing it, I fired off a couple of shorter builds that build only the package that you want. (Or, hopefully it does that...) See https://github.com/mstorsjo/llvm-mingw/actions/workflows/build.yml?query=branch%3Atest-pinpoint-cgo-regression for the list of builds. First I have two build jobs for validating the build setup for the good/bad end points (with 3873 commits inbetween), and then I have three builds that build three different versions inbetween. Once they complete (should take 2-3 hours), which ones of these that succeed/fail should let us narrow the range down to ~960 commits. With another couple of rounds of such bisection, we should have culprit found in a day or two :-)

[roffe]

I can absolutely try the new builds and see if we can pinpoint where the problem is :)

I'll report back the results of the different builds with the same Go version I've done the previous tests with.

[roffe]

https://github.com/mstorsjo/llvm-mingw/actions/runs/17987957928 #1953 - works
https://github.com/mstorsjo/llvm-mingw/actions/runs/17988025661 #1954 - fails
https://github.com/mstorsjo/llvm-mingw/actions/runs/17988087626 #1955 - fails
https://github.com/mstorsjo/llvm-mingw/actions/runs/17988096787 #1956 - fails
https://github.com/mstorsjo/llvm-mingw/actions/runs/17988107871 #1957 - fails

[mstorsjo]

Thanks! I fired off three new builds at https://github.com/mstorsjo/llvm-mingw/actions/workflows/build.yml?query=branch%3Atest-pinpoint-cgo-regression - ETA ~1h25min.