x/build: windows builders and try-bots mingw is out of date

[alexbrainman]

Builders and try-bots have gcc 5.1 installed on them. According to https://sourceforge.net/projects/mingw-w64/files/ we could get gcc 8.1.

I discovered problem with our builders while fixing #23649. For example, current debug/pe tests fail, if gcc 8.1 is used, but pass fine on our builders. CL 197977 fixes broken debug/pe tests, but I cannot even verify it on builders / try-bots.

I propose we install gcc 8.1 so we can test #23649 fix, and keep it fixed. gcc 5.1 is many years old, we should use recent gcc on our builders.

I used these files from https://sourceforge.net/projects/mingw-w64/files/

https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/threads-win32/seh/x86_64-8.1.0-release-win32-seh-rt_v6-rev0.7z

https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/8.1.0/threads-win32/dwarf/i686-8.1.0-release-win32-dwarf-rt_v6-rev0.7z

amd64 and 386 versions. I am not sure, if I used correct versions, but they seems to work with my change. I am happy to use any alternative location / version - as long as it is recent. @mattn maybe you have suggestions which mingw version to use.

I also am not sure what to do about old gcc 5.1 installed on our builders. Perhaps we should have both old and new gcc versions installed onto our disk images. And then have different builders use different gcc versions. For better coverage. I noticed that golang.org/x/build/cmd/buildlet.windowsPath adds gcc bin directory to the path, perhaps we could adjust that code.

@bradfitz can you help?

Thank you.

Alex

[bradfitz]

I'm too busy to work on this unfortunately.

/cc @toothrot @dmitshur

[alexbrainman]

I'm too busy to work on this unfortunately.

No worries. I will try to have a go at this myself. When I have time.

Alex

[zx2c4]

I'd recommend using the ones from https://musl.cc/ . They're up to date and work very well:

http://musl.cc/x86_64-w64-mingw32-native.zip
http://musl.cc/i686-w64-mingw32-native.zip

I mirror the snapshot that I use for WireGuard here, if you'd prefer to grab a build that's been "battle-tested":

https://download.wireguard.com/windows-toolchain/distfiles/x86_64-w64-mingw32-native-20190903.zip
https://download.wireguard.com/windows-toolchain/distfiles/i686-w64-mingw32-native-20190903.zip

[alexbrainman]

I'd recommend using the ones from https://musl.cc/ . They're up to date and work very well:

http://musl.cc/x86_64-w64-mingw32-native.zip
http://musl.cc/i686-w64-mingw32-native.zip

I tried running all.bat on the current tip against

http://musl.cc/x86_64-w64-mingw32-native.zip

and it all passes. While, if I do the same against

https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/threads-win32/seh/x86_64-8.1.0-release-win32-seh-rt_v6-rev0.7z

it fails in debug/pe with error similar to #23649 (comment)

Why is that?

And, while I, probably, prefer your version of gcc, our builders should test against what most users use.

Alex

[zx2c4]

I'll CC @zv-io to talk about why the musl.cc compilers are better.

Usually those compilers are up to date and use recent MingW that fixes bugs.

And, while I, probably, prefer your version of gcc, our builders should test against what most users use.

We should direct users to using actually maintained compilers. Windows toolchains are a moving target, and those old ones have many problems.

[alexbrainman]

We should direct users to using actually maintained compilers.

I am all for making better tools available to Go users.

But we still need to fix bugs that Go users experience while using whatever tools they use. How do you propose we handle bugs like #23649 if we don't use same compilers as #23649 users?

Alex

[zx2c4]

But we still need to fix bugs that Go users experience while using whatever tools they use. How do you propose we handle bugs like #23649 if we don't use same compilers as #23649 users?

I have no interest adding hacks to Golang to work around known bugs in ancient compilers that aren't supported by Microsoft in the first place. It would be one thing if we supported MSVC and there was a bug there we have to deal with. But here we're talking about some random compilers people download from Sourceforge. I won't waste my time fighting with that junk. Rather, most of these things have been fixed upstream already, and @zv-io is nice enough to make builds available at musl.cc.

[zv-io]

Hi, I maintain these.

I'd recommend using the ones from https://musl.cc/ . They're up to date and work very well:
http://musl.cc/x86_64-w64-mingw32-native.zip
http://musl.cc/i686-w64-mingw32-native.zip

I tried running all.bat on the current tip against

http://musl.cc/x86_64-w64-mingw32-native.zip

and it all passes. While, if I do the same against

https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/threads-win32/seh/x86_64-8.1.0-release-win32-seh-rt_v6-rev0.7z

it fails in debug/pe with error similar to #23649 (comment)

Why is that?

I can't explain why the other toolchain causes the error, and the build log referenced by that comment is nonexistent. I'm having a hard time finding information about how those toolchains are built. I use the latest MinGW-W64 when I prepare each release, and provide recipes and build scripts for all releases.

And, while I, probably, prefer your version of gcc, our builders should test against what most users use.

There's #23649 (comment) which maybe someone else can advise on, but I'm going to agree with @zx2c4 that it's entirely a waste of time to figure out how to hack around bugs in broken toolchains. Further, not encouraging their use might reduce your maintenance burden, not add to it.

I'm guessing that "most users" won't complain about using up-to-date, reproducible toolchains that don't require JavaScript or advertisements to download. They even work on Windows XP.

[alexbrainman]

But here we're talking about some random compilers people download from Sourceforge.

This is what all Go users use. Including our builders (see https://github.com/golang/go/wiki/WindowsBuild ).

If we are to change this, then we should all agree what Mingw we use. Perhaps we should even provide the url in https://golang.org/doc/install - if there are many Mingw versions and they work differently, then we should be explicit there. And we should update our builders to run the same software.

I am fine using http://musl.cc if everyone agrees. @rsc and @ianlancetaylor ?

And we won't be fixing issues like #23649, because they are not reproducible with http://musl.cc . @mattn can you, please, verify? I used http://musl.cc/x86_64-w64-mingw32-native.zip

I can't explain why the other toolchain causes the error,

@zv-io thank you for taking time to comment here. The fix for #23649 is here

https://go-review.googlesource.com/c/go/+/197977/

And the gist of the fix is, it works around commit b295099 from git://git.code.sf.net/p/mingw-w64/mingw-w64 Does your Mingw build includes this commit? The commit (as I understand it) adds __imp___acrt_iob_func symbol. That breaks Go internal linker, because Go assumes that all symbols starting with __imp_ live in external DLLs. The bug does not happens with http://musl.cc - I did not investigate it enough to understand why.

Alex

[gopherbot]

Change https://golang.org/cl/203603 mentions this issue: Revert "cmd/link: use libmsvcrt.a during internal link"

[alexbrainman]

@rsc and @ianlancetaylor

The MINGW gcc our builders use is very very old. We need to decide on the replacement - see my comment #35006 (comment) and whole thread in general.

What do you think?

Thank you.

Alex

[mattn]

@alexbrainman

@ mattn can you, please, verify? I used http://musl.cc/x86_64-w64-mingw32-native.zip

I checked this compiler.

• mattn/go-sqlite3 (sqlite3 driver)
• mattn/go-tflite (tensorflow lite binding)
• mattn/go-oci8 (oracle driver)

All works fine. And -race too.

[ALTree]

I recently had to set up a Windows machine to compile Go (with cgo) from source and it was a truly awful experience.

I somewhat managed to hit every single issue and incompatibility between windows, cgo and several versions of gcc/cygwin/mingw/mingw-w64/msys/TDM-GCC that was ever reported on the issue tracker. At some point I had probably half a dozen GNU toolchains installed, all giving me a different combination of errors, crashes, and gcc warnings when running make.bat.

At the end I was able to find some installer that downloaded and unpacked a mingw-w64 toolchain that was miraculously able to make a cgo-enabled make.bat run finish with no issues. I don't even know where that toolchain came from and what version of gcc shipped.

So yes, let's settle on some maintained, up-to-date GNU toolchain distribution for Windows, bless that one, and document it in

• https://golang.org/doc/install
• https://golang.org/doc/install/source#ccompiler
• https://github.com/golang/go/wiki/InstallFromSource

So that people using cgo or building Go from source on Windows know what GNU toolchain they are supposed to put on their machine.

Anyway after finding this thread I nuked all the GNU toolchains I had and tried the musl.cc linked above and it worked just fine. I like that

• It seems to be maintained and ships a recent GCC version
• It can be used to run a cgo-enabled make.bat with no warnings whatsoever
• It's portable (download, unzip, add to path); no installers