runtime: crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM64 windows 11.

[oddy]

What version of Go are you using (go version)?

$ go version
go version go1.20 windows/amd64

Does this issue reproduce with the latest release?

Yes. Confirmed behavior with go1.21rc2 also.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\bb\AppData\Local\go-build
set GOENV=C:\Users\bb\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\bb\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\bb\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go120
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go120\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\work\go\minimal_cfguard_test\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\bb\AppData\Local\Temp\go-build1785003883=/tmp/go-build -gno-record-gcc-switches
gdb --version: GNU gdb (GDB) 8.1

What did you do?

• Behaviour exhibits reliably when running as amd64 DLL in either C or Python host EXE on windows ARM machines.
• Crash/Hang is intermittent but can be reliably triggered in between 5-20 repeated runs of the exe.
• Trigger condition tracked down to the host exe having windows anti-exploit measure "Control Flow Guard" enabled.

Pre-reqs:

• Windows 11 ARM machine (vm or baremetal)
• Go 1.20 or 1.21rc2 installed
• Visual C Build Tools 2019 (or 2020) installed (for the caller exe)
• MingGW 8.10 or TDM-GCC latest (for Go DLL compiling)

Steps to reproduce as follows:

mindll.go

package main

import "C"
import "fmt"

//export Fred
func Fred() {
	fmt.Println("go: MINDLL: i am fred")
}

func main() { }

compile it

go mod init
go mod init mindll
go mod tidy
go build -buildmode=c-shared -o mindll.dll .

c_call.c

#include <windows.h>
#include <stdio.h>
typedef void (*fred_func)(void);
int main() {
	HINSTANCE hinstLib = LoadLibrary(TEXT("mindll.dll"));
	fred_func Fred;
	Fred = (fred_func)GetProcAddress(hinstLib, "Fred");
	printf("Calling Fred\n");  fflush(stdout);
	Fred();
	printf("Back from Fred\n");
    return 0;
}

Compile it (no CFGuard)

"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
cl c_call.c -o ccall_nocfg.exe -Wall

ccall_nocfg.bat

ccall_nocfg.exe
ccall_nocfg.bat

Run ccall_nocfg.bat for a while, runs fine, no issues.

Compile caller with CFGuard on

cl c_call.c -o ccall_withcfg.exe -Wall /guard:cf

ccall_withcfg.bat

ccall_withcfg.exe
ccall_withcfg.bat

Run ccall_withcfg.bat, process crashes/hangs quite quickly, anywhere from 5th to 20th run. Then keeps crashing throughout, every 5-20 runs or so.

What did you expect to see?

The C process and Go DLL running reliably 100%

What did you see instead?

The process hanging/crashing after LoadLibrary and/or at first DLL function call (Fred) every 5-20 runs.

Workarounds (& why they are workarounds)

• "Compile everything for ARM" - Don't have control of the python exe 'mothership'. Viable arm python has only recently appeared with 3.11 but we must support back to py 3.7 for windows 7. Also many python libraries not fully available native for arm windows yet.
• "Compile caller without Control Flow Guard" - as above + unhappy security auditors
• "Have the end users disable Control Flow Guard for this process (or system-wide) in windows control panel for ARM windows machines" - very unhappy security auditors. Bad reputational for product.

Notes

• This behavior does not happen when a minimal C-DLL is substituted for the Go DLL above, regardless of whether or not the test C DLL has control flow guard enabled or disabled.

• There is only one Go runtime in the process as the 'mothership' is either a python or C exe.

• I am not freeing or attempting to unload the Go DLL at all.

• The process briefly hangs then suddenly exits and no errors, stacktraces or anything are printed. Also nothing in the Windows event log, which seems to imply that Go is eating the exception and calling ExitProcess?

• I see support for CFG itself IN go was deemed not-useful here cmd/compile: add control flow integrity options #35940 however interactions with CFG when GO is a DLL do possibly still need to be addressed.

• (Speculation) perhaps this issue is similar to that of runtime: TLS slot index over 64 and crash #59213 however this issue is still present in Go 1.21rc2

[qmuntal]

Thanks for reporting this issue.

I'm afraid I can't reproduce it (I made sure CFG is enabled). Could it be that you are using an older gcc toolchain? I use mingw-w64 12.2.0:

gcc.exe (MinGW-W64 x86_64-ucrt-posix-seh, built by Brecht Sanders) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[oddy]

Hi Quim, thanks, i appreciate your time and effort :)

Just double-checking you're trying to repro on an ARM windows machine?

I will try mingw-w64 12.2.0 also and report back.

[oddy]

Ok i tried with mingGW gcc 12.2.0 also, both the UCRT and MSVCRT versions, behaviour remains unchanged.

[qmuntal]

Just double-checking you're trying to repro on an ARM windows machine?

Huh, I tried to repro on an ARM64 windows machine, not ARM. Can't help here, as I don't have one.

The windows/arm port is not in a good state given that it hasn't had a builder for years. See #57960.

[oddy]

Sorry, arm64 is what i meant. Apologies for confusion.

[qmuntal]

Sorry, arm64 is what i meant. Apologies for confusion.

Oh, ok. Still can't repro it. Could it be that you are using an older C toolchain? I used Visual C Build Tools 2022:

cl.exe
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30151 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.