Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: buildmode=pie causes silent failures when using PIE-incompatible C libraries, windows exit status 3221225477 / 0xC0000005 #42233

Open
jasonpfox opened this issue Oct 27, 2020 · 10 comments

Comments

@jasonpfox
Copy link

@jasonpfox jasonpfox commented Oct 27, 2020

What version of Go are you using (go version)?

go version go1.15.3 windows/amd64

Does this issue reproduce with the latest release?

Yes,
Tested and failed with 1.15.2 and 1.15.3.
No problems with prior versions including 1.13.15 and 1.14.9.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Fox\AppData\Local\go-build
set GOENV=C:\Users\Fox\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=c:\GoPath\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=c:\GoPath
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Fox\AppData\Local\Temp\go-build476616686=/tmp/go-build -gno-record-gcc-switches
GOROOT/bin/go version: go version go1.15.3 windows/amd64
GOROOT/bin/go tool compile -V: compile version go1.15.3
gdb --version: GNU gdb (GDB) 7.9.1

What did you do?

Attempted to migrate working program to 1.15. Silent failure with no indication as to why.
Traced down problematic package to a go wrapper around the C libde265 video decoder.
Program "appears" to build and run, but never progresses to user code when executed and fails with an exit status.
"go run" returns the error exit status.

I do not know now to dig into the build / link process to troubleshoot further.....

What did you expect to see?

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go version
go version go1.14.9 windows/amd64

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go run .
Starting test program, initializing decoder
Decoder created, exiting program

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
0

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go install .

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>test.exe
Starting test program, initializing decoder
Decoder created, exiting program

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
0

What did you see instead?

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go version
go version go1.15.3 windows/amd64

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go install . note silent failure with no error or messages

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>test.exe silent failure to do anything, never gets to any init functions much less main()

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
-1073741819

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go run .
exit status 3221225477 3221225477== 0xC0000005 indicates windows access violation

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
1

libde.zip

@davecheney
Copy link
Contributor

@davecheney davecheney commented Oct 28, 2020

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go install . note silent failure with no error or messages

Was a %errorlevel% reported?

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>test.exe silent failure to do anything, never gets to any init functions much less main()

If the previous command did not succeed could it be possible that test.exe was not what you expected?

Lastly, I’m sorry to ask, but it’s windows, so I must, have you ruled out interference by AV software?

@davecheney
Copy link
Contributor

@davecheney davecheney commented Oct 28, 2020

You can also try ‘go install -x’ to encourage the go tool to print the commands it intends to run.

@jasonpfox
Copy link
Author

@jasonpfox jasonpfox commented Oct 28, 2020

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go install . note silent failure with no error or messages

Was a %errorlevel% reported?

No, install command returned errorlevel 0 when queried. After install, running the executable generated returns a different errorlevel when queried (-1073741819, which I have not found any info on). I get the errorlevel return when using 'go run .' I am hoping the difference in errorlevel between go install, running executable, and go run may be a clue as to where or how it is breaking???

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go version
go version go1.15.3 windows/amd64

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go install .

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
0

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>ls -l c:\GoPath\bin\test.exe I have unix tools installed in Win
-rwxr-xr-x 1 Fox None 11687009 Oct 28 10:42 'c:\GoPath\bin\test.exe'

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>c:\GoPath\bin\test.exe

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
-1073741819

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go run .
exit status 3221225477

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>test.exe silent failure to do anything, never gets to any init functions much less main()

If the previous command did not succeed could it be possible that test.exe was not what you expected?
Good question, but no. I just reverified again with which test.exe, but more directly my test was run back to back between go 1.14.9 and 1.15.3 with the only intervening steps being go resinstall...

Lastly, I’m sorry to ask, but it’s windows, so I must, have you ruled out interference by AV software?

I have run exactly the same commands back and forth with the only change being reinstalling a different go version between the tests with AV and windows firewall all turned off, so I am pretty sure this is not it (but as you say, it is windows... :)

@jasonpfox
Copy link
Author

@jasonpfox jasonpfox commented Oct 28, 2020

You can also try ‘go install -x’ to encourage the go tool to print the commands it intends to run.

Install thinks is succeeds. My hypothesis is that one of the intermediate steps of the installer is / could be returning an errorlevel that is not checked that is then overwritten by the next step which succeeds with errorlevel 0.

After install I have an 11 kB executable that does nothing (fails before user code with an errorlevel).

Run from 'go install -x":

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>go install -x .
WORK=C:\Users\Fox\AppData\Local\Temp\go-build024877162
mkdir -p $WORK\b001
cat >$WORK\b001\importcfg.link << 'EOF' # internal
packagefile github.com/axionat/eagle/piglet/libde265/test=C:\Users\Fox\AppData\Local\go-build\29\291ee374c5b3eb9336b4c2d0a0e0de034be7fa3a3a0ab5082374eecd9b3058e0-d
packagefile fmt=c:\go\pkg\windows_amd64\fmt.a
packagefile github.com/axionat/eagle/piglet/libde265=C:\Users\Fox\AppData\Local\go-build\16\16a726d7627eab3263438f17f67c5a6c8bc41e0678a4b871189969c2d293f25e-d
packagefile runtime=c:\go\pkg\windows_amd64\runtime.a
packagefile errors=c:\go\pkg\windows_amd64\errors.a
packagefile internal/fmtsort=c:\go\pkg\windows_amd64\internal\fmtsort.a
packagefile io=c:\go\pkg\windows_amd64\io.a
packagefile math=c:\go\pkg\windows_amd64\math.a
packagefile os=c:\go\pkg\windows_amd64\os.a
packagefile reflect=c:\go\pkg\windows_amd64\reflect.a
packagefile strconv=c:\go\pkg\windows_amd64\strconv.a
packagefile sync=c:\go\pkg\windows_amd64\sync.a
packagefile unicode/utf8=c:\go\pkg\windows_amd64\unicode\utf8.a
packagefile encoding/base64=c:\go\pkg\windows_amd64\encoding\base64.a
packagefile runtime/cgo=c:\go\pkg\windows_amd64\runtime\cgo.a
packagefile syscall=c:\go\pkg\windows_amd64\syscall.a
packagefile internal/bytealg=c:\go\pkg\windows_amd64\internal\bytealg.a
packagefile internal/cpu=c:\go\pkg\windows_amd64\internal\cpu.a
packagefile runtime/internal/atomic=c:\go\pkg\windows_amd64\runtime\internal\atomic.a
packagefile runtime/internal/math=c:\go\pkg\windows_amd64\runtime\internal\math.a
packagefile runtime/internal/sys=c:\go\pkg\windows_amd64\runtime\internal\sys.a
packagefile internal/reflectlite=c:\go\pkg\windows_amd64\internal\reflectlite.a
packagefile sort=c:\go\pkg\windows_amd64\sort.a
packagefile math/bits=c:\go\pkg\windows_amd64\math\bits.a
packagefile internal/oserror=c:\go\pkg\windows_amd64\internal\oserror.a
packagefile internal/poll=c:\go\pkg\windows_amd64\internal\poll.a
packagefile internal/syscall/execenv=c:\go\pkg\windows_amd64\internal\syscall\execenv.a
packagefile internal/syscall/windows=c:\go\pkg\windows_amd64\internal\syscall\windows.a
packagefile internal/testlog=c:\go\pkg\windows_amd64\internal\testlog.a
packagefile sync/atomic=c:\go\pkg\windows_amd64\sync\atomic.a
packagefile time=c:\go\pkg\windows_amd64\time.a
packagefile unicode/utf16=c:\go\pkg\windows_amd64\unicode\utf16.a
packagefile internal/unsafeheader=c:\go\pkg\windows_amd64\internal\unsafeheader.a
packagefile unicode=c:\go\pkg\windows_amd64\unicode.a
packagefile internal/race=c:\go\pkg\windows_amd64\internal\race.a
packagefile encoding/binary=c:\go\pkg\windows_amd64\encoding\binary.a
packagefile internal/syscall/windows/sysdll=c:\go\pkg\windows_amd64\internal\syscall\windows\sysdll.a
packagefile internal/syscall/windows/registry=c:\go\pkg\windows_amd64\internal\syscall\windows\registry.a
EOF
mkdir -p $WORK\b001\exe
cd .
"c:\go\pkg\tool\windows_amd64\link.exe" -o "$WORK\b001\exe\a.out.exe" -importcfg "$WORK\b001\importcfg.link" -buildmode=pie -buildid=jRSTOByoppzB5ofTwt_S/7S_F-1wXxcPeRIoqjSgV/uZzVaGEIrRYiTi4-yhSs/jRSTOByoppzB5ofTwt_S -extld=gcc "C:\Users\Fox\AppData\Local\go-build\29\291ee374c5b3eb9336b4c2d0a0e0de034be7fa3a3a0ab5082374eecd9b3058e0-d"
"c:\go\pkg\tool\windows_amd64\buildid.exe" -w "$WORK\b001\exe\a.out.exe" # internal
mkdir -p c:\GoPath\bin
cp $WORK\b001\exe\a.out.exe c:\GoPath\bin\test.exe
rm -r $WORK\b001\

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
0

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>test.exe

C:\GoPath\src\github.com\axionat\eagle\piglet\libde265\test>echo %errorlevel%
-1073741819

@jasonpfox
Copy link
Author

@jasonpfox jasonpfox commented Oct 28, 2020

@davecheney I manually ran though the install commands reported by 'go install -a -x -work' related to my packages and had no errors reported, I also checked the windows errorlevel between each step and it was always 0. I am not sure where else to look to find out where it is failing...

@networkimprov
Copy link

@networkimprov networkimprov commented Oct 28, 2020

Have you checked the related issues, both open & closed?

https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+0xC0000005+

@jasonpfox
Copy link
Author

@jasonpfox jasonpfox commented Oct 28, 2020

Have you checked the related issues, both open & closed?

https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+0xC0000005+

@networkimprov I had looked at open issues and found nothing, going back to closed issues I found #41329 which was closed without a fix and does seem to be my issue....

Forcing buildmode=exe (vice new 1.15 default of buildmode=pie) allows my program to compile and execute without further issues. (@escholtz @zx2c4 )

I strongly doubt I will be the last person who comes across this problem with the 1.15 migration, so I would offer a couple of thoughts to consider...

The easiest fix would be to go back to buildmode=exe as default windows behavior to maintain compatibility and not break anything for others. (@alexbrainman @ianlancetaylor @bradfitz @FiloSottile discussion on #35192)

If there is consensus that going back to buildmode=exe by default to maintain compatibility is unacceptable or undesirable, then some thoughts / suggestions based on my experience working through this one:

(1) Add a very conspicuous note / caveat to 1.15 release notes along the following lines as a proposed new second paragraph under Ports-Windows:
"Note that buildmode=pie will break programs using Cgo if any imported C libraries are not pie/aslr compliant, generating executables that silently crash. Either forcing buildmode=exe or acquiring or generating updated pie/aslr compliant C libraries will be necessary for those programs."

(2) Does anyone know if there is a way to detect if Cgo imported libraries are pie/aslr compliant so that we can either (a) fail the compilation issuing an error that buildmode=exe is required to support the C libraries or (b) automatically revert to buildmode=exe and issue a warning that buildmode=exe is being used for compliance with C libraries?

(3) If (2) is not simple, is there some way it would be possible to catch this OS error and exit code and generate a useful error or warning instead of a silent crash from the failing program? Something that includes "if you are using Cgo, try buildmode=exe to resolve this issue" would be really nice and helpful here.

Although I am back up and running, I would not recommend closing this issue until at least (1) is done, and hopefully one of (2) and (3) as well if possible.

Given the nature of the failure encountered and the difficulty characterizing it to even search for solutions, my recommendation would be to make a 1.15 minor release that reverts default behavior to buildmode=exe until at least one of (2) or (3) can be completed along with (1) to enable a graceful migration to default behavior of buildmode=pie without difficult to diagnose failures of existing programs.

Thanks all for your quick responses and thoughts.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 28, 2020

Sorry you have run into trouble. I hope that people more familiar with Windows will be able to answer your questions. It is extremely unfortunate that linking non-PIE code into a PIE does not produce an error at link time (it does on Unix systems).

I don't think it would be a good idea to change the default build mode in a minor release. That will be confusing in a different way.

@jasonpfox jasonpfox changed the title 1.15 unable to build/link failing with windows exit status 3221225477 / 0xC0000005 buildmode=pie causes silent failures when using PIE-incompatible C libraries, windows exit status 3221225477 / 0xC0000005 Oct 28, 2020
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Oct 31, 2020

@jasonpfox

Traced down problematic package to a go wrapper around the C libde265 video decoder.

Sorry. I did not even looked at your code. I don't have free time.

The easiest fix would be to go back to buildmode=exe as default windows behavior to maintain compatibility and not break anything for others.

Yes, that would work for people who use Cgo. But we would have to disable ASLR for every Go user. We provided buildmode=exe fallback specifically for cases like yours. I think our decision is reasonable.

(1) Add a very conspicuous note / caveat to 1.15 release notes along the following lines as a proposed new second paragraph under Ports-Windows:
"Note that buildmode=pie will break programs using Cgo if any imported C libraries are not pie/aslr compliant, generating executables that silently crash. Either forcing buildmode=exe or acquiring or generating updated pie/aslr compliant C libraries will be necessary for those programs."

I am not Cgo expert. I don't use if what you say in this paragraph is true. I will let others decide if it is appropriate to include statement likes in the docs.

(2) Does anyone know if there is a way to detect if Cgo imported libraries are pie/aslr compliant ...

I do not know, if what you propose is possible. I believe that ASLR is controlled by combination of GCC flags. So it is up to package user / author to set these flags appropriately for their Cgo build.

(3) If (2) is not simple, is there some way it would be possible to catch this OS error and exit code and generate a useful error ...

Again. I am not expert enough to answer your question.

Alex

@networkimprov
Copy link

@networkimprov networkimprov commented Nov 3, 2020

@gopherbot remove WaitingForInfo

@toothrot toothrot changed the title buildmode=pie causes silent failures when using PIE-incompatible C libraries, windows exit status 3221225477 / 0xC0000005 cmd/go: buildmode=pie causes silent failures when using PIE-incompatible C libraries, windows exit status 3221225477 / 0xC0000005 Nov 3, 2020
@toothrot toothrot added this to the Backlog milestone Nov 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.