Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misc/cgo/testsigfwd: Test fails starting with 1.19 on x86 Alpine Linux #54422

Open
nmeum opened this issue Aug 12, 2022 · 10 comments
Open

misc/cgo/testsigfwd: Test fails starting with 1.19 on x86 Alpine Linux #54422

nmeum opened this issue Aug 12, 2022 · 10 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@nmeum
Copy link

nmeum commented Aug 12, 2022

What version of Go are you using (go version)?

$ go version
go version go1.19 linux/386

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="386"
GOBIN="/home/buildozer/src/aports/community/go/src/go/bin"
GOCACHE="/home/buildozer/.cache/go-build"
GOENV="/home/buildozer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS="-buildmode=pie -modcacherw"
GOHOSTARCH="386"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/buildozer/.local/share/go/modcache"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/buildozer/src/aports/community/go/src"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/home/buildozer/src/aports/community/go/src/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/home/buildozer/src/aports/community/go/src/go/pkg/tool/linux_386"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
GO386="sse2"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m32 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1188913618=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Compiled Go from source using ./make.bash on x86 Alpine Linux Edge with 365ca69 backported. Afterwards, invoked the test suite using ./run.bash -run='!(^cgo_test$)' (cgo tests are known to fail on Alpine due to #39857).

What did you expect to see?

Successful test run.

What did you see instead?

A test failure of ../misc/cgo/testsigfwd:

##### ../misc/cgo/testsigfwd
# misc/cgo/testsigfwd
cgo: cannot load DWARF output from $WORK/b001//_cgo_.o: zlib: invalid header
2022/08/12 18:10:59 Failed: exit status 2

Full build log: alpine-linux-x86-edge-go-1.19.txt

I am aware that Alpine/musl is not fully supported by the test suite (see #19938 etc.), however, I am still reporting this as the test case passed with Go 1.18 (and also passes on all other architectures supported by Alpine). Furthermore, this seems to be related to #52919 and #54313 as the test case passes successfully with the following change (as discussed in the referenced issues, Alpine uses -fstack-protector by default):

diff -upr go.orig/misc/cgo/testsigfwd/main.go go/misc/cgo/testsigfwd/main.go
--- go.orig/misc/cgo/testsigfwd/main.go	2022-08-12 20:28:17.222319036 +0200
+++ go/misc/cgo/testsigfwd/main.go	2022-08-12 20:28:45.752384953 +0200
@@ -7,7 +7,7 @@ package main
 import "fmt"
 
 /*
-#cgo CFLAGS: -pthread
+#cgo CFLAGS: -pthread -fno-stack-protector
 #cgo LDFLAGS: -pthread
 
 #include <signal.h>

If -fstack-protector is generally unsupported on all Go architectures, then I would suggest having cgo add that to the compiler flags by default unconditionally. If it is just about this test case it would also be sufficient to commit the diff from above (with this diff the run.bash invocation from above succeeds). However, I don't fully understand why it worked prior to Go 1.19 without -fno-stack-protector.

@thanm thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 15, 2022
@thanm thanm added this to the Go1.20 milestone Aug 15, 2022
@thanm
Copy link
Contributor

thanm commented Aug 15, 2022

@golang/runtime

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Aug 15, 2022

This may be fixed on tip by https://go.dev/cl/421935. If so, we could consider backporting that to 1.19.

@nmeum
Copy link
Author

nmeum commented Aug 15, 2022

This may be fixed on tip by https://go.dev/cl/421935. If so, we could consider backporting that to 1.19.

That is the same as 365ca69, no?

Because if so I already have that backported (see What did you do? in my original issue description).

algitbot pushed a commit to alpinelinux/aports that referenced this issue Aug 15, 2022
@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Aug 17, 2022

Sorry for the confusion.

I agree that we shouldn't start adding -fno-stack-protector everywhere, and we should figure out what changed to make it required.

Looking further at the issue I don't understand why not adding it would lead to the "invalid header" error that you mention.

@nmeum
Copy link
Author

nmeum commented Aug 19, 2022

Looking further at the issue I don't understand why not adding it would lead to the "invalid header" error that you mention.

git-bisect(1) tells me that testsigfwd started failing on Alpine Linux x86 with commit
d34287a prior to that commit the test passes fine without -fno-stack-protector.

I don't understand though how -fstack-protector relates to ELF compression.

Any ideas?

@nmeum
Copy link
Author

nmeum commented Aug 23, 2022

git-bisect(1) tells me that testsigfwd started failing on Alpine Linux x86 with commit d34287a prior to that commit the test passes fine without -fno-stack-protector.

I don't understand though how -fstack-protector relates to ELF compression.

@mengzhuo since you authored the aforementioned commit, do you have an idea how this might be related?

I was also able to extract the $WORK/b001//_cgo_.o object file, which the go build invocation from above complains about. Furthermore, I also extracted this object file in a build with -fno-stack-protector to make it easier to compare the two:

Maybe this helps with debugging the error regarding compressed dwarf information?

I am personally not sure what to look for in these files though.

@mengzhuo
Copy link
Contributor

mengzhuo commented Aug 24, 2022

I had no idea how they are related, sorry.
By the way, I can't reproduce this on the latest Alpine release.

cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.16.2
PRETTY_NAME="Alpine Linux v3.16"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
Using built-in specs.

gcc -v
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-alpine-linux-musl/11.2.1/lto-wrapper
Target: x86_64-alpine-linux-musl
Configured with: /home/buildozer/aports/main/gcc/src/gcc-11.2.1_git20220219/configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --build=x86_64-alpine-linux-musl --host=x86_64-alpine-linux-musl --target=x86_64-alpine-linux-musl --with-pkgversion='Alpine 11.2.1_git20220219' --enable-checking=release --disable-fixed-point --disable-libstdcxx-pch --disable-multilib --disable-nls --disable-werror --disable-symvers --enable-__cxa_atexit --enable-default-pie --enable-default-ssp --enable-cloog-backend --enable-languages=c,c++,d,objc,go,fortran,ada,jit --disable-libssp --disable-libmpx --disable-libmudflap --disable-libsanitizer --enable-shared --enable-threads --enable-tls --enable-host-shared --with-system-zlib --with-linker-hash-style=gnu
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 11.2.1 20220219 (Alpine 11.2.1_git20220219)

alpine:~/godev/src# ./run.bash -run='testsigfwd'

##### Building packages and commands.

##### Test execution environment.
# GOARCH: amd64
# CPU: Intel(R) Xeon(R) W-2150B CPU @ 3.00GHz
# GOOS: linux
# OS Version: Linux 5.15.62-0-lts #1-Alpine SMP Tue, 23 Aug 2022 13:50:24 +0000 x86_64

##### ../misc/cgo/testsigfwd
ok      testsigfwd

ALL TESTS PASSED (some were excluded)
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/root/godev"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/root/godev/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/godev/src/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1767409198=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.19 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.19
uname -sr: Linux 5.15.62-0-lts

@nmeum
Copy link
Author

nmeum commented Aug 29, 2022

By the way, I can't reproduce this on the latest Alpine release.

Thanks for pointing this out. We ran into this on Alpine Linux Edge (the development version) since cgo is involved here this might then be related to a specific binutils/gcc/… version. On Edge we use binutils 2.39 (2.38 in the latest Alpine release) and gcc 12.1.1 (11.2.1 on the latest Alpine release).

@nmeum
Copy link
Author

nmeum commented Sep 23, 2022

I had no idea how they are related, sorry. By the way, I can't reproduce this on the latest Alpine release.

@mengzhuo while debugging this further I just noticed that you tried to reproduce this issue on amd64 (at least GOARCH is set to amd64 in your test environment output). However, as described above this issue can only be reproduced on x86 (i.e. GOARCH=i386) even on Alpine Linux Edge.

The code that seems to be emitting the error message is:

go/src/debug/elf/file.go

Lines 1183 to 1189 in d34287a

r, err := zlib.NewReader(bytes.NewBuffer(b[s.compressionOffset:]))
if err != nil {
return nil, err
}
if _, err := io.ReadFull(r, dbuf); err != nil {
return nil, err
}

I added the following printf debug output:

diff --git a/src/debug/elf/file.go b/src/debug/elf/file.go
index aff2b00aae..6af82828d0 100644
--- a/src/debug/elf/file.go
+++ b/src/debug/elf/file.go
@@ -1253,6 +1253,10 @@ func (f *File) DWARF() (*dwarf.Data, error) {
                        }
                }
                if dlen > 0 {
+                       fmt.Printf("Section: %s\n", s.Name)
+                       fmt.Printf("b = %x\n", b)
+                       fmt.Printf("compressionOffset = %v\n", s.compressionOffset)
+                       fmt.Printf("decoding: %x\n", b[s.compressionOffset:])
                        r, err := zlib.NewReader(bytes.NewBuffer(b[s.compressionOffset:]))
                        if err != nil {
                                return nil, err

With this patch the output is:

Section: .debug_ranges
b = 010000000600000028000000310000003400000036000000000000000000000000000000b500000000000000820000000000000000000000
compressionOffset = 12
decoding: 310000003400000036000000000000000000000000000000b500000000000000820000000000000000000000
cgo: cannot load DWARF output from $WORK/b001//_cgo_.o: zlib: invalid header

And 0x31 is indeed not a valid zlib header.

In fact, it seems that the .debug_ranges section is not compressed at all?

$ objdump --dwarf _cgo_with_stack_protector.o
Contents of the .debug_ranges section:


    Offset   Begin    End
    00000000 00000001 00000006
    00000000 00000028 00000031
    00000000 00000034 00000036
    00000000 <End of list>
    00000020 00000000 000000b5
    00000020 00000000 00000082
    00000020 <End of list


I am not sure though if .debug_ranges section should be compressed or if Go should not attempt to decompress is as it is not compressed. Any hints on that?

The file is available here: https://dev.alpinelinux.org/~nmeum/go-issue-%2354422/_cgo_with_stack_protector.o

@thanm
Copy link
Contributor

thanm commented Sep 23, 2022

I note that when the Go linker does DWARF compression, if a given section doesn't benefit from compression it will leave it as is (e.g. see https://go.googlesource.com/go/+/2551324cd01b295915c10c6d3d06625676401610/src/cmd/link/internal/ld/dwarf.go#2225). Could this be what's happening here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants