Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link,runtime: k8s unit tests on ppc64le panics with index out of range #48837

Closed
Karthik-K-N opened this issue Oct 7, 2021 · 8 comments
Closed

Comments

@Karthik-K-N
Copy link

@Karthik-K-N Karthik-K-N commented Oct 7, 2021

What version of Go are you using (go version)?

$ go version
go version devel go1.18-6f74ed06c5 Thu Oct 7 03:41:40 2021 +0000 linux/ppc64le

Does this issue reproduce with the latest release?

No

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/go/bin/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/go/bin:/root/bin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="devel go1.18-6f74ed06c5 Thu Oct 7 03:41:40 2021 +0000"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/kubernetes/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2441321377=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Install upstream dev version of golang:
    Have a stable version of go already installed.
    git clone https://github.com/golang/go.git
    cd go/src
    ./make.bash
    cd ../..
    rm -rf /usr/local/go
    cp -r ./go /usr/local
    export PATH=/usr/local/go/bin:$PATH
    export PATH=$GOPATH/bin:$PATH
Clone k8s/k8s repo and run below steps: (git clone https://github.com/kubernetes/kubernetes)
    export KUBE_TIMEOUT='--timeout=600s' export KUBE_COVER="n" export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export LOG_LEVEL=4
    cd kubernetes
    pushd ./hack/tools
    GO111MODULE=on go install gotest.tools/gotestsum
    popd
    make test KUBE_RACE=-race WHAT=./cmd/genkubedocs

What did you expect to see?

Test passing successfully

What did you see instead?

+++ [1007 06:05:26] Running tests without code coverage and with -race
fatal error: index out of range

runtime stack:
runtime.throw({0x1748a03d, 0x1a85d840})
	/usr/local/go/src/runtime/panic.go:965 +0x58
runtime.panicCheck1(0x19c8bdc0, {0x1748a03d, 0x12})
	/usr/local/go/src/runtime/panic.go:35 +0xb0
runtime.goPanicIndex(0x26fd9, 0x26fd9)
	/usr/local/go/src/runtime/panic.go:88 +0x4c
runtime.findfunc(0x163bb150)
	/usr/local/go/src/runtime/symtab.go:831 +0x1d8
runtime.gentraceback(0x7ffff7490198, 0xffffffffffffffff, 0x17617ff6, 0xc0000021a0, 0x4, 0xc000f277a0, 0x20, 0x0, 0x17617ff6, 0x0)
	/usr/local/go/src/runtime/traceback.go:221 +0x564
runtime.callers.func1()
	/usr/local/go/src/runtime/traceback.go:857 +0x64
runtime.systemstack()
	/usr/local/go/src/runtime/asm_ppc64x.s:256 +0x68

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_ppc64x.s:213 +0x10 fp=0xc000f276c0 sp=0xc000f276a0 pc=0x11deb360
runtime.callers(0xc00019a000, {0xc000f277a0, 0x11db2124, 0x11d8c318})
	/usr/local/go/src/runtime/traceback.go:856 +0xb8 fp=0xc000f27738 sp=0xc000f276c0 pc=0x11de1fe8
runtime.mProf_Malloc(0xc000d15000, 0x1000)
	/usr/local/go/src/runtime/mprof.go:343 +0x74 fp=0xc000f278b8 sp=0xc000f27738 pc=0x11dad804
runtime.profilealloc(0x163e13f0, 0x0, 0x0)
	/usr/local/go/src/runtime/malloc.go:1255 +0xac fp=0xc000f27900 sp=0xc000f278b8 pc=0x11d8c91c
runtime.mallocgc(0x1000, 0x167b7720, 0x1)
	/usr/local/go/src/runtime/malloc.go:1134 +0x7ac fp=0xc000f27990 sp=0xc000f27900 pc=0x11d8c40c
runtime.makeslice(0x1a916a60, 0x11defc90, 0x11f13d34)
	/usr/local/go/src/runtime/slice.go:100 +0xb4 fp=0xc000f279c8 sp=0xc000f27990 pc=0x11dd1f64
bufio.NewWriterSize(...)
	/usr/local/go/src/bufio/bufio.go:590
bufio.NewWriter(...)
	/usr/local/go/src/bufio/bufio.go:597
testing/internal/testdeps.TestDeps.StartTestLog({}, {0x183dc6e0, 0xc00019e6e8})
	/usr/local/go/src/testing/internal/testdeps/deps.go:110 +0xbc fp=0xc000f27a38 sp=0xc000f279c8 pc=0x11ff677c
testing/internal/testdeps.(*TestDeps).StartTestLog(0x1b371cf0, {0x183dc6e0, 0xc00019e6e8})
	<autogenerated>:1 +0x88 fp=0xc000f27a68 sp=0xc000f27a38 pc=0x11ff7b68
testing.(*M).before(0xc00080f4a0)
	/usr/local/go/src/testing/testing.go:1850 +0x82c fp=0xc000f27b68 sp=0xc000f27a68 pc=0x11f13e5c
fatal error: index out of range
panic during panic

runtime stack:
runtime.throw({0x1748a03d, 0x1a85d840})
	/usr/local/go/src/runtime/panic.go:965 +0x58
runtime.panicCheck1(0x183c44a0, {0x1748a03d, 0x12})
	/usr/local/go/src/runtime/panic.go:35 +0xb0
runtime.goPanicIndex(0x26fd9, 0x26fd9)
	/usr/local/go/src/runtime/panic.go:88 +0x4c
runtime.findfunc(0x163bb150)
	/usr/local/go/src/runtime/symtab.go:831 +0x1d8
runtime.gentraceback(0x8, 0x0, 0x11de1758, 0xc0000021a0, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/traceback.go:221 +0x564
runtime.traceback1(0xc0000021a0, 0x0, 0x11db758c, 0xc0000021a0, 0x10)
	/usr/local/go/src/runtime/traceback.go:789 +0x1a4
runtime.traceback(...)
	/usr/local/go/src/runtime/traceback.go:743
runtime.tracebackothers(0x1a924aa0)
	/usr/local/go/src/runtime/traceback.go:981 +0xb4
runtime.dopanic_m(0x1a924aa0, 0x11db6e14, 0x11db6e18)
	/usr/local/go/src/runtime/panic.go:1165 +0x2b0
runtime.fatalthrow.func1()
	/usr/local/go/src/runtime/panic.go:1020 +0x4c
runtime.systemstack()
	/usr/local/go/src/runtime/asm_ppc64x.s:282 +0xa8
FAIL	k8s.io/kubernetes/cmd/genkubedocs	0.407s
FAIL
make: *** [Makefile:184: test] Error 1

Job History can be found here: https://prow.ppc64le-cloud.org/job-history/s3/prow-logs/logs/postsubmit-master-golang-kubernetes-unit-test-ppc64le

I suspect a recent commit 8238f82 is causing the issue.

Verified that same test passed successfully on x86 machine

@mkumatag
Copy link

@mkumatag mkumatag commented Oct 7, 2021

Loading

@josharian
Copy link
Contributor

@josharian josharian commented Oct 7, 2021

I won't be at a keyboard until later today. If you have time, can you confirm exactly which commit caused the regression? There have been a couple recently that touched that code. Thank you!

Loading

@laboger
Copy link
Contributor

@laboger laboger commented Oct 7, 2021

@archanaravindar verified for me earlier that it fails on e31c9ab but not on the commit prior.

Note that k8 is a test that generates very large binaries, requiring trampolines and maybe splitting text sections. It seems to me we've had issues with the functab with large programs like this in the past, but I'd like to see if I can find anything on it.

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Oct 7, 2021

Yes, it is related to multiple text sections. That CL is supposed to take that into consideration. But it is possible something is not really correct.

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Oct 7, 2021

I think I kind of understand it. It is an off-by-1 error of some sort. Will try to fix later today.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 7, 2021

Change https://golang.org/cl/354636 mentions this issue: cmd/link: mark holes in functab with end PC-1

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 7, 2021

Change https://golang.org/cl/354635 mentions this issue: runtime: handle end PC in textAddr

Loading

gopherbot pushed a commit that referenced this issue Oct 7, 2021
As the func table contains the end marker of the text section, we
sometimes need to get that address from an offset. Currently
textAddr doesn't handle that address, as it is not within any
text section. Instead of letting the callers not call textAddr
with the end offset, just handle it more elegantly in textAddr.

For #48837.

Change-Id: I6e97e455f6cb66e9680a7aac6152ba6f4cda2e12
Reviewed-on: https://go-review.googlesource.com/c/go/+/354635
Trust: Cherry Mui <cherryyz@google.com>
Trust: Josh Bleecher Snyder <josharian@gmail.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
gopherbot pushed a commit that referenced this issue Oct 7, 2021
When we have multiple text sections, we need to mark holes between
the sections in the functab. A hole is marked with an entry with
the end PC of the previous section. As we now use offsets instead
of (relocated) PCs, the end offset of a section may be the same of
the start of the next one. Distinguish it by using the end address
-1.

For #48837.

Change-Id: I121aac53b32a869378632cf151cb1b6f98ad3089
Reviewed-on: https://go-review.googlesource.com/c/go/+/354636
Trust: Cherry Mui <cherryyz@google.com>
Trust: Josh Bleecher Snyder <josharian@gmail.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
@mknyszek mknyszek changed the title k8s unit tests on ppc64le panics with index out of range cmd/link,runtime: k8s unit tests on ppc64le panics with index out of range Oct 7, 2021
@mknyszek mknyszek added this to the Go1.18 milestone Oct 7, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Oct 7, 2021

With the CL above, it passes now.

Loading

@cherrymui cherrymui closed this Oct 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants