runtime: name offset base pointer out of range

[stevenh]

What version of Go are you using (go version)?

$ go version go1.14 linux/amd64

Does this issue reproduce with the latest release?

Came from the latest release, reproducable, unknown

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="XXX"
GOENV="XXX"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="XXX"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build970845303=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran our standard edge node service.

What did you expect to see?

The service to run and process requests

What did you see instead?

A strange panic from the runtime.

runtime: nameOff 0xffffffff8b480000 base 0x43d78e not in ranges:
        types 0x941000 etypes 0xc3d540
fatal error: runtime: name offset base pointer out of range

runtime stack:
reflect.resolveNameOff(0x43d78e, 0x8b480000, 0x16)
        /downloads/go/src/runtime/runtime1.go:482 +0x33
reflect.(*rtype).nameOff(...)
        /downloads/go/src/reflect/type.go:683
reflect.(*rtype).String(0x43d78e, 0x4428bd, 0x7f7d4128a560)
        /downloads/go/src/reflect/type.go:753 +0x36
reflect.(*rtype).Len(0x43d78e, 0x0)
        /downloads/go/src/reflect/type.go:965 +0x4f
runtime: unexpected return pc for runtime.schedule called from 0xa2ec7
stack: frame={sp:0x7f7d037fdd50, fp:0x7f7d037fddb8} stack=[0x7f7d02ffe2e8,0x7f7d037fdee8)
00007f7d037fdc50:  00007f7d8b480000  000000000101cb60
00007f7d037fdc60:  00007f7d037fdc00  0000000000457378 <runtime.runOneTimer+328>
00007f7d037fdc70:  000000000043d78e <runtime.findrunnable+2142>  0000000000c3d540
00007f7d037fdc80:  0000000000941000  0000000000000000
00007f7d037fdc90:  0000000000f94e20  00007f7d037fdcc0
00007f7d037fdca0:  0000000000447723 <reflect.resolveNameOff+51>  000000000043d78e <runtime.findrunnable+2142>
00007f7d037fdcb0:  000000008b480000  0000000000000000
00007f7d037fdcc0:  00007f7d037fdcf8  00000000004b4556 <reflect.(*rtype).String+54>
00007f7d037fdcd0:  000000000043d78e <runtime.findrunnable+2142>  000000008b480000
00007f7d037fdce0:  0000000000000016  00007f7d037fdd20
00007f7d037fdcf0:  000000000040cc3f <runtime.notesleep+159>  00007f7d037fdd40
00007f7d037fdd00:  00000000004b5fbf <reflect.(*rtype).Len+79>  000000000043d78e <runtime.findrunnable+2142>
00007f7d037fdd10:  00000000004428bd <runtime.acquirep+61>  00007f7d4128a560
00007f7d037fdd20:  00007f7d037fdd48  000000000043c348 <runtime.stopm+232>
00007f7d037fdd30:  000000c000035800  000000c000000001
00007f7d037fdd40:  00007f7d037fde30  000000000043e2fc <runtime.schedule+380>
00007f7d037fdd50: <000000000043d78e <runtime.findrunnable+2142>  0000000000000000
00007f7d037fdd60:  0000000000702180 <net/http.(*writerOnly).Write+16>  0000000552b50f00
00007f7d037fdd70:  0000000552bf3e81  0000000200000000
00007f7d037fdd80:  0000000000419c6e <runtime.sysUnused+222>  000000c000700000
00007f7d037fdd90:  0000000552bf3e81  0000000552b50fba
00007f7d037fdda0:  00007f7d037fddd8  000000000046a4e5 <runtime.nanotime1+101>
00007f7d037fddb0: !00000000000a2ec7 >0000000000000020
00007f7d037fddc0:  0000000000000016  000000003367b31e
00007f7d037fddd0:  000000000101d7b0  000000000000001f
00007f7d037fdde0:  00007f7d037fde30  000000000043e8c5 <runtime.checkTimers+533>
00007f7d037fddf0:  0000000552b50f1e  000000c000035800
00007f7d037fde00:  00007f7d037fde40  00007ffd1ff97d00
00007f7d037fde10:  000000c00002c000  0000000200000001
00007f7d037fde20:  000000c000702180  000000c000035800
00007f7d037fde30:  00007f7d037fde98  000000000043e47c <runtime.schedule+764>
00007f7d037fde40:  000000c000035800  0000000000000000
00007f7d037fde50:  0000000552b50f00  0000000552bf3e81
00007f7d037fde60:  0000000000000000  000000003367b07b
00007f7d037fde70:  00007f7d037fdeb0  00007ffd1ff97d00
00007f7d037fde80:  000000c000702180  000000000043e93b <runtime.parkunlock_c+43>
00007f7d037fde90:  0000000000feff00  00007f7d037fdec8
00007f7d037fdea0:  000000000043e9fd <runtime.park_m+157>  000000c000001080
00007f7d037fdeb0:  0000000000feff00

Host was running:
Linux XXX 4.15.0-1052-gcp #56-Ubuntu SMP Thu Dec 19 10:54:56 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[toothrot]

/cc @aclements @rsc @randall77 @ianlancetaylor

Could this be related to #37664?

[randall77]

Something is garbage here. The type pointer is 0x43d78e, which is not a valid pointer. Type pointers should always be 8-byte aligned. There's no indication where reflect.(*rtype).Len gets its argument - everything else in that trace is runtime functions, which can't call into reflect.

I'm not sure there's more we can do here without a reproducer.
I would recommend running the race detector, and auditing any uses of unsafe.

[bcmills]

greplogs --dashboard -md -l -e 'fatal error: runtime: name offset base pointer out of range'

2022-02-17T17:36:57-cda4201-eaf0405/linux-ppc64le-buildlet
2020-12-10T15:37:19-0aba8f2/plan9-arm
2019-09-17T18:22:37-7d16e44/plan9-386-0intro

[bcmills]

One more on linux-ppc64le-buildlet:

greplogs --dashboard -md -l -e 'fatal error: runtime: name offset base pointer out of range' --since=2022-02-25

2022-04-04T15:12:26-153e30b-f86f9a3/linux-ppc64le-buildlet