runtime: fatal error: invalid runtime symbol table (not able to reproduce)

[dotwaffle]

What version of Go are you using (go version)?

$ go version
go version go1.11.5 linux/amd64
(running inside Docker container on 64 CPU host, 4 CPU quota)

Does this issue reproduce with the latest release?

I have not been able to reproduce the issue. The dataset is much too large, and code far too complex to determine the trigger.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/data/gvm/pkgsets/go1.11.5/global"
GOPROXY=""
GORACE=""
GOROOT="/data/gvm/gos/go1.11.5"
GOTMPDIR=""
GOTOOLDIR="/data/gvm/gos/go1.11.5/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build246258981=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I ran a newly compiled version of my program, and it crashed. The changes I had made had not yet been reached during operation of the program, it was still in a "bootstrapping" phase of loading in data.

GOGC was set to 50.
GODEBUG was set to gctrace=1

What did you expect to see?

Normal execution.

What did you see instead?

[...logs, including gc runs...]
runtime: invalid pc-encoded table f=github.com/mitchellh/mapstructure.typedDecodeHook pc=0x7b6b06 targetpc=0x7b6b06 tab=[0/0]0x0
	value=0 until pc=0x7b67df
	value=208 until pc=0x7b6a08
	value=0 until pc=0x7b6a09
	value=208 until pc=0x7b6a8a
	value=0 until pc=0x7b6a8b
	value=208 until pc=0x7b6afc
	value=0 until pc=0x7b6b06
fatal error: invalid runtime symbol table

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0x7fce1ed23390
stack: frame={sp:0xc000169a68, fp:0xc000169ac0} stack=[0xc000162000,0xc00016a000)
000000c000169968:  0000000000000000  0000000000000000
000000c000169978:  0000000000000000  0000000000000000
000000c000169988:  0000000000000000  0000000000000000
000000c000169998:  000000c000169bf0  000000c000169ac0
000000c0001699a8:  000000c000169a58  00000000004426ca <runtime.sigtrampgo+538>
000000c0001699b8:  000000c00000001b  000000c000169bf0
000000c0001699c8:  000000c000169ac0  000000c0003da480
000000c0001699d8:  0000000000000000  0000000000000000
000000c0001699e8:  0000000000000000  0000000000000000
000000c0001699f8:  0000000000000000  0000000000000000
000000c000169a08:  0000000000000000  000000c0003da480
000000c000169a18:  0000000000000000  0000000000000000
000000c000169a28:  0000000000000000  0000000000000000
000000c000169a38:  0000000000000000  0000000000000000
000000c000169a48:  0000000000000000  0000000000000000
000000c000169a58:  000000c000169ab0  000000000045c563 <runtime.sigtramp+67>
000000c000169a68: <000000000000001b  000000c000169bf0
000000c000169a78:  000000c000169ac0  000000000001972d
000000c000169a88:  0000000000019726  000000000001972d
000000c000169a98:  000000c0af6a541c  000000c000169ab0
000000c000169aa8:  000000c0000f1180  00007fce0affcd20
000000c000169ab8: !00007fce1ed23390 >0000000000000007
000000c000169ac8:  0000000000000000  000000c000162000
000000c000169ad8:  0000000000000000  0000000000008000
000000c000169ae8:  0000000000000394  0000000000000001
000000c000169af8:  00007fce0affcd10  0029a7984f371e34
000000c000169b08:  000000c0af6a541c  000000000001972d
000000c000169b18:  0000000000019726  000000000001972d
000000c000169b28:  000000c00036c234  000000c00036c231
000000c000169b38:  00007fce0affcd20  000000c0000f1180
000000c000169b48:  00007fce0affcd20  00007fffa5d61a30
000000c000169b58:  000000c00036c234  00007fce0affcd10
000000c000169b68:  000000000045c3e5 <runtime.nanotime+69>  0000000000000202
000000c000169b78:  002b000000000033  0000000000000000
000000c000169b88:  0000000000000000  0000000000000000
000000c000169b98:  0000000000000000  000000c000169c80
000000c000169ba8:  0000000000000000  0000000000000000
000000c000169bb8:  0000000000000000
runtime.throw(0xe93fd6, 0x1c)
	/data/gvm/gos/go1.11.5/src/runtime/panic.go:608 +0x72
runtime.pcvalue(0x126e5d8, 0x1633ba0, 0x2aae23, 0x7b6b06, 0xc000169480, 0x1, 0x0)
	/data/gvm/gos/go1.11.5/src/runtime/symtab.go:791 +0x50a
runtime.funcspdelta(0x126e5d8, 0x1633ba0, 0x7b6b06, 0xc000169480, 0x0)
	/data/gvm/gos/go1.11.5/src/runtime/symtab.go:843 +0x5f
runtime.gentraceback(0x45c3e5, 0x7fce0affcd10, 0x0, 0xc0003da480, 0x0, 0xc0001696d8, 0x40, 0x0, 0x0, 0x6, ...)
	/data/gvm/gos/go1.11.5/src/runtime/traceback.go:208 +0x1828
runtime.sigprof(0x45c3e5, 0x7fce0affcd10, 0x0, 0xc0003da480, 0xc0000f1180)
	/data/gvm/gos/go1.11.5/src/runtime/proc.go:3823 +0x4c8
runtime.sighandler(0xc00000001b, 0xc000169bf0, 0xc000169ac0, 0xc0003da480)
	/data/gvm/gos/go1.11.5/src/runtime/signal_sighandler.go:38 +0x73c
runtime.sigtrampgo(0x1b, 0xc000169bf0, 0xc000169ac0)
	/data/gvm/gos/go1.11.5/src/runtime/signal_unix.go:351 +0x21a
runtime: unexpected return pc for runtime.sigtramp called from 0x7fce1ed23390
stack: frame={sp:0xc000169a68, fp:0xc000169ac0} stack=[0xc000162000,0xc00016a000)
000000c000169968:  0000000000000000  0000000000000000
000000c000169978:  0000000000000000  0000000000000000
000000c000169988:  0000000000000000  0000000000000000
000000c000169998:  000000c000169bf0  000000c000169ac0
000000c0001699a8:  000000c000169a58  00000000004426ca <runtime.sigtrampgo+538>
000000c0001699b8:  000000c00000001b  000000c000169bf0
000000c0001699c8:  000000c000169ac0  000000c0003da480
000000c0001699d8:  0000000000000000  0000000000000000
000000c0001699e8:  0000000000000000  0000000000000000
000000c0001699f8:  0000000000000000  0000000000000000
000000c000169a08:  0000000000000000  000000c0003da480
000000c000169a18:  0000000000000000  0000000000000000
000000c000169a28:  0000000000000000  0000000000000000
000000c000169a38:  0000000000000000  0000000000000000
000000c000169a48:  0000000000000000  0000000000000000
000000c000169a58:  000000c000169ab0  000000000045c563 <runtime.sigtramp+67>
000000c000169a68: <000000000000001b  000000c000169bf0
000000c000169a78:  000000c000169ac0  000000000001972d
000000c000169a88:  0000000000019726  000000000001972d
000000c000169a98:  000000c0af6a541c  000000c000169ab0
000000c000169aa8:  000000c0000f1180  00007fce0affcd20
000000c000169ab8: !00007fce1ed23390 >0000000000000007
000000c000169ac8:  0000000000000000  000000c000162000
000000c000169ad8:  0000000000000000  0000000000008000
000000c000169ae8:  0000000000000394  0000000000000001
000000c000169af8:  00007fce0affcd10  0029a7984f371e34
000000c000169b08:  000000c0af6a541c  000000000001972d
000000c000169b18:  0000000000019726  000000000001972d
000000c000169b28:  000000c00036c234  000000c00036c231
000000c000169b38:  00007fce0affcd20  000000c0000f1180
000000c000169b48:  00007fce0affcd20  00007fffa5d61a30
000000c000169b58:  000000c00036c234  00007fce0affcd10
000000c000169b68:  000000000045c3e5 <runtime.nanotime+69>  0000000000000202
000000c000169b78:  002b000000000033  0000000000000000
000000c000169b88:  0000000000000000  0000000000000000
000000c000169b98:  0000000000000000  000000c000169c80
000000c000169ba8:  0000000000000000  0000000000000000
000000c000169bb8:  0000000000000000
runtime.sigtramp(0x7, 0x0, 0xc000162000, 0x0, 0x8000, 0x394, 0x1, 0x7fce0affcd10, 0x29a7984f371e34, 0xc0af6a541c, ...)
	/data/gvm/gos/go1.11.5/src/runtime/sys_linux_amd64.s:353 +0x43

goroutine 44 [GC worker (idle)]:
runtime.systemstack_switch()
	/data/gvm/gos/go1.11.5/src/runtime/asm_amd64.s:311 fp=0xc0003a5760 sp=0xc0003a5758 pc=0x458890
runtime.gcBgMarkWorker(0xc000075900)
	/data/gvm/gos/go1.11.5/src/runtime/mgc.go:1826 +0x1b8 fp=0xc0003a57d8 sp=0xc0003a5760 pc=0x41ccb8
runtime.goexit()
	/data/gvm/gos/go1.11.5/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003a57e0 sp=0xc0003a57d8 pc=0x45a971
created by runtime.gcBgMarkStartWorkers
	/data/gvm/gos/go1.11.5/src/runtime/mgc.go:1720 +0x77

goroutine 1 [chan send]:
stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap(0xc00022dce0, 0x7fffa5c07c0f, 0x10, 0x0, 0x0)
	/data/bgptools/codaload/bootstrap.go:88 +0x417
stash.corp.netflix.com/nco/bgptools/cmd.glob..func1(0x167b7e0, 0xc0001e0380, 0x0, 0xe)
	/data/bgptools/cmd/check.go:75 +0xc8f
github.com/spf13/cobra.(*Command).execute(0x167b7e0, 0xc0001e02a0, 0xe, 0xe, 0x167b7e0, 0xc0001e02a0)
	/data/gvm/pkgsets/go1.11.5/global/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0x167ba40, 0xc0001dbf88, 0x4075a0, 0xc00012e058)
	/data/gvm/pkgsets/go1.11.5/global/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0x167ba40, 0x0, 0x0)
	/data/gvm/pkgsets/go1.11.5/global/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800 +0x2b
main.main()
	/data/bgptools/main.go:9 +0x2d

goroutine 5 [sleep]:
time.Sleep(0x5f5e100)
	/data/gvm/gos/go1.11.5/src/runtime/time.go:105 +0x14f
runtime/pprof.profileWriter(0xf7f580, 0xc00000e130)
	/data/gvm/gos/go1.11.5/src/runtime/pprof/pprof.go:778 +0x6f
created by runtime/pprof.StartCPUProfile
	/data/gvm/gos/go1.11.5/src/runtime/pprof/pprof.go:763 +0x10d

goroutine 6 [semacquire]:
sync.runtime_SemacquireMutex(0xc00022dcf4, 0x0)
	/data/gvm/gos/go1.11.5/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc00022dcf0)
	/data/gvm/gos/go1.11.5/src/sync/mutex.go:134 +0xff
sync.(*RWMutex).Lock(0xc00022dcf0)
	/data/gvm/gos/go1.11.5/src/sync/rwmutex.go:93 +0x2d
stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).insertPrefixData(0xc00022dce0, 0xc1910c0890, 0x10, 0x10, 0xc1910c0868, 0x4, 0x4, 0xc190e4d160, 0x5, 0x6, ...)
	/data/bgptools/codaload/table-ops.go:56 +0x7e
stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap.func1(0xc0000395f0, 0xc000116e40, 0xc00022dce0)
	/data/bgptools/codaload/bootstrap.go:66 +0x150
created by stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap
	/data/bgptools/codaload/bootstrap.go:61 +0x331

goroutine 7 [semacquire]:
sync.runtime_SemacquireMutex(0xc00022dcf4, 0x1)
	/data/gvm/gos/go1.11.5/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc00022dcf0)
	/data/gvm/gos/go1.11.5/src/sync/mutex.go:134 +0xff
sync.(*RWMutex).Lock(0xc00022dcf0)
	/data/gvm/gos/go1.11.5/src/sync/rwmutex.go:93 +0x2d
stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).insertPrefixData(0xc00022dce0, 0xc191309d60, 0x10, 0x10, 0xc191309d48, 0x4, 0x4, 0xc191309da0, 0x1, 0x4, ...)
	/data/bgptools/codaload/table-ops.go:56 +0x7e
stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap.func1(0xc0000395f0, 0xc000116e40, 0xc00022dce0)
	/data/bgptools/codaload/bootstrap.go:66 +0x150
created by stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap
	/data/bgptools/codaload/bootstrap.go:61 +0x331

goroutine 8 [semacquire]:
sync.runtime_SemacquireMutex(0xc00022dcf4, 0x0)
	/data/gvm/gos/go1.11.5/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc00022dcf0)
	/data/gvm/gos/go1.11.5/src/sync/mutex.go:134 +0xff
sync.(*RWMutex).Lock(0xc00022dcf0)
	/data/gvm/gos/go1.11.5/src/sync/rwmutex.go:93 +0x2d
stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).insertPrefixData(0xc00022dce0, 0xc1910d71f0, 0x10, 0x10, 0xc1910d71dc, 0x4, 0x4, 0xc190e4db40, 0x5, 0x6, ...)
	/data/bgptools/codaload/table-ops.go:56 +0x7e
stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap.func1(0xc0000395f0, 0xc000116e40, 0xc00022dce0)
	/data/bgptools/codaload/bootstrap.go:66 +0x150
created by stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap
	/data/bgptools/codaload/bootstrap.go:61 +0x331

goroutine 9 [running]:
	goroutine running on other thread; stack unavailable
created by stash.corp.netflix.com/nco/bgptools/codaload.(*Coda).Bootstrap
	/data/bgptools/codaload/bootstrap.go:61 +0x331

[agnivade]

/cc @randall77 @aclements

[randall77]

Strange. It is complaining that it's trying to find out how big the stack frame is for the function github.com/mitchellh/mapstructure.typedDecodeHook. But the pc is just beyond the end of the function, so the lookup fails.

The weird part is that I don't see that function in the tracebacks anywhere. So why is the runtime asking about it? How is it getting that pc, given that it couldn't possibly be the pc of an instruction actually executed by the program? There's some additional sigtramp nonsense in there which may be obscuring the part of the stack trace that contains the mentioned function.

Could you verify that in your executable, address 0x7b6b06 is in that function mentioned above, and that it is junk? It should look like this:

> objdump -d <executable>
  ...
00000000007b6??0 <github.com/mitchellh/mapstructure.typedDecodeHook>:
  ...
  7b6b05:       c3                      retq     // or maybe jmpq or ud2 a few addresses earlier
  7b6b06:       cc                      int3      // the address in question
  7b6b07:       cc                      int3   
  7b6b08:       cc                      int3   
  7b6b09:       cc                      int3   
  7b6b0a:       cc                      int3   
  7b6b0b:       cc                      int3   
  7b6b0c:       cc                      int3   
  7b6b0d:       cc                      int3   
  7b6b0e:       cc                      int3   
  7b6b0f:       cc                      int3   

00000000007b6b10 <someOtherFunction>:

If you could paste in the issue the actual disassembly around 7b6b06 that would help.

Even if that checks out, I'm not sure what to do about it. It seems like the pc got corrupted somehow.
Possibly related to #24925 or #27540?

[dotwaffle]

# objdump -d bgptools | grep -C 20 7b6b06
  7b6ab8:	e9 d0 fe ff ff       	jmpq   7b698d <github.com/mitchellh/mapstructure.typedDecodeHook+0x1cd>
  7b6abd:	31 f6                	xor    %esi,%esi
  7b6abf:	31 c0                	xor    %eax,%eax
  7b6ac1:	31 c9                	xor    %ecx,%ecx
  7b6ac3:	e9 e8 fd ff ff       	jmpq   7b68b0 <github.com/mitchellh/mapstructure.typedDecodeHook+0xf0>
  7b6ac8:	48 8d 3d 11 e7 ec 00 	lea    0xece711(%rip),%rdi        # 16851e0 <reflect.dummy+0x10>
  7b6acf:	48 8b 84 24 e0 00 00 	mov    0xe0(%rsp),%rax
  7b6ad6:	00
  7b6ad7:	e8 a4 3e ca ff       	callq  45a980 <runtime.gcWriteBarrier>
  7b6adc:	0f 57 c0             	xorps  %xmm0,%xmm0
  7b6adf:	e9 93 fd ff ff       	jmpq   7b6877 <github.com/mitchellh/mapstructure.typedDecodeHook+0xb7>
  7b6ae4:	48 8b 84 24 e0 00 00 	mov    0xe0(%rsp),%rax
  7b6aeb:	00
  7b6aec:	e9 86 fd ff ff       	jmpq   7b6877 <github.com/mitchellh/mapstructure.typedDecodeHook+0xb7>
  7b6af1:	31 f6                	xor    %esi,%esi
  7b6af3:	31 c0                	xor    %eax,%eax
  7b6af5:	31 c9                	xor    %ecx,%ecx
  7b6af7:	e9 b4 fd ff ff       	jmpq   7b68b0 <github.com/mitchellh/mapstructure.typedDecodeHook+0xf0>
  7b6afc:	e8 ef 1e ca ff       	callq  4589f0 <runtime.morestack_noctxt>
  7b6b01:	e9 ba fc ff ff       	jmpq   7b67c0 <github.com/mitchellh/mapstructure.typedDecodeHook>
  7b6b06:	cc                   	int3
  7b6b07:	cc                   	int3
  7b6b08:	cc                   	int3
  7b6b09:	cc                   	int3
  7b6b0a:	cc                   	int3
  7b6b0b:	cc                   	int3
  7b6b0c:	cc                   	int3
  7b6b0d:	cc                   	int3
  7b6b0e:	cc                   	int3
  7b6b0f:	cc                   	int3

00000000007b6b10 <github.com/mitchellh/mapstructure.DecodeHookExec>:
  7b6b10:	64 48 8b 0c 25 f8 ff 	mov    %fs:0xfffffffffffffff8,%rcx
  7b6b17:	ff ff
  7b6b19:	48 3b 61 10          	cmp    0x10(%rcx),%rsp
  7b6b1d:	0f 86 47 02 00 00    	jbe    7b6d6a <github.com/mitchellh/mapstructure.DecodeHookExec+0x25a>
  7b6b23:	48 83 ec 68          	sub    $0x68,%rsp
  7b6b27:	48 89 6c 24 60       	mov    %rbp,0x60(%rsp)
  7b6b2c:	48 8d 6c 24 60       	lea    0x60(%rsp),%rbp
  7b6b31:	48 8b 44 24 70       	mov    0x70(%rsp),%rax
  7b6b36:	48 89 04 24          	mov    %rax,(%rsp)

(edited above as I accidentally ran it on macos instead of the original linux env)

I can confirm that the function header is github.com/mitchellh/mapstructure.typedDecodeHook: -- please shout if you need further context.

[randall77]

Thanks for all the info.
Unfortunately I don't think I can make any more progress here. Without a standalone reproducer it's hard to investigate. This does provide another data point for #27540, if they are in fact related.

[dotwaffle]

@randall77 no worries, I figured it was worth posting, just in case it was useful. Feel free to close it unless you want to keep it open for reference.

[agnivade]

Closing since the data is insufficient. Please feel free to reopen if you are able to reproduce this.