runtime/pprof: TestStackBarrierProfiling fails with "SIGTRAP: trace trap PC=0x459cfd m=2" on linux

[mikioh]

all.bash fails on fedora-vm,

uname -a
Linux fedora-amd64.localdomain 4.2.8-200.fc22.x86_64 #1 SMP Tue Dec 15 16:50:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

git rev-parse HEAD
acc7161320038ee6377d589890526cd11912b4e8

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7851
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

with the following dying message:

--- FAIL: TestStackBarrierProfiling (0.02s)
    pprof_test.go:363: subprocess failed with exit status 2:
        SIGTRAP: trace trap
        PC=0x459cfd m=2

        goroutine 0 [idle]:
        runtime.morestack()
            /go/src/runtime/asm_amd64.s:331 +0x1d

        goroutine 4 [running]:
        runtime.systemstack_switch()
            /go/src/runtime/asm_amd64.s:245 fp=0xc8200414f8 sp=0xc8200414f0
        runtime.gcMarkTermination()
            /go/src/runtime/mgc.go:1255 +0x3a9 fp=0xc820041748 sp=0xc8200414f8
        runtime.gcMarkDone()
            /go/src/runtime/mgc.go:1141 +0x286 fp=0xc820041768 sp=0xc820041748
        runtime.gcBgMarkWorker(0xc82001c000)
            /go/src/runtime/mgc.go:1465 +0x4d8 fp=0xc8200417d8 sp=0xc820041768
        runtime.goexit()
            /go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8200417e0 sp=0xc8200417d8
        created by runtime.gcBgMarkStartWorkers
            /go/src/runtime/mgc.go:1335 +0x93

        goroutine 1 [chan receive]:
        testing.RunTests(0x610ac0, 0x6b03e0, 0x8, 0x8, 0x7f2e00000001)
            /go/src/testing/testing.go:583 +0x8d2
        testing.(*M).Run(0xc8200bef38, 0x408ee7)
            /go/src/testing/testing.go:515 +0x81
        main.main()
            runtime/pprof/_test/_testmain.go:70 +0x117

        goroutine 19 [runnable]:
        runtime/pprof_test.deepStack(0x1c1, 0xc820161800)
            /go/src/runtime/pprof/pprof_test.go:393 +0x52
        runtime/pprof_test.deepStack(0x1c2, 0xc820161400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1c3, 0xc820161000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1c4, 0xc820160c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1c5, 0xc820160800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1c6, 0xc820160400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1c7, 0xc820160000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1c8, 0xc82015fc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1c9, 0xc82015f800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ca, 0xc82015f400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1cb, 0xc82015f000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1cc, 0xc82015ec00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1cd, 0xc82015e800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ce, 0xc82015e400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1cf, 0xc82015e000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d0, 0xc82015dc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d1, 0xc82015d800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d2, 0xc82015d400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d3, 0xc82015d000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d4, 0xc82015cc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d5, 0xc82015c800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d6, 0xc82015c400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d7, 0xc82015c000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d8, 0xc82015bc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1d9, 0xc82015b800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1da, 0xc82015b400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1db, 0xc82015b000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1dc, 0xc82015ac00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1dd, 0xc82015a800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1de, 0xc82015a400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1df, 0xc82015a000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e0, 0xc820159c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e1, 0xc820159800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e2, 0xc820159400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e3, 0xc820159000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e4, 0xc820158c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e5, 0xc820158800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e6, 0xc820158400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e7, 0xc820158000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e8, 0xc820157c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1e9, 0xc820157800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ea, 0xc820157400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1eb, 0xc820157000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ec, 0xc820156c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ed, 0xc820156800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ee, 0xc820156400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ef, 0xc820156000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f0, 0xc820155c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f1, 0xc820155800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f2, 0xc820155400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f3, 0xc820155000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f4, 0xc820154c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f5, 0xc820154800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f6, 0xc820154400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f7, 0xc820154000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f8, 0xc820153c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1f9, 0xc820153800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1fa, 0xc820153400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1fb, 0xc820153000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1fc, 0xc820152c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1fd, 0xc820152800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1fe, 0xc820152400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x1ff, 0xc820152000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x200, 0xc820151c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x201, 0xc820151800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x202, 0xc820151400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x203, 0xc820151000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x204, 0xc820150c00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x205, 0xc820150800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x206, 0xc820150400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x207, 0xc820150000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x208, 0xc82014fc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x209, 0xc82014f800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x20a, 0xc82014f400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x20b, 0xc82014f000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x20c, 0xc82014ec00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x20d, 0xc82014e800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x20e, 0xc82014e400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x20f, 0xc82014e000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x210, 0xc82014dc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x211, 0xc82014d800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x212, 0xc82014d400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x213, 0xc82014d000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x214, 0xc82014cc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x215, 0xc82014c800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x216, 0xc82014c400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x217, 0xc82014c000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x218, 0xc82014bc00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x219, 0xc82014b800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x21a, 0xc82014b400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x21b, 0xc82014b000)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x21c, 0xc82014ac00)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x21d, 0xc82014a800)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        runtime/pprof_test.deepStack(0x21e, 0xc82014a400)
            /go/src/runtime/pprof/pprof_test.go:394 +0x90
        created by testing.RunTests
            /go/src/testing/testing.go:582 +0x892

        goroutine 20 [runnable]:
        runtime/pprof.profileWriter(0x7f2e60199650, 0xc82009a310)
            /go/src/runtime/pprof/pprof.go:597
        created by runtime/pprof.StartCPUProfile
            /go/src/runtime/pprof/pprof.go:593 +0x145

        rax    0xf80
        rbx    0xc82002a400
        rcx    0xc820000900
        rdx    0x0
        rdi    0x42eec0
        rsi    0xc820000900
        rbp    0x2
        rsp    0xc82003af78
        r8     0xc820000f00
        r9     0x47ae147ae147ae15
        r10    0x4d9e5
        r11    0x203
        r12    0x42f150
        r13    0x60fe08
        r14    0x0
        r15    0x8
        rip    0x459cfd
        rflags 0x246
        cs     0x33
        fs     0x0
        gs     0x0
FAIL

[rsc]

Pretty sure this is a duplicate of #13363. Not that that helps. /cc @aclements

[aclements]

I actually don't think this is a duplicate of #13363. It's crashing a few lines after morestack successfully got the m, and is indicating we blew the system stack. Unfortunately, the traceback on the system stack cut off at the morestack right at the top because that's normally the right thing to do, but it hid whatever was overflowing the system stack.

@mikioh, are you able to reproduce this? If so, could you try https://go-review.googlesource.com/18761, which should fix #13363 (but I suspect won't fix this).

[mikioh]

are you able to reproduce this?

Yesterday I tried and failed to reproduce it. Also I've stashed my Linux VM for investigating signal stack setup issue on NetBSD and DragonFly BSD.

[aclements]

Now that I understand the cause of #13363 (the same G running on two Ms at once), I think it is possible this had the same root cause, mostly because pretty much anything could happen once it got in to that state and the traceback shows it's clearly around the area where that problem occurred.

I'm going to close this as a dup of #13363, but if it happens again, please reopen.