runtime: TestSegv/Segv failure with 'unknown pc' on linux/ppc64le

[bcmills]

#!watchflakes
post <- goarch == "ppc64le" && pkg == "runtime" && test == "TestSegv" && `unknown pc`

--- FAIL: TestSegv (0.00s)
    --- FAIL: TestSegv/Segv (0.02s)
        testenv.go:460: [/tmp/workdir-host-linux-riscv64-unmatched/tmp/go-build3484739273/testprogcgo.exe Segv] exit status: exit status 2
        crash_cgo_test.go:611: SIGSEGV: segmentation violation
            PC=0x3fe45b3c1a m=3 sigcode=0
            
            goroutine 0 [idle]:
            runtime: g 0: unknown pc 0x3fe45b3c1a
            stack: frame={sp:0x3fb7ffe3e0, fp:0x0} stack=[0x3fb77feb60,0x3fb7ffe760)

greplogs -l -e 'FAIL: TestSegv/Segv .*(?:\n[ ]{8}.*)*unknown pc' --since=2022-01-01
2022-05-16T19:48:35-99d6300/linux-riscv64-unmatched

Compare #50979, #47537.

(attn @golang/riscv64; CC @golang/runtime)

[mengzhuo]

coredump shows the command that failed is testprogcgo CgoExternalThreadSignal crash

I'm confused by the stack shows "net.(*file).getLineFromData+0x58", which is an offset to net.data.cap inside the function.

[mengzhuo]

@prattmic @cherrymui any advise?

[cherrymui]

The signal PC 0x3fe45b3c1a is weird. It doesn't look like a Go PC. If you have a core dump, can you check what that address is? Does it belong to (say) a memory mapping of a C shared library?

[mengzhuo]

No mapping around 0x3fe45b3c1a.

Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
             0x10000           0x22c000   0x21c000        0x0 /tmp/workdir-host-linux-riscv64-unmatched/tmp/go-build3484739273/testprogcgo.exe
            0x22c000           0x22d000     0x1000   0x21b000 /tmp/workdir-host-linux-riscv64-unmatched/tmp/go-build3484739273/testprogcgo.exe
            0x22d000           0x249000    0x1c000   0x21c000 /tmp/workdir-host-linux-riscv64-unmatched/tmp/go-build3484739273/testprogcgo.exe
        0x3fc176b000       0x3fc186e000   0x103000        0x0 /usr/lib/riscv64-linux-gnu/libc-2.33.so
        0x3fc186e000       0x3fc1871000     0x3000   0x102000 /usr/lib/riscv64-linux-gnu/libc-2.33.so
        0x3fc1871000       0x3fc1874000     0x3000   0x105000 /usr/lib/riscv64-linux-gnu/libc-2.33.so
        0x3fc187c000       0x3fc1890000    0x14000        0x0 /usr/lib/riscv64-linux-gnu/libpthread-2.33.so
        0x3fc1890000       0x3fc1891000     0x1000    0x13000 /usr/lib/riscv64-linux-gnu/libpthread-2.33.so
        0x3fc1891000       0x3fc1892000     0x1000    0x14000 /usr/lib/riscv64-linux-gnu/libpthread-2.33.so
        0x3fc18a2000       0x3fc18bd000    0x1b000        0x0 /usr/lib/riscv64-linux-gnu/ld-2.33.so
        0x3fc18be000       0x3fc18bf000     0x1000    0x1b000 /usr/lib/riscv64-linux-gnu/ld-2.33.so
        0x3fc18bf000       0x3fc18c1000     0x2000    0x1c000 /usr/lib/riscv64-linux-gnu/ld-2.33.so

maintenance info sections:
...
[71]     0x3fc18a2000->0x3fc18a3000 at 0x05a8c000: load47a ALLOC LOAD READONLY CODE HAS_CONTENTS
[72]     0x3fc18a3000->0x3fc18bd000 at 0x05a8d000: load47b ALLOC READONLY CODE
[73]     0x3fc18be000->0x3fc18bf000 at 0x05a8d000: load48 ALLOC LOAD READONLY HAS_CONTENTS
[74]     0x3fc18bf000->0x3fc18c1000 at 0x05a8e000: load49 ALLOC LOAD HAS_CONTENTS
[75]     0x3fffea7000->0x3fffec8000 at 0x05a90000: load50 ALLOC LOAD HAS_CONTENTS

[prattmic]

Just to double check, that is from the core dump of the exact crash shown above, not a different run, correct? Because those mappings seem similar enough for 0x3fe45b3c1a to be the randomized load address in a different run.

[mengzhuo]

Yes, It's from coredumpctl and here is the file
core_issue_52963.gz

I've tried to run the test 100 times and no luck.

[cherrymui]

Thanks. Then the address is not a PC at all. It's unclear to me how we get there, or how it gets into the signal context.

[mengzhuo]

Debug logs, Something is weird.
Failed log shows frame.pc should be 3fb7ffe548, but findfunc got a 3fe45b3c1a.

            0x0000003fb7ffe3e0: <0x0000003fb7ffe480  0x00000000000f4240 <net.(*file).getLineFromData+0x0000000000000058> 
            0x0000003fb7ffe3f0:  0x0000003fb7ffe548  0x000000000010efb6 
            0x0000003fb7ffe400:  0x00000000000f4240 <net.(*file).getLineFromData+0x0000000000000058>  0x0000003fb7ffe560 

This frame.lr != 0 so it should be two defer of frame.sp 0x3fb7ffe560 but unfortunately corefile didn't dump those memory.

IMHO three possible explanations:

1. compiler error that miscompiled two defers(will look into)
2. frame stack had changed before frame.pc defers (highly likely)
3. regabi misuse register (accident clobber? I double it)

[cherrymui]

frame.pc should be 3fb7ffe548

Where did you see that? I didn't see it.

PC=0x3fe45b3c1a m=3 sigcode=0

This line is printed from the signal handler, which indicates the signal context's PC is 0x3fe45b3c1a. For such a bad PC I don't think the traceback code can do anything useful. The question is why we get such a PC in the signal context.

[mengzhuo]

frame.pc should be 3fb7ffe548

Where did you see that? I didn't see it.

Just my understand on frame stack:

            0x0000003fb7ffe3e0: <0x0000003fb7ffe480  0x00000000000f4240 <net.(*file).getLineFromData+0x0000000000000058> 
            0x0000003fb7ffe3f0:  0x0000003fb7ffe548  0x000000000010efb6 
            0x0000003fb7ffe400:  0x00000000000f4240 <net.(*file).getLineFromData+0x0000000000000058>  0x0000003fb7ffe560 
            0x0000003fb7ffe410:  0x000000000010efb6  0x8458d8a6295c2900 

which aligned with frame struct

// stack traces
type stkframe struct {
	fn       funcInfo   // 0x0000003fb7ffe480  0x00000000000f4240
	pc       uintptr    // 0x0000003fb7ffe548
	continpc uintptr    // 0x000000000010efb6
	lr       uintptr    // 0x00000000000f4240 
	sp       uintptr    // 0x0000003fb7ffe560 
	fp       uintptr    // 0x000000000010efb6  
	varp     uintptr    // 0x8458d8a6295c2900 
        .....
}

The question is why we get such a PC in the signal context.

The code seems fine (why sigcode=0?)
The ucontext with cgo padding didn't generated by cgo, misaligned?

runtime/defs_linux_riscv64.go

type ucontext struct {
	uc_flags     uint64
	uc_link      *ucontext
	uc_stack     stackt
	uc_sigmask   usigset
	uc_x__unused [0]uint8 // according to signal.h it's 0
	uc_pad_cgo_0 [8]byte
	uc_mcontext  sigcontext
}

[cherrymui]

No. That hex dump is not the stkframe structure. They are unrelated. Also, traceback probably doesn't matter here. We start with a bad signal PC in the first place.

why sigcode=0?

sigcode=0 (i.e. _SI_USER) is expected. The test is testing a user-sent SIGSEGV.

Hmmm, if the ucontext fields are wrong, I'd think many things would go wrong, not just this failure.

[mengzhuo]

You're right about ucontext fields.
I tried setting this padding to 0 and can't even build it.