runtime: regression segfault on linux 3.4 on mipsle

[Jorropo]

In #77621 (comment) @wwqgtxx reports:

We encountered a problem on the MIPSLE platform running Linux 3.4, where a panic occurred during program startup:

futexwakeup addr=0x586232c returned -89
SIGSEGV: segmentation violation
PC=0x816f0 m=0 sigcode=128 addr=0x0

goroutine 0 gp=0x273ada0 m=0 mp=0x273ba00 [idle]:
runtime.futexwakeup(0x586232c, 0x1)
        runtime/os_linux.go:98 +0x84 fp=0x7fcd8240 sp=0x7fcd8214 pc=0x816f0
runtime.semawakeup(...)
        runtime/lock_futex.go:161
runtime.unlock2Wake(0x273ee7c, 0x0, 0x0)
        runtime/lock_spinbit.go:461 +0x2f0 fp=0x7fcd825c sp=0x7fcd8240 pc=0x3fd0c
runtime.unlock2(0x273ee7c)
        runtime/lock_spinbit.go:314 +0xc4 fp=0x7fcd8278 sp=0x7fcd825c pc=0x3f7a8
runtime.unlockWithRank(...)
        runtime/lockrank_off.go:35
runtime.unlock(...)
        runtime/lock_spinbit.go:261
runtime.mcommoninit(0x5862808, 0x2)
        runtime/proc.go:1030 +0x298 fp=0x7fcd829c sp=0x7fcd8278 pc=0x8beb0
runtime.allocm(0x5852f08, 0x18b4128, 0x2)
        runtime/proc.go:2326 +0x128 fp=0x7fcd82c8 sp=0x7fcd829c pc=0x8f2ec
runtime.newm(0x18b4128, 0x5852f08, 0x2)
        runtime/proc.go:2870 +0x70 fp=0x7fcd82e4 sp=0x7fcd82c8 pc=0x901a8
runtime.startm(0x5852f08, 0x1, 0x0)
        runtime/proc.go:3096 +0x1cc fp=0x7fcd8310 sp=0x7fcd82e4 pc=0x90a88
runtime.wakep()
        runtime/proc.go:3243 +0x1b4 fp=0x7fcd8330 sp=0x7fcd8310 pc=0xd7b4c
runtime.newproc.func1()
        runtime/proc.go:5305 +0x94 fp=0x7fcd8348 sp=0x7fcd8330 pc=0x97d00
runtime.systemstack(0x73ada0)
        runtime/asm_mipsx.s:186 +0x7c fp=0x7fcd834c sp=0x7fcd8348 pc=0xdeb10

goroutine 1 gp=0x5802148 m=0 mp=0x273ba00 [running, locked to thread]:
runtime.systemstack_switch()
        runtime/asm_mipsx.s:147 +0x8 fp=0x585e670 sp=0x585e66c pc=0xdea74
runtime.newproc(0x18b3fc4)
        runtime/proc.go:5298 +0x60 fp=0x585e688 sp=0x585e670 pc=0x97c5c
runtime.init.6()
        runtime/proc.go:363 +0x44 fp=0x585e690 sp=0x585e688 pc=0x89e84
runtime.doInit1(0x2665f40)
        runtime/proc.go:8103 +0x13c fp=0x585e7b0 sp=0x585e690 pc=0x9fa38
runtime.doInit(...)
        runtime/proc.go:8070
runtime.main()
        runtime/proc.go:203 +0x168 fp=0x585e7ec sp=0x585e7b0 pc=0x8991c
runtime.goexit({})
        runtime/asm_mipsx.s:657 +0x4 fp=0x585e7ec sp=0x585e7ec pc=0xe0a18

goroutine 2 gp=0x5802508 m=nil [runnable]:
runtime.forcegchelper()
        runtime/proc.go:366 fp=0x585efec sp=0x585efec pc=0x89e94
runtime.goexit({})
        runtime/asm_mipsx.s:657 +0x4 fp=0x585efec sp=0x585efec pc=0xe0a18
created by runtime.init.6 in goroutine 1
        runtime/proc.go:363 +0x44

r0   0x0        r1   0x1006
r2   0x1006     r3   0x6
r4   0x6        r5   0x1
r6   0x1        r7   0xffffff00
r8   0x1        r9   0x1
r10  0x1        r11  0x0
r12  0xffffffe2 r13  0x0
r14  0xffffffe1 r15  0x0
r16  0xbe2e5d27 r17  0xc5e1527a
r18  0xa561bdf7 r19  0x8b6cf700
r20  0x3f7f6e35 r21  0xcf38c650
r22  0x0        r23  0xffffffa0
r24  0xfe04629  r25  0x522c8331
r26  0x0        r27  0x0
r28  0xc2fa2760 r29  0x7fcd8214
r30  0x273ada0  r31  0x816e8
pc   0x816f0    link 0x816e8
lo   0x0        hi   0x0

Commenting out the calls to the timer_settime64 and futex_time64 functions in the runtime resolved the problem.

This issue did not appear in versions of Go prior to 1.26, and according to https://go.dev/wiki/MinimumRequirements, Linux kernel version 3.4 is still supported. Therefore, we believe this issue is not limited to older Android platforms and is a bug that needs fixing.

[Jorropo]

Ok well, the bug is trivial MIPSLE uses a different return code of -89 for ENOSYS.

I wasn't aware that ENOSYS wasn't portable and we hardcoded the check to -38.

So futex tries to call futex_time64, linux returns -ENOSYS (-89).
We check ret != -38, get false, so our code conclude futex_time64 is supported and return with the -89 error code.
The caller (futexwakeup) expects the operation to never fail and crashes the program when reaching this unreachable state.

---

If you update to 5.0+ or 5.1+ (I don't remember) you can't test for this bug since the call to futex_time64 will work.

[Jorropo]

@gopherbot please open a backport issue for 1.26

[gopherbot]

Backport issue(s) opened: #77731 (for 1.26).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[Jorropo]

Somewhat related to #75437, as in we have poor coverage of theses conditions.

[Jorropo]

Here are the architectures who feel like using fancy ENOSYS-es:

hugo@rikus ~/k/linux (master)> git grep -E "^#define\s+ENOSYS\b" -- arch/ include/
arch/alpha/include/uapi/asm/errno.h:#define     ENOSYS          78      /* Function not implemented */
arch/mips/include/uapi/asm/errno.h:#define ENOSYS               89      /* Function not implemented */
arch/parisc/include/uapi/asm/errno.h:#define    ENOSYS          251     /* Function not implemented */
arch/sparc/include/uapi/asm/errno.h:#define     ENOSYS          90      /* Function not implemented */
include/uapi/asm-generic/errno.h:#define        ENOSYS          38      /* Invalid system call number */

After cross referencing with the src/runtime/defs_linux_* files we do not have this bug on an other architecture (aka all other arches with that file all use 38 enosys).

[Jorropo]

cc @wwqgtxx given CI isn't exercising this codepath could you please try the fix in CL 747662 ? And report if the fix is successful.

You can download it with:

git fetch https://go.googlesource.com/go refs/changes/62/747662/1 && git checkout -b change-747662 FETCH_HEAD

inside a clone of the go repository.
Thanks.

[gabyhelp]

Related Issues

• runtime: SIGSEGV in runtime.deltimer on linux-mips-rtrk during ReadMemStats #43712 (closed)
• runtime: fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted [mipsle] #56426 (closed)
• runtime: fatal error: evacuation not done in time on MIPS64 #19740 (closed)
• runtime: misc/cgo/test.TestSetgid panic with `unexpected return pc for runtime.sigpanic` on linux-arm64-packet since 2022-06-04 #53374 (closed)
• cmd/go: cross build mips64/mips64le on ubuntu24 crash(only go1.22) #72125 (closed)
• runtime: TestNeedmDeadlock failure with SIGSEGV on linux-mips-rtrk builder #43625 (closed)
• runtime: SIGSEGV in (*mheap).alloc via mallocgc #43843 (closed)
• runtime: TestCrashDumpsAllThreads fails on various openbsd architectures #42464
• crypto/internal/fips140/subtle: Go 1.25 subtle.xorBytes panic on MIPS #74998 (closed)
• runtime: TestAsyncPreempt failed on linux-mips64le-mengzhuo builder #35742 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}