runtime: crash on linux/arm64 during os.Setenv #69829

cbandy · 2024-10-09T19:27:06Z

Go version

go1.22.6 linux/arm64

Output of `go env` in your module/workspace:

n/a

What did you do?

I was running an application test suite against an arm64 K3s built with go1.22.6:

$ docker run --rm --entrypoint=k3s docker.io/rancher/k3s:v1.31.1-k3s1 --version
k3s version v1.31.1+k3s1 (452dbbc1)
go version go1.22.6

What did you see happen?

kubelet reported that an exec probe failed with SIGSEGV:

Liveness probe failed: SIGSEGV: segmentation violation
…
        goroutine 1 gp=0x40000021c0 m=0 mp=0xffff80951580 [running, locked to thread]:
        runtime.systemstack_switch()
        	runtime/asm_arm64.s:200 +0x8 fp=0x400018f980 sp=0x400018f970 pc=0xffff8013c228
        runtime.setenv_c({0x4000166190?, 0x1?}, {0x4000166195, 0x3c})
        	runtime/env_posix.go:54 +0xb0 fp=0x400018f9e0 sp=0x400018f980 pc=0xffff800d6d10
        syscall.runtimeSetenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
        	runtime/runtime.go:201 +0x2c fp=0x400018fa10 sp=0x400018f9e0 pc=0xffff80139c4c
        syscall.Setenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
        	syscall/env_unix.go:122 +0x310 fp=0x400018fac0 sp=0x400018fa10 pc=0xffff80185870
        os.Setenv({0x4000166190?, 0x3c?}, {0x4000166195?, 0xffffffffffffffff?})
        	os/env.go:120 +0x24 fp=0x400018fb00 sp=0x400018fac0 pc=0xffff801b3ea4
…

full message

Liveness probe failed: SIGSEGV: segmentation violation
        PC=0xffff804967e4 m=0 sigcode=1 addr=0x0
        
        goroutine 0 gp=0xffff80950d00 m=0 mp=0xffff80951580 [idle]:
        runtime: g 0 gp=0xffff80950d00: unknown pc 0xffff804967e4
        stack: frame={sp:0xffffe77cbef0, fp:0x0} stack=[0xffffe77ad190,0xffffe77cc270)
        0x0000ffffe77cbdf0:  0x0000ffff800dce40 <runtime.persistentalloc.func1+0x0000000000000030>  0x0000000000000000 
        0x0000ffffe77cbe00:  0xffffffffffffefe8  0x0000000000000080 
        0x0000ffffe77cbe10:  0xffffffffffffff80  0x0000ffff80951580 
        0x0000ffffe77cbe20:  0x0000004000035220  0x0000ffff3936eb80 
        0x0000ffffe77cbe30:  0xfffffffffffff800  0x0000ffffe77cbe68 
        0x0000ffffe77cbe40:  0x0000ffffe77cbe90  0x0000ffff80496388 
        0x0000ffffe77cbe50:  0x0000ffffe77cbe90  0x0000ffff80496710 
        0x0000ffffe77cbe60:  0x0000000000000002  0x0000aaaac2860130 
        0x0000ffffe77cbe70:  0x0000ffffe77cbe90  0x0000ffff80496710 
        0x0000ffffe77cbe80:  0x0000000000000002  0x0000ffff3936eb80 
        0x0000ffffe77cbe90:  0x0000ffffe77cbef0  0x0000ffff80496910 
        0x0000ffffe77cbea0:  0x0000000000000002  0x0000aaaac2860108 
        0x0000ffffe77cbeb0:  0x0000000000000023  0x0000ffff809b7000 
        0x0000ffffe77cbec0:  0x0000000000000023  0x0000000000003ff0 
        0x0000ffffe77cbed0:  0x0000000000000027  0x0000ffff8056d000 
        0x0000ffffe77cbee0:  0x0000000000000002  0x0000ffffe77cbf68 
        0x0000ffffe77cbef0: <0x0000ffffe77cbf50  0x0000ffff80496910 
        0x0000ffffe77cbf00:  0x0000000000000002  0x0000aaaac28600e0 
        0x0000ffffe77cbf10:  0x000000000000001f  0x0000ffff809b7000 
        0x0000ffffe77cbf20:  0x000000000000001f  0x0000000000001ff0 
        0x0000ffffe77cbf30:  0x0000000000000023  0x0000ffff8056d000 
        0x0000ffffe77cbf40:  0x0000000000000002  0x0000ffffe77cbfb8 
        0x0000ffffe77cbf50:  0x0000ffffe77cbfb0  0x0000ffff80496910 
        0x0000ffffe77cbf60:  0x0000000000000002  0x0000aaaac28600b8 
        0x0000ffffe77cbf70:  0x000000000000001b  0x0000ffff809b7000 
        0x0000ffffe77cbf80:  0x000000000000001b  0x0000000000000ff0 
        0x0000ffffe77cbf90:  0x000000000000001f  0x0000ffff8056d000 
        0x0000ffffe77cbfa0:  0x0000000000000002  0x0000ffffe77cbfd8 
        0x0000ffffe77cbfb0:  0x0000ffffe77cc010  0x0000ffff80496910 
        0x0000ffffe77cbfc0:  0x0000000000000002  0x0000aaaac2860090 
        0x0000ffffe77cbfd0:  0x0000000000000017  0x0000ffff809b7000 
        0x0000ffffe77cbfe0:  0x0000000000000017  0x00000000000007f0 
        runtime: g 0 gp=0xffff80950d00: unknown pc 0xffff804967e4
        stack: frame={sp:0xffffe77cbef0, fp:0x0} stack=[0xffffe77ad190,0xffffe77cc270)
        0x0000ffffe77cbdf0:  0x0000ffff800dce40 <runtime.persistentalloc.func1+0x0000000000000030>  0x0000000000000000 
        0x0000ffffe77cbe00:  0xffffffffffffefe8  0x0000000000000080 
        0x0000ffffe77cbe10:  0xffffffffffffff80  0x0000ffff80951580 
        0x0000ffffe77cbe20:  0x0000004000035220  0x0000ffff3936eb80 
        0x0000ffffe77cbe30:  0xfffffffffffff800  0x0000ffffe77cbe68 
        0x0000ffffe77cbe40:  0x0000ffffe77cbe90  0x0000ffff80496388 
        0x0000ffffe77cbe50:  0x0000ffffe77cbe90  0x0000ffff80496710 
        0x0000ffffe77cbe60:  0x0000000000000002  0x0000aaaac2860130 
        0x0000ffffe77cbe70:  0x0000ffffe77cbe90  0x0000ffff80496710 
        0x0000ffffe77cbe80:  0x0000000000000002  0x0000ffff3936eb80 
        0x0000ffffe77cbe90:  0x0000ffffe77cbef0  0x0000ffff80496910 
        0x0000ffffe77cbea0:  0x0000000000000002  0x0000aaaac2860108 
        0x0000ffffe77cbeb0:  0x0000000000000023  0x0000ffff809b7000 
        0x0000ffffe77cbec0:  0x0000000000000023  0x0000000000003ff0 
        0x0000ffffe77cbed0:  0x0000000000000027  0x0000ffff8056d000 
        0x0000ffffe77cbee0:  0x0000000000000002  0x0000ffffe77cbf68 
        0x0000ffffe77cbef0: <0x0000ffffe77cbf50  0x0000ffff80496910 
        0x0000ffffe77cbf00:  0x0000000000000002  0x0000aaaac28600e0 
        0x0000ffffe77cbf10:  0x000000000000001f  0x0000ffff809b7000 
        0x0000ffffe77cbf20:  0x000000000000001f  0x0000000000001ff0 
        0x0000ffffe77cbf30:  0x0000000000000023  0x0000ffff8056d000 
        0x0000ffffe77cbf40:  0x0000000000000002  0x0000ffffe77cbfb8 
        0x0000ffffe77cbf50:  0x0000ffffe77cbfb0  0x0000ffff80496910 
        0x0000ffffe77cbf60:  0x0000000000000002  0x0000aaaac28600b8 
        0x0000ffffe77cbf70:  0x000000000000001b  0x0000ffff809b7000 
        0x0000ffffe77cbf80:  0x000000000000001b  0x0000000000000ff0 
        0x0000ffffe77cbf90:  0x000000000000001f  0x0000ffff8056d000 
        0x0000ffffe77cbfa0:  0x0000000000000002  0x0000ffffe77cbfd8 
        0x0000ffffe77cbfb0:  0x0000ffffe77cc010  0x0000ffff80496910 
        0x0000ffffe77cbfc0:  0x0000000000000002  0x0000aaaac2860090 
        0x0000ffffe77cbfd0:  0x0000000000000017  0x0000ffff809b7000 
        0x0000ffffe77cbfe0:  0x0000000000000017  0x00000000000007f0 
        
        goroutine 1 gp=0x40000021c0 m=0 mp=0xffff80951580 [running, locked to thread]:
        runtime.systemstack_switch()
        	runtime/asm_arm64.s:200 +0x8 fp=0x400018f980 sp=0x400018f970 pc=0xffff8013c228
        runtime.setenv_c({0x4000166190?, 0x1?}, {0x4000166195, 0x3c})
        	runtime/env_posix.go:54 +0xb0 fp=0x400018f9e0 sp=0x400018f980 pc=0xffff800d6d10
        syscall.runtimeSetenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
        	runtime/runtime.go:201 +0x2c fp=0x400018fa10 sp=0x400018f9e0 pc=0xffff80139c4c
        syscall.Setenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
        	syscall/env_unix.go:122 +0x310 fp=0x400018fac0 sp=0x400018fa10 pc=0xffff80185870
        os.Setenv({0x4000166190?, 0x3c?}, {0x4000166195?, 0xffffffffffffffff?})
        	os/env.go:120 +0x24 fp=0x400018fb00 sp=0x400018fac0 pc=0xffff801b3ea4
        github.com/opencontainers/runc/libcontainer.populateProcessEnvironment({0x400017a200?, 0xffff805b5440?, 0x400004a0f8?})
        	github.com/opencontainers/runc/libcontainer/init_linux.go:138 +0xe0 fp=0x400018fb80 sp=0x400018fb00 pc=0xffff80401190
        github.com/opencontainers/runc/libcontainer.newContainerInit({0x4000028017, 0x5}, 0x400004a0f0, 0x0, 0xffffffffffffffff, 0x4, {0x0, 0x0, 0x0})
        	github.com/opencontainers/runc/libcontainer/init_linux.go:85 +0xb4 fp=0x400018fbe0 sp=0x400018fb80 pc=0xffff80400da4
        github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0x0?)
        	github.com/opencontainers/runc/libcontainer/factory_linux.go:334 +0x3d0 fp=0x400018fdc0 sp=0x400018fbe0 pc=0xffff80400310
        main.init.0()
        	github.com/opencontainers/runc/init.go:36 +0x194 fp=0x400018fe10 sp=0x400018fdc0 pc=0xffff8047c7b4
        runtime.doInit1(0xffff809399e0)
        	runtime/proc.go:7176 +0xd4 fp=0x400018ff40 sp=0x400018fe10 pc=0xffff80119a44
        runtime.doInit(...)
        	runtime/proc.go:7143
        runtime.main()
        	runtime/proc.go:253 +0x344 fp=0x400018ffd0 sp=0x400018ff40 pc=0xffff80109bf4
        runtime.goexit({})
        	runtime/asm_arm64.s:1222 +0x4 fp=0x400018ffd0 sp=0x400018ffd0 pc=0xffff8013e754
        
        goroutine 2 gp=0x4000002c40 m=nil [force gc (idle)]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	runtime/proc.go:402 +0xc8 fp=0x4000046f90 sp=0x4000046f70 pc=0xffff80109f68
        runtime.goparkunlock(...)
        	runtime/proc.go:408
        runtime.forcegchelper()
        	runtime/proc.go:326 +0xb8 fp=0x4000046fd0 sp=0x4000046f90 pc=0xffff80109df8
        runtime.goexit({})
        	runtime/asm_arm64.s:1222 +0x4 fp=0x4000046fd0 sp=0x4000046fd0 pc=0xffff8013e754
        created by runtime.init.6 in goroutine 1
        	runtime/proc.go:314 +0x24
        
        goroutine 3 gp=0x4000003180 m=nil [GC sweep wait]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	runtime/proc.go:402 +0xc8 fp=0x4000047760 sp=0x4000047740 pc=0xffff80109f68
        runtime.goparkunlock(...)
        	runtime/proc.go:408
        runtime.bgsweep(0x400006e000)
        	runtime/mgcsweep.go:278 +0xa0 fp=0x40000477b0 sp=0x4000047760 pc=0xffff800f54a0
        runtime.gcenable.gowrap1()
        	runtime/mgc.go:203 +0x28 fp=0x40000477d0 sp=0x40000477b0 pc=0xffff800e96b8
        runtime.goexit({})
        	runtime/asm_arm64.s:1222 +0x4 fp=0x40000477d0 sp=0x40000477d0 pc=0xffff8013e754
        created by runtime.gcenable in goroutine 1
        	runtime/mgc.go:203 +0x6c
        
        goroutine 4 gp=0x4000003340 m=nil [GC scavenge wait]:
        runtime.gopark(0x400006e000?, 0xffff80555fe8?, 0x1?, 0x0?, 0x4000003340?)
        	runtime/proc.go:402 +0xc8 fp=0x4000047f60 sp=0x4000047f40 pc=0xffff80109f68
        runtime.goparkunlock(...)
        	runtime/proc.go:408
        runtime.(*scavengerState).park(0xffff80950a20)
        	runtime/mgcscavenge.go:425 +0x5c fp=0x4000047f90 sp=0x4000047f60 pc=0xffff800f2e9c
        runtime.bgscavenge(0x400006e000)
        	runtime/mgcscavenge.go:653 +0x44 fp=0x4000047fb0 sp=0x4000047f90 pc=0xffff800f33e4
        runtime.gcenable.gowrap2()
        	runtime/mgc.go:204 +0x28 fp=0x4000047fd0 sp=0x4000047fb0 pc=0xffff800e9658
        runtime.goexit({})
        	runtime/asm_arm64.s:1222 +0x4 fp=0x4000047fd0 sp=0x4000047fd0 pc=0xffff8013e754
        created by runtime.gcenable in goroutine 1
        	runtime/mgc.go:204 +0xac
        
        goroutine 5 gp=0x4000003c00 m=nil [finalizer wait]:
        runtime.gopark(0x0?, 0x0?, 0xb8?, 0x65?, 0xffff80139b64?)
        	runtime/proc.go:402 +0xc8 fp=0x4000046580 sp=0x4000046560 pc=0xffff80109f68
        runtime.runfinq()
        	runtime/mfinal.go:194 +0x108 fp=0x40000467d0 sp=0x4000046580 pc=0xffff800e8788
        runtime.goexit({})
        	runtime/asm_arm64.s:1222 +0x4 fp=0x40000467d0 sp=0x40000467d0 pc=0xffff8013e754
        created by runtime.createfing in goroutine 1
        	runtime/mfinal.go:164 +0x80
        
        r0      0x0
        r1      0x50
        r2      0x0
        r3      0x0
        r4      0x2e
        r5      0xffff394e0010
        r6      0xaaaac2860130
        r7      0x7fec
        r8      0x0
        r9      0xffff394e7ffc
        r10     0x2f7273752f3a6e69
        r11     0x73752f3a6e696273
        r12     0x732f3a6e69622f72
        r13     0xffffe77cc220
        r14     0x2f
        r15     0x30
        r16     0x80
        r17     0x400018f920
        r18     0x0
        r19     0x2
        r20     0xaaaac2860108
        r21     0x23
        r22     0xffff809b7000
        r23     0x23
        r24     0x3ff0
        r25     0x27
        r26     0xffff8056dd20
        r27     0x2
        r28     0xffff80950d00
        r29     0xffffe77cbef0
        lr      0xffff80496910
        sp      0xffffe77cbef0
        pc      0xffff804967e4
        fault   0x0

What did you expect to see?

No crash.

Perhaps the issue discussed in #59692 can also happen on arm64?

The text was updated successfully, but these errors were encountered:

gabyhelp · 2024-10-09T19:27:18Z

Related Issues and Documentation

runtime: misc/cgo/test.TestSetgid panic with `unexpected return pc for runtime.sigpanic` on linux-arm64-packet since 2022-06-04 #53374 (closed)
runtime: segmentation violation runtime.(*unwinder).next #66151 (closed)
runtime: linux/arm64 crash in runtime.sigtrampgo #32912 (closed)
fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x2 addr=0xc002000000 pc=0x46e698] Linux version 5.10.135-yocto-standard #58377 (closed)
cmd/go: get in docker buildx gets SIGSEGV error #40969 (closed)
runtime: fatal error: unexpected signal during runtime execution #60783 (closed)
runtime: frequent TestSegv failures on linux-mips64le-mengzhuo #50605 (closed)
runtime: `TestSegv/Segv` failure with `unexpected return pc` on `openbsd-386-68` #52043
runtime #60196 (closed)
runtime/cgo: SIGSEGV: segmentation violation #6051 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

cherrymui · 2024-10-09T20:17:16Z

What is your host machine? Are you running docker as a VM?

It looks like this is crashing in the C setenv function. What are your libc and C toolchain? Thanks.

mateusz834 · 2024-10-10T06:46:19Z

See #63567

mpx · 2024-10-17T08:58:55Z

As mentioned elsewhere (#63567), libc setenv(3) is not threadsafe - it's a fundamental flaw. Quite a few different libc calls can access the environment in parallel to setenv(3) and cause a segfault (eg, getaddrinfo(3)).

Some options to avoid the libc segfault:

Avoid libc by disabling CGO (CGO_ENABLE=0 during compilation)
Ensure your program never modifies the environment once there are multiple goroutines (eg, limit changes to func init).

gopherbot · 2024-11-09T20:17:52Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Oct 9, 2024

cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 9, 2024

mknyszek added this to Go Compiler / Runtime Oct 23, 2024

gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 9, 2024

github-project-automation bot moved this to Done in Go Compiler / Runtime Nov 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: crash on linux/arm64 during os.Setenv #69829

runtime: crash on linux/arm64 during os.Setenv #69829

cbandy commented Oct 9, 2024

gabyhelp commented Oct 9, 2024

cherrymui commented Oct 9, 2024

mateusz834 commented Oct 10, 2024

mpx commented Oct 17, 2024

gopherbot commented Nov 9, 2024

runtime: crash on linux/arm64 during os.Setenv #69829

runtime: crash on linux/arm64 during os.Setenv #69829

Comments

cbandy commented Oct 9, 2024

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

gabyhelp commented Oct 9, 2024

cherrymui commented Oct 9, 2024

mateusz834 commented Oct 10, 2024

mpx commented Oct 17, 2024

gopherbot commented Nov 9, 2024

Output of `go env` in your module/workspace: