cmd/link: regression since 1.22 arm7 binary fails to start on 4.19.163 kernel with `could not verify signature of ./main`

[Jorropo]

Go version

go version go1.24.0 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='0'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm'
GOARM='7'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -marm -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build161984771=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Source:

package main

import "fmt"

func main() {
    fmt.Println("Hallo")
}

Build command:

GOARCH=arm GOARM=7 GOOS=linux go build main.go

Built in Docker official image golang1.24:

Linux linux 6.8.0-55-generic #57-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 23:42:21 UTC 2025 x86_64 GNU/Linux

What did you see happen?

Running in NetModule linux image:

Linux 0000-002 4.19.163 #1 SMP Fri Nov 10 10:54:34 UTC 2023 armv7l GNU/Linux

Running on Hardware:

NB2810 - https://www.netmodule.com/de/produkte/router/nb2810-2lwac-g

Exec

root@0000-002:~/gotest# ./main
Killed

dmesg

[    0.000000] could not verify signature of ./main

What did you expect to see?

root@0000-002:~/gotest# ./main
Hallo

[Jorropo]

Reported by icsharp. in discord gophers: https://discord.com/channels/118456055842734083/1344643961246978118/1344643961246978118

[Jorropo]

Works if the target hardware is running kernel 5.15.147 #1 SMP Fri Jul 19 14:30:48 UTC 2024 armv7l armv7l armv7l GNU/Linux

[Jorropo]

Bisected to 3251006 cc @thanm

[Jorropo]

Surprisingly this works if built with CGO:

GOOS=linux GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go build  -ldflags "-linkmode=external -extldflags=-static" main.go

[gabyhelp]

Related Issues

• Go 1.17+ crashes on Docker Desktop (M1) when running with "--platform=linux/amd64" platform #48439 (closed)
• go does not run at all #62639 (closed)
• cmd/link: regression on building an app with `GOOS=wasip1 GOARCH=wasm` #65786 (closed)
• runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64 #69255
• cmd/link: panic: operation not permitted #41356 (closed)
• bug #68989 (closed)
• Plis fixit! Its not good!!! #35957 (closed)
• x/build/cmd/gorebuild: expects go1.21.1.linux-armv6l.tar.gz and v0.0.1-go1.21.1.linux-arm.zip to be rebuildable without explicitly setting GOARM=6 #62514 (closed)
• vendor/golang.org/x/arch/arm64/arm64asm: Go 1.10.beta1: test failures #23237 (closed)
• go version go1.21.5 linux/amd64 #67008 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[thanm]

4.19.163 is an ancient kernel, I am not sure it really makes sense to try to pursue a fix for something that old.

Also, the change in question is fixing an important bug. I am not sure why this particular elderly kernel would object to the new way of doing things, but the old way was definitely broken -- putting the ".shstrtab" section in a loadable segment (as we were doing before the CL) is non-standard and is not the way C compilers generate ELF files.

[Jorropo]

@thanm #67001 recently moved the go1.24 baseline to 3.2 (edit 3.17), what does it mean to announce a minimum linux release if we are then gonna reconsider any bug impacting old but above the minimum kernels through an is it really worth our time to fix this lense ?

Note: I'm being devil's advocate here.

[prattmic]

Do we know if this bug is general to any 4.19 kernel, or some custom change this vendor made to their kernels?

I checked out a copy of the 4.19 kernel source and can't find the string "could not verify signature" at all. (Though admittedly I didn't look very hard).

I'm at a bit of a loss what signature it would be verifying in the first place? Linux binaries aren't signed.

[HESS-BEA]

Hi, im icsharp.

First of all, i can't not say im sure they did no changes to the kernel...

Is there any way i could make more tests or something like that?
Im indirect in contact with the PO of them, i will ask more questions on monday, maybe he can tell me more.

Something specific i could ask?

[prattmic]

I'd ask them what the verification is, and whether they can test Go binaries themselves.

I tested a Debian Buster image from https://cdimage.debian.org/images/cloud/, which also uses a 4.19 kernel, and Go 1.24 binaries ran just fine. Though this was on amd64, not arm.

[ianlancetaylor]

@HESS-BEA To the best of my knowledge that error message must be coming from a non-standard authentication module of some sort, probably via Linux-PAM. Look at /etc/pam.conf and /etc/pam.d. It's very unlikely that this is a part of the Linux kernel proper. I would guess that this is some sort of authentication installed by whoever built the router software.

If you can find the PAM module that is requiring the signature, then perhaps you can find out what will satisfy it.

@Jorropo Are you able to recreate the problem, or are you passing on data from Discord? Either way, if it works with -linkmode=external, that means that the C linker is either producing the required signatures or is generating an executable that somehow does not require a signature.