Repository navigation
runtime: container-aware GOMAXPROCS silently disabled when an nsfs mountinfo line precedes the cgroup mount #81864
Description
Go version
go1.27.1 linux/amd64
Output of go env in your module/workspace:
Output
``` shell AR='ar' CC='gcc' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_ENABLED='1' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' CXX='g++' GCCGO='gccgo' GO111MODULE='' GOARCH='arm64' GOARM64='v8.0' GOAUTH='netrc' GOBIN='/go/bin' GOCACHE='/root/.cache/go-build' GOCACHEPROG='' GODEBUG='' GOENV='/root/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFIPS140='off' GOFLAGS='' GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2673028497=/tmp/go-build -gno-record-gcc-switches' GOHOSTARCH='arm64' GOHOSTOS='linux' GOINSECURE='' GOMOD='/dev/null' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPACKAGESDRIVER='' GOPATH='/go' GOPRIVATE='' GOPROXY='https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTELEMETRY='local' GOTELEMETRYDIR='/root/.config/go/telemetry' GOTMPDIR='' GOTOOLCHAIN='local' GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64' GOVCS='' GOVERSION='go1.27.1' GOWORK='' PKG_CONFIG='pkg-config' ```What did you do?
This is a minimal reproduction derived from a production issue.
This script sets up a Go progam that prints runtime.GOMAXPROCS(0) in a Linux container with a 2-CPU quota. The module has go 1.25, so the container-aware default is on. No GODEBUG or GOMAXPROCS is set.
When a named network namespace (e.g. ip netns add foo) is present and /sys/fs/cgroup is remounted, the nsfs line in /proc/self/mountinfo comes before the cgroup2 line. The CPU quota does not change.
#!/usr/bin/env bash
# Public, self-contained repro for the Go cgroup-mount-ordering GOMAXPROCS bug.
# Requires only a working Docker and public images:
# golang:1.26 (build) and debian:stable-slim (run).
set -euo pipefail
workdir="$(mktemp -d)"
cat > "$workdir/main.go" <<'EOF'
package main
import (
"fmt"
"runtime"
)
func main() {
fmt.Printf("NumCPU=%d\n", runtime.NumCPU())
fmt.Printf("GOMAXPROCS=%d\n", runtime.GOMAXPROCS(0))
}
EOF
cat > "$workdir/go.mod" <<'EOF'
module gomaxprocsrepro
go 1.25
EOF
echo "--- building static linux/arm64 binary with golang:1.27.1 ---"
echo docker run --rm -v "$workdir":/src -w /src -e CGO_ENABLED=0 golang:1.27.1 \
go build -o gomaxprocs main.go
docker run --rm -v "$workdir":/src -w /src -e CGO_ENABLED=0 golang:1.27.1 \
go build -o gomaxprocs main.go
echo "--- base container: --cpus=2 gives GOMAXPROCS=2 when NumCPU is greater ---"
docker run --rm --privileged --cpus=2 \
-v "$workdir/gomaxprocs":/gomaxprocs:ro debian:stable-slim bash -c '
set -e
echo -- /proc/self/mountinfo --
grep -E "cgroup|nsfs" /proc/self/mountinfo
echo -- /sys/fs/cgroup/cpu.max --
cat /sys/fs/cgroup/cpu.max
echo -- /gomaxprocs output --
/gomaxprocs
'
echo
echo "--- reproduction: nsfs mount created then cgroup2 is remounted ---"
docker run --rm --privileged --cpus=2 \
-v "$workdir/gomaxprocs":/gomaxprocs:ro debian:stable-slim bash -c '
set -e
apt-get update -qq >/dev/null 2>&1
apt-get install -y -qq iproute2 >/dev/null 2>&1
mkdir -p /run/netns
mount -t tmpfs tmpfs /run/netns
ip netns add foo
echo "[mountinfo baseline: cgroup2 line comes first]"
echo -- /proc/self/mountinfo --
grep -E "cgroup|nsfs" /proc/self/mountinfo
echo -- /sys/fs/cgroup/cpu.max --
cat /sys/fs/cgroup/cpu.max
echo -- /gomaxprocs output --
/gomaxprocs
# Remount cgroup2 so its mountinfo line now comes AFTER the nsfs line
# (this simulates any sequence of events on a long-lived host where a
# netns is created before the cgroup controller is (re)mounted/remounted,
# e.g. container runtime restart, cgroup migration, etc).
umount /sys/fs/cgroup
mount -t cgroup2 -o nosuid,nodev,noexec,relatime cgroup /sys/fs/cgroup
echo "[mountinfo after remount: nsfs line now comes first]"
echo -- /proc/self/mountinfo --
grep -E "cgroup|nsfs" /proc/self/mountinfo
echo -- /sys/fs/cgroup/cpu.max --
cat /sys/fs/cgroup/cpu.max
echo -- /gomaxprocs output --
/gomaxprocs
'
rm -rf "$workdir"What did you see happen?
--- reproduction: nsfs mount created then cgroup2 is remounted ---
[mountinfo baseline: cgroup2 line comes first]
-- /proc/self/mountinfo --
571 570 0:32 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,nsdelegate
344 343 0:5 net:[4026545549] /run/netns/foo rw shared:86 - nsfs nsfs rw
-- /sys/fs/cgroup/cpu.max --
200000 100000
-- /gomaxprocs output --
NumCPU=12
GOMAXPROCS=2
[mountinfo after remount: nsfs line now comes first]
-- /proc/self/mountinfo --
344 343 0:5 net:[4026545549] /run/netns/foo rw shared:86 - nsfs nsfs rw
345 570 0:32 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,nsdelegate
-- /sys/fs/cgroup/cpu.max --
200000 100000
-- /gomaxprocs output --
NumCPU=12
GOMAXPROCS=12The quota is still 2 CPUs, but GOMAXPROCS falls back to NumCPU. Nothing reports the fallback: there is no error and no /godebug/non-default-behavior/containermaxprocs:events count. The sysmon update never picks up the limit for the life of the process, because cgroupOK remains false.
I first saw this with a systemd service on a host with named network namespaces under /run/netns and hybrid cgroup v1/v2. In the service's mountinfo, the nsfs lines came before the v1 cpu,cpuacct mount. GOMAXPROCS relying on the Go 1.25 behavior was unexpectedly the full CPU count, above the unit's CPUQuota= rather than equal to it.
What did you expect to see?
GOMAXPROCS=2. An unrelated mount should be skipped no matter where it appears in mountinfo, the same way it is already skipped when it comes after the cgroup line.
Cause
parseCPUMount in src/internal/runtime/cgroup/cgroup.go#L303-306 checks that the root field (4) starts with "/", and returns errMalformedFile when it doesn't. It does this for every line, before checking the filesystem type:
root := line[:i]
if len(root) == 0 || root[0] != '/' {
// We rely on this in hasPathPrefix.
return 0, errMalformedFile
}The filesystem-type switch version runs later, so it can't skip the nsfs line. The whole scan fails, cgroup.OpenCPU returns an error, and defaultGOMAXPROCSInit (src/runtime/cgroup_linux.go) returns without setting cgroupOK.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
[-]import/path: issue title[/-][+]runtime: container-aware GOMAXPROCS silently disabled when an nsfs mountinfo line precedes the cgroup mount[/+]gopherbot commentedon Sep 29, 2026
Change https://go.dev/cl/841405 mentions this issue:
internal/runtime/cgroup: only validate root of cgroup mountsinternal/runtime/cgroup: only validate root of cgroup mounts