Skip to content

runtime: container-aware GOMAXPROCS silently disabled when an nsfs mountinfo line precedes the cgroup mount #81864

Closed

Description

@cee-dub

Go version

go1.27.1 linux/amd64

Output of go env in your module/workspace:

Output ``` shell AR='ar' CC='gcc' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_ENABLED='1' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' CXX='g++' GCCGO='gccgo' GO111MODULE='' GOARCH='arm64' GOARM64='v8.0' GOAUTH='netrc' GOBIN='/go/bin' GOCACHE='/root/.cache/go-build' GOCACHEPROG='' GODEBUG='' GOENV='/root/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFIPS140='off' GOFLAGS='' GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2673028497=/tmp/go-build -gno-record-gcc-switches' GOHOSTARCH='arm64' GOHOSTOS='linux' GOINSECURE='' GOMOD='/dev/null' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPACKAGESDRIVER='' GOPATH='/go' GOPRIVATE='' GOPROXY='https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTELEMETRY='local' GOTELEMETRYDIR='/root/.config/go/telemetry' GOTMPDIR='' GOTOOLCHAIN='local' GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64' GOVCS='' GOVERSION='go1.27.1' GOWORK='' PKG_CONFIG='pkg-config' ```

What did you do?

This is a minimal reproduction derived from a production issue.

This script sets up a Go progam that prints runtime.GOMAXPROCS(0) in a Linux container with a 2-CPU quota. The module has go 1.25, so the container-aware default is on. No GODEBUG or GOMAXPROCS is set.

When a named network namespace (e.g. ip netns add foo) is present and /sys/fs/cgroup is remounted, the nsfs line in /proc/self/mountinfo comes before the cgroup2 line. The CPU quota does not change.

#!/usr/bin/env bash
# Public, self-contained repro for the Go cgroup-mount-ordering GOMAXPROCS bug.
# Requires only a working Docker and public images:
# golang:1.26 (build) and debian:stable-slim (run).
set -euo pipefail

workdir="$(mktemp -d)"
cat > "$workdir/main.go" <<'EOF'
package main

import (
	"fmt"
	"runtime"
)

func main() {
	fmt.Printf("NumCPU=%d\n", runtime.NumCPU())
	fmt.Printf("GOMAXPROCS=%d\n", runtime.GOMAXPROCS(0))
}
EOF
cat > "$workdir/go.mod" <<'EOF'
module gomaxprocsrepro

go 1.25
EOF

echo "--- building static linux/arm64 binary with golang:1.27.1 ---"
echo docker run --rm -v "$workdir":/src -w /src -e CGO_ENABLED=0 golang:1.27.1 \
  go build -o gomaxprocs main.go
docker run --rm -v "$workdir":/src -w /src -e CGO_ENABLED=0 golang:1.27.1 \
  go build -o gomaxprocs main.go

echo "--- base container: --cpus=2 gives GOMAXPROCS=2 when NumCPU is greater ---"
docker run --rm --privileged --cpus=2 \
  -v "$workdir/gomaxprocs":/gomaxprocs:ro debian:stable-slim bash -c '
set -e
echo -- /proc/self/mountinfo --
grep -E "cgroup|nsfs" /proc/self/mountinfo
echo -- /sys/fs/cgroup/cpu.max --
cat /sys/fs/cgroup/cpu.max
echo -- /gomaxprocs output --
/gomaxprocs
'

echo
echo "--- reproduction: nsfs mount created then cgroup2 is remounted ---"
docker run --rm --privileged --cpus=2 \
  -v "$workdir/gomaxprocs":/gomaxprocs:ro debian:stable-slim bash -c '
set -e
apt-get update -qq >/dev/null 2>&1
apt-get install -y -qq iproute2 >/dev/null 2>&1

mkdir -p /run/netns
mount -t tmpfs tmpfs /run/netns
ip netns add foo

echo "[mountinfo baseline: cgroup2 line comes first]"
echo -- /proc/self/mountinfo --
grep -E "cgroup|nsfs" /proc/self/mountinfo
echo -- /sys/fs/cgroup/cpu.max --
cat /sys/fs/cgroup/cpu.max
echo -- /gomaxprocs output --
/gomaxprocs

# Remount cgroup2 so its mountinfo line now comes AFTER the nsfs line
# (this simulates any sequence of events on a long-lived host where a
# netns is created before the cgroup controller is (re)mounted/remounted,
# e.g. container runtime restart, cgroup migration, etc).
umount /sys/fs/cgroup
mount -t cgroup2 -o nosuid,nodev,noexec,relatime cgroup /sys/fs/cgroup

echo "[mountinfo after remount: nsfs line now comes first]"
echo -- /proc/self/mountinfo --
grep -E "cgroup|nsfs" /proc/self/mountinfo
echo -- /sys/fs/cgroup/cpu.max --
cat /sys/fs/cgroup/cpu.max
echo -- /gomaxprocs output --
/gomaxprocs
'

rm -rf "$workdir"

What did you see happen?

--- reproduction: nsfs mount created then cgroup2 is remounted ---
[mountinfo baseline: cgroup2 line comes first]
-- /proc/self/mountinfo --
571 570 0:32 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,nsdelegate
344 343 0:5 net:[4026545549] /run/netns/foo rw shared:86 - nsfs nsfs rw
-- /sys/fs/cgroup/cpu.max --
200000 100000
-- /gomaxprocs output --
NumCPU=12
GOMAXPROCS=2
[mountinfo after remount: nsfs line now comes first]
-- /proc/self/mountinfo --
344 343 0:5 net:[4026545549] /run/netns/foo rw shared:86 - nsfs nsfs rw
345 570 0:32 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,nsdelegate
-- /sys/fs/cgroup/cpu.max --
200000 100000
-- /gomaxprocs output --
NumCPU=12
GOMAXPROCS=12

The quota is still 2 CPUs, but GOMAXPROCS falls back to NumCPU. Nothing reports the fallback: there is no error and no /godebug/non-default-behavior/containermaxprocs:events count. The sysmon update never picks up the limit for the life of the process, because cgroupOK remains false.

I first saw this with a systemd service on a host with named network namespaces under /run/netns and hybrid cgroup v1/v2. In the service's mountinfo, the nsfs lines came before the v1 cpu,cpuacct mount. GOMAXPROCS relying on the Go 1.25 behavior was unexpectedly the full CPU count, above the unit's CPUQuota= rather than equal to it.

What did you expect to see?

GOMAXPROCS=2. An unrelated mount should be skipped no matter where it appears in mountinfo, the same way it is already skipped when it comes after the cgroup line.

Cause

parseCPUMount in src/internal/runtime/cgroup/cgroup.go#L303-306 checks that the root field (4) starts with "/", and returns errMalformedFile when it doesn't. It does this for every line, before checking the filesystem type:

root := line[:i]
if len(root) == 0 || root[0] != '/' {
	// We rely on this in hasPathPrefix.
	return 0, errMalformedFile
}

The filesystem-type switch version runs later, so it can't skip the nsfs line. The whole scan fails, cgroup.OpenCPU returns an error, and defaultGOMAXPROCSInit (src/runtime/cgroup_linux.go) returns without setting cgroupOK.

Activity

  1. changed the title [-]import/path: issue title[/-] [+]runtime: container-aware GOMAXPROCS silently disabled when an nsfs mountinfo line precedes the cgroup mount[/+] on Sep 29, 2026
  2. gopherbot commented on Sep 29, 2026

    @gopherbot
    Contributor

    Change https://go.dev/cl/841405 mentions this issue: internal/runtime/cgroup: only validate root of cgroup mounts

  3. added
    FixPendingIssues that have a fix which has not yet been reviewed or submitted.
    on Sep 29, 2026
  4. added this to the Go1.28 milestone on Oct 6, 2026
  5. added a commit that references this issue on Oct 7, 2026
    f91a6fa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

FixPendingIssues that have a fix which has not yet been reviewed or submitted.compiler/runtimeIssues related to the Go compiler and/or runtime.

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions