Repository navigation
runtime: container-aware GOMAXPROCS ignores a tighter visible ancestor cgroup limit #81180
Description
Go's container-aware GOMAXPROCS reads the CPU limit from the cgroup that contains the process. If that cgroup is unlimited but a visible parent has a limit, the kernel enforces the parent limit while Go defaults GOMAXPROCS to the CPU affinity count.
I reproduced this with Go 1.25.12, 1.26.5, and 1.27.0 on Linux 6.18.35. The hierarchy is:
parent cpu.max = 200000 100000
└── leaf cpu.max = max/absent
└── process
Both levels are visible in the process's cgroup mount.
For all three releases, using module language go 1.25:
process directly in parent: GOMAXPROCS=2
process in leaf: GOMAXPROCS=5
leaf with
GODEBUG=containermaxprocs=1,updatemaxprocs=1:
GOMAXPROCS=5
The parent is the limit the kernel enforces. Busy workers in the leaf were throttled to roughly two CPUs of aggregate runtime. An explicit GOMAXPROCS=2 also avoids the mismatch.
Automatic updates have the same boundary. When I changed the parent from 2 to 3 CPUs, a process directly in the parent updated from 2 to 3. A process in the unlimited leaf stayed at 5.
A minimal reproducer is at https://gist.github.com/thc1006/9bc9d2306b7a8b8009b8ad673544c445 and uses an unprivileged user and cgroup namespace: it mounts a writable cgroup v2 hierarchy, sets the 2-CPU limit on a parent, leaves the child unlimited, and runs a small probe in each. The raw output is included with it.
The current runtime source documents the leaf-only behavior. The original proposal described the effective limit as the minimum through the hierarchy, but the implementation deliberately cached one leaf FD so updates are a single pread.
I built a proof of concept that opens the CPU limit files from the leaf through the visible cgroup mount root at startup, then uses the minimum visible limit. It produces the expected value and follows a parent change from 2 to 3. It does not cross a cgroup mount or namespace boundary, and it does not address process migration.
The main trade-off is the retained FD set. In ordinary 1 to 32 level test hierarchies I could not measure a startup difference. At 128 visible levels the prototype added about 0.6 ms per process. At 512 levels it added about 8.6 ms. Its fixed FD array adds 32 KiB of BSS. A deeper hierarchy may also need one or two FDs per visible ancestor.
Would the runtime accept visible-ancestor limits with a bounded, stable FD set, or would maintainers prefer reopening the hierarchy on updates, or considering ancestors only at startup? I can send the implementation once the resource and fallback policy is agreed.
This does not help when the restrictive parent is hidden by the process's cgroup namespace. I tested that case separately, and both stock and patched runtimes correctly stay unable to see the hidden limit.
Activity
thc1006 commentedon Aug 28, 2026
This came out of a Kubernetes CI investigation, but the reproducer is standalone and does not depend on Kubernetes.
I have not sent a CL yet. I am looking for agreement on the expected visible-ancestor behavior and resource policy first.
Please feel free to point out any errors. Thank you very much.
gabyhelp commentedon Aug 28, 2026
Related Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
[-]Runtime: container-aware GOMAXPROCS ignores a tighter visible ancestor cgroup limit[/-][+]runtime: container-aware GOMAXPROCS ignores a tighter visible ancestor cgroup limit[/+]dr2chase commentedon Aug 28, 2026
@golang/runtime
prattmic commentedon Sep 15, 2026
This is a deliberate limitation in the original implementation. See #73193 (comment).
The primary rationale comes from the desire to support periodic GOMAXPROCS updates without re-opening files. Thus, we keep the
cpu.maxFD open for the duration of process execution to reread it. If we want to support the entire hierarchy, we'd need to keep at least one FD open per level.That is perhaps doable, but at the time did not seem worthwhile, especially since most container runtimes seem to hide parent cgroups anyways (they mount only the leaf cgroup at
/sys/fs/cgroup. So this wouldn't work for most container runtimes anyways. I'd be curious to hear more about your case that does have visible parents.thc1006 commentedon Sep 15, 2026
Dear @prattmic ,
Thanks, that helps. I agree the leaf-only design is deliberate, and that walking ancestors cannot help once the limiting cgroup is outside the visible mount.
The case I hit is not an ordinary container reading its host parent. It is Kubernetes Prow's bootstrap/kubekins runner subdividing its own container cgroup for Docker-in-Docker. To satisfy the cgroup-v2 no-internal-process rule, it moves the runner into an
initleaf, then starts Docker and the Go job there:Cholds the limit and stays visible to the process ininit, as the root of the container's own cgroup mount. The host ancestors aboveCstay hidden, so this works without reaching past the mount. The move also happens before the Go job starts, so nothing has to detect a running process changing cgroups.This is a real, already-fixed workload, not a live bug report. kubernetes/test-infra#37440 deployed the subdivision, and #37733 worked around the sizing by copying
C/cpu.maxintoinit(bootstrap/runner.shL56-L110). Its author measuredGOMAXPROCSat 16 before the copy and 2 after, on a 16-CPU host with a 2-CPU quota.Two caveats: the standalone reproducer in this issue is a threaded-cgroup isolation test, not a Docker reproduction; and I would not extend this to kind-node processes, whose own cgroup namespace does hide
C.That is the visible-parent shape behind the report. I do not know how common it is, and it does not settle the FD trade-off on its own. Would a shallow, delegated hierarchy like this be worth exploring for bounded visible-ancestor support, or would you rather keep it at the container-manager layer?