x/sys: SchedGetaffinity fails on systems with more than 1024 hardware threads

[dblueman]

Go version

go version go1.25.0 (Red Hat 1.25.0-4.el10) linux/amd64 (from CentOS Stream 10)

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v3'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2000012266=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/golang'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib/golang/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.0 (Red Hat 1.25.0-4.el10)'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

import (
        "fmt"
        "os"

        "golang.org/x/sys/unix"
)

func main() {
        var cpuset unix.CPUSet
        err := unix.SchedGetaffinity(0, &cpuset)
        if err != nil {
                fmt.Printf("SchedGetaffinity: %v\n", err)
                os.Exit(1)
        }
        fmt.Printf("%d\n", cpuset.Count())
}

What did you see happen?

$ go run .
SchedGetaffinity: invalid argument
exit status 1
$ strace -f ./main
sched_getaffinity(0, 128, 0xc001386ea0) = -1 EINVAL (Invalid argument)

Here, we see a Golang called sched_getaffinity with a bitmask large enough for only 1024 threads; 8192 is the universally accepted limit and what libnuma and related uses.

What did you expect to see?

1920

I'm running on a 16-socket system with 1920 hardware threads.

[gabyhelp]

Related Issues

• x/sys: SchedGetaffinity fails on Linux machines with enough (potential) CPUs #23639 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dblueman]

With that said, though the CentOS Stream 10 kernel has MAXSMP set (8192 hardware threads), libnuma supports 4096 (512*8), which seems reasonable today:

# grep MAXSMP /boot/config-6.12.0-66.el10.x86_64 
CONFIG_MAXSMP=y
# strace -e sched_getaffinity numactl --show >/dev/null
sched_getaffinity(0, 512, [0 ... 1919]) = 240

[JunyangShao]

@golang/runtime

[mknyszek]

The generator underlying x/sys/unix just gets the size from the C library definition of __CPU_SETSIZE in bits/cpu-set.h (AFAICT). It seems to be 1024 everywhere I look. Does this vary on systems that actually have more? If so, this should be as simple as just running the generator on a system with that many hardware threads, though I don't know that any of us have that. Would you be willing to send a patch?

[mknyszek]

On second thought, this might be worth manual intervention if that's the case. Next time someone runs the generator it'll bring right back down. 😅

[prattmic]

It looks like glibc hardcodes the size to 1024, but libnuma probes for the real size (https://github.com/numactl/numactl/blob/b34da01b0017946a1a77057c247cfc6d231a0215/libnuma.c#L592) and provides its own helper to allocate a buffer that is actually large enough: https://github.com/numactl/numactl/blob/b34da01b0017946a1a77057c247cfc6d231a0215/numa.3#L924-L932.

The problem is that x/sys/unix effectively mirrors glibc's interface that assumes a fixed size, while the underlying system call is actually variable size.

[mknyszek]

That's unfortunate. We could just change the definition of unix.CPUSet to be a slice and probe for the size, maybe? That would be a breaking change though, unfortunately, because we didn't make the underlying structure opaque.

Or we could add a new type and new functions. Or we could just ignore __CPU_SETSIZE and have our own, larger, hard-coded size?

[ianlancetaylor]

glibc provides a set of macros for dealing with CPU sets, as documented at https://man7.org/linux/man-pages/man3/CPU_SET.3.html. Code can use CPU_ALLOC to create a CPU set of whatever size. According to the sched_getaffinity man page code is expected to determine the number of CPUs by looking at /proc/cpuinfo or by using sysconf.

I think in the long run we should provide new functions and types that handle this correctly for Go programs, and deprecate the current SchedGetaffinity and SchedSetaffinity calls.

CC @tklauser

[dblueman]

If the concern is possible ABI breakage of increasing cpuSetSize to 4096 eg by const cpuSetSize = max(_CPU_SETSIZE, 4096) / _NCPUBITS, would it make sense to define (eg) SchedGetaffinityX and SchedSetaffinityX to use:

type CPUSetX interface {
    Len() int
}

thus any bitmask implementation could be used?

As an aside, hardware thread count is asymptotic over time, libnuma's limit of 4096 appears to appropriate for the forseeable future, largely due to RASM and efficiency concerns.