Skip to content

[Release 10.0] Support hardware with more than 1024 CPUs - #131740

Open
janvorli wants to merge 5 commits into
dotnet:release/10.0from
janvorli:backport/support-more-than-1024-cpus-10.0
Open

[Release 10.0] Support hardware with more than 1024 CPUs#131740
janvorli wants to merge 5 commits into
dotnet:release/10.0from
janvorli:backport/support-more-than-1024-cpus-10.0

Conversation

@janvorli

@janvorli janvorli commented Aug 3, 2026

Copy link
Copy Markdown
Member

Backport of #126763 and #127572 to release/10.0

Customer Impact

.NET runtime fails to initialize on Linux on machines that have more than 1024 CPUs due to sched_getaffinity being passed the default instance of cpu_set_t that supports max 1024 CPUs and fails if there are more CPUs on the current machine.
This occurs also in case of containers limited to a small number of CPUs running on a host with more than 1024 CPUs. So it is impossible to use .NET on such machines.

Regression

  • Yes
  • No

Testing

Directed test on Azure VM with 1792 CPU cores, CI testing coreclr and libraries tests.

Risk

Low. The change has been in main since the beginning of April and no issues were discovered.

Backport of dotnet#126763 to release/10.0

Customer Impact

- [x] Customer reported
- [ ] Found internally

A customer has reported that .NET runtime fails to initialize on machines
that have more than 1024 CPUs due to sched_getaffinity being passed
the default instance of cpu_set_t that supports max 1024 CPUs and
fails if there are more CPUs on the current machine.

This happens even when .NET is running in a container limited to a small
number of CPUs on such machine.

This change fixes sched_getaffinity calls to use a dynamically allocated
CPU set data structure so that it can support any number of CPUs.

Regression
- [ ] Yes
- [x] No

Testing
CI tests, local manual debugging,

Risk
…128069)

On unix, during initialization, the runtime obtains the total number of
CPUs via `sysconf(_SC_NPROCESSORS_CONF)`. This should return the current
number of cpus that are currently present on the system. It turns out
linux has cpu hotplug support, so this number can increase. When hotplug
is enabled, the kernel reserves storage for the max possible number of
CPUs. This max number is exported in `/sys/devices/system/cpu/possible`.
The problem is that, when allocating the `cpu_set_t*` for use with
`sched_getaffinity`, this api failed because the OS expected for the cpu
set to have reserved space for the maximum amount of cpu's, not just for
the ones that are currently present.
@janvorli janvorli added this to the 10.0.x milestone Aug 3, 2026
@janvorli
janvorli requested a review from jkotas August 3, 2026 13:38
@janvorli janvorli self-assigned this Aug 3, 2026
Copilot AI review requested due to automatic review settings August 3, 2026 13:38
@janvorli janvorli added Servicing-consider Issue for next servicing release review area-GC-coreclr labels Aug 3, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @anicka-net, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR backports the >1024 CPU support work to release/10.0 by switching Linux affinity handling from fixed-size cpu_set_t (CPU_SETSIZE/1024) to dynamically sized CPU sets, and updating GC affinity/heap mapping code to avoid conflating CPU limits with heap limits.

Changes:

  • Add a minipal_get_cpu_max_possible_count() helper to size CPU_ALLOC buffers correctly (including Linux hotplug “possible” CPUs).
  • Update PAL / NativeAOT / GC Unix code paths to use CPU_ALLOC + sched_getaffinity/sched_setaffinity with dynamically sized masks.
  • Rename GC’s “CPU” limit constant to MAX_SUPPORTED_HEAPS and allocate CPU-indexed GC maps based on the actual maximum processor count.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/native/minipal/cpucount.h Declares helper API to query max possible CPU count for sizing affinity masks.
src/native/minipal/cpucount.c Implements Linux sysfs-based parsing with sysconf fallback.
src/native/minipal/CMakeLists.txt Builds cpucount.c on Unix hosts.
src/coreclr/pal/src/thread/thread.cpp Uses dynamically allocated CPU sets when resetting inherited thread affinity.
src/coreclr/pal/src/misc/sysinfo.cpp Uses dynamically allocated CPU sets to compute logical CPU count from affinity.
src/coreclr/nativeaot/Runtime/unix/PalUnix.cpp Uses dynamically allocated CPU sets for process CPU count initialization.
src/coreclr/gc/windows/gcenv.windows.cpp Initializes g_processAffinitySet dynamically and updates loops to use total processor count.
src/coreclr/gc/unix/gcenv.unix.cpp Initializes affinity tracking with dynamic CPU counts and uses dynamic CPU sets for affinity operations.
src/coreclr/gc/gcpriv.h Updates GC internal APIs to use MAX_SUPPORTED_HEAPS rather than CPU-based sizing.
src/coreclr/gc/gcconfig.cpp Validates configured heap-affinitize ranges against max processor count.
src/coreclr/gc/gc.cpp Dynamically sizes CPU-indexed GC maps and switches many arrays to heap-based sizing.
src/coreclr/gc/env/gcenv.os.h Makes AffinitySet dynamically sized and adds GetMaxProcessorCount API surface.
Suppressed comments (2)

src/coreclr/gc/env/gcenv.os.h:217

  • AffinitySet::Add relies on assert-only bounds checking. In release builds, a bad cpuIndex would become an OOB write and could corrupt memory. Add a defensive bounds check (keep the assert for debug).
    void Add(size_t cpuIndex)
    {
        assert(GetBitsetEntryIndex(cpuIndex) < m_bitsetDataSize);
        m_bitset[GetBitsetEntryIndex(cpuIndex)] |= GetBitsetEntryMask(cpuIndex);
    }

src/coreclr/gc/env/gcenv.os.h:224

  • AffinitySet::Remove relies on assert-only bounds checking. In release builds, a bad cpuIndex would become an OOB write and could corrupt memory. Add a defensive bounds check (keep the assert for debug).
    void Remove(size_t cpuIndex)
    {
        assert(GetBitsetEntryIndex(cpuIndex) < m_bitsetDataSize);
        m_bitset[GetBitsetEntryIndex(cpuIndex)] &= ~GetBitsetEntryMask(cpuIndex);
    }

Comment thread src/coreclr/gc/unix/gcenv.unix.cpp
Comment thread src/coreclr/gc/unix/gcenv.unix.cpp Outdated
Comment thread src/coreclr/gc/unix/gcenv.unix.cpp
Comment thread src/coreclr/gc/unix/gcenv.unix.cpp
Comment thread src/coreclr/gc/env/gcenv.os.h
Comment thread src/coreclr/gc/env/gcenv.os.h
Comment thread src/coreclr/pal/src/thread/thread.cpp
Comment thread src/coreclr/gc/unix/gcenv.unix.cpp Outdated
Copilot AI review requested due to automatic review settings August 3, 2026 14:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

src/coreclr/gc/unix/gcenv.unix.cpp:1121

  • CPU_ALLOC(g_configuredCpuCount) can return null; the current code dereferences pCpuSet unconditionally (CPU_ZERO_S / CPU_SET_S), which would crash under memory pressure or if g_configuredCpuCount is invalid. Add a null check and fail gracefully before using the buffer.
    cpu_set_t* pCpuSet = CPU_ALLOC(g_configuredCpuCount);
    size_t cpuSetSize = CPU_ALLOC_SIZE(g_configuredCpuCount);
    CPU_ZERO_S(cpuSetSize, pCpuSet);
    CPU_SET_S((int)procNo, cpuSetSize, pCpuSet);

src/coreclr/gc/unix/gcenv.unix.cpp:187

  • g_configuredCpuCount is only used within this translation unit (gcenv.unix.cpp). As a non-static global it gets external linkage unnecessarily; make it static like the other file-scope state here to avoid exporting an unintended symbol.
uint32_t g_configuredCpuCount = 0;

src/native/minipal/cpucount.c:48

  • The fallback sysconf(_SC_NPROCESSORS_CONF) return value can be 0 or -1 on failure; returning it directly may lead to CPU_ALLOC(0) call sites. Normalize invalid values to -1 so callers can reliably trigger their fallback paths.
    return (int)sysconf(_SC_NPROCESSORS_CONF);

Comment thread src/coreclr/gc/unix/gcenv.unix.cpp
@JulieLeeMSFT

Copy link
Copy Markdown
Member

Servicing approved. Please check test failures and get code review.

@svick

svick commented Aug 6, 2026

Copy link
Copy Markdown
Member

Hi,

the code complete date for 10.0.12 (the September 2026 release) is Monday 10 August. Make sure to merge this PR on that date at the latest, or it won't make it into that release.

As a reminder, if this is a product change, you also need Tactics approval before merging this PR (test-only or infra-only changes don't require Tactics approval).

Copilot AI review requested due to automatic review settings August 6, 2026 14:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/coreclr/gc/unix/gcenv.unix.cpp:1121

  • GCToOSInterface::SetThreadAffinity dereferences the result of CPU_ALLOC without checking for allocation failure. If CPU_ALLOC returns nullptr, CPU_ZERO_S/CPU_SET_S will crash; other call sites in this PR handle this case explicitly.
    cpu_set_t* pCpuSet = CPU_ALLOC(g_configuredCpuCount);
    size_t cpuSetSize = CPU_ALLOC_SIZE(g_configuredCpuCount);
    CPU_ZERO_S(cpuSetSize, pCpuSet);
    CPU_SET_S((int)procNo, cpuSetSize, pCpuSet);

src/coreclr/gc/unix/gcenv.unix.cpp:187

  • g_configuredCpuCount is a file-local implementation detail but currently has external linkage, which can lead to accidental symbol collisions across the runtime. It should be file-static like the other state in this translation unit.
// The number of CPUs that are configured in the OS.
uint32_t g_configuredCpuCount = 0;

src/native/minipal/cpucount.c:49

  • minipal_get_cpu_max_possible_count can return 0 from sysconf (or a negative value on error). Several call sites treat only -1 as failure and will pass the value directly to CPU_ALLOC/CPU_ALLOC_SIZE, so normalizing non-positive values to -1 here avoids potential zero-sized allocations and keeps failure signaling consistent.
#endif

    return (int)sysconf(_SC_NPROCESSORS_CONF);
}

@janvorli

janvorli commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

Github didn't pick my commit with build break fix and an attempt to close and reopen the PR doesn't work - it doesn't allow me to reopen it complaining with completely bogus error message: "There was a problem saving your comment. Please try again."

@janvorli janvorli reopened this Aug 7, 2026
@janvorli

janvorli commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

I've managed to reopen it using the "gh" tool.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-GC-coreclr Servicing-approved Approved for servicing release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants