[Release 10.0] Support hardware with more than 1024 CPUs - #131740
[Release 10.0] Support hardware with more than 1024 CPUs#131740janvorli wants to merge 5 commits into
Conversation
Backport of dotnet#126763 to release/10.0 Customer Impact - [x] Customer reported - [ ] Found internally A customer has reported that .NET runtime fails to initialize on machines that have more than 1024 CPUs due to sched_getaffinity being passed the default instance of cpu_set_t that supports max 1024 CPUs and fails if there are more CPUs on the current machine. This happens even when .NET is running in a container limited to a small number of CPUs on such machine. This change fixes sched_getaffinity calls to use a dynamically allocated CPU set data structure so that it can support any number of CPUs. Regression - [ ] Yes - [x] No Testing CI tests, local manual debugging, Risk
…128069) On unix, during initialization, the runtime obtains the total number of CPUs via `sysconf(_SC_NPROCESSORS_CONF)`. This should return the current number of cpus that are currently present on the system. It turns out linux has cpu hotplug support, so this number can increase. When hotplug is enabled, the kernel reserves storage for the max possible number of CPUs. This max number is exported in `/sys/devices/system/cpu/possible`. The problem is that, when allocating the `cpu_set_t*` for use with `sched_getaffinity`, this api failed because the OS expected for the cpu set to have reserved space for the maximum amount of cpu's, not just for the ones that are currently present.
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Tagging subscribers to this area: @anicka-net, @dotnet/gc |
There was a problem hiding this comment.
Pull request overview
This PR backports the >1024 CPU support work to release/10.0 by switching Linux affinity handling from fixed-size cpu_set_t (CPU_SETSIZE/1024) to dynamically sized CPU sets, and updating GC affinity/heap mapping code to avoid conflating CPU limits with heap limits.
Changes:
- Add a
minipal_get_cpu_max_possible_count()helper to sizeCPU_ALLOCbuffers correctly (including Linux hotplug “possible” CPUs). - Update PAL / NativeAOT / GC Unix code paths to use
CPU_ALLOC+sched_getaffinity/sched_setaffinitywith dynamically sized masks. - Rename GC’s “CPU” limit constant to
MAX_SUPPORTED_HEAPSand allocate CPU-indexed GC maps based on the actual maximum processor count.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/native/minipal/cpucount.h | Declares helper API to query max possible CPU count for sizing affinity masks. |
| src/native/minipal/cpucount.c | Implements Linux sysfs-based parsing with sysconf fallback. |
| src/native/minipal/CMakeLists.txt | Builds cpucount.c on Unix hosts. |
| src/coreclr/pal/src/thread/thread.cpp | Uses dynamically allocated CPU sets when resetting inherited thread affinity. |
| src/coreclr/pal/src/misc/sysinfo.cpp | Uses dynamically allocated CPU sets to compute logical CPU count from affinity. |
| src/coreclr/nativeaot/Runtime/unix/PalUnix.cpp | Uses dynamically allocated CPU sets for process CPU count initialization. |
| src/coreclr/gc/windows/gcenv.windows.cpp | Initializes g_processAffinitySet dynamically and updates loops to use total processor count. |
| src/coreclr/gc/unix/gcenv.unix.cpp | Initializes affinity tracking with dynamic CPU counts and uses dynamic CPU sets for affinity operations. |
| src/coreclr/gc/gcpriv.h | Updates GC internal APIs to use MAX_SUPPORTED_HEAPS rather than CPU-based sizing. |
| src/coreclr/gc/gcconfig.cpp | Validates configured heap-affinitize ranges against max processor count. |
| src/coreclr/gc/gc.cpp | Dynamically sizes CPU-indexed GC maps and switches many arrays to heap-based sizing. |
| src/coreclr/gc/env/gcenv.os.h | Makes AffinitySet dynamically sized and adds GetMaxProcessorCount API surface. |
Suppressed comments (2)
src/coreclr/gc/env/gcenv.os.h:217
- AffinitySet::Add relies on assert-only bounds checking. In release builds, a bad cpuIndex would become an OOB write and could corrupt memory. Add a defensive bounds check (keep the assert for debug).
void Add(size_t cpuIndex)
{
assert(GetBitsetEntryIndex(cpuIndex) < m_bitsetDataSize);
m_bitset[GetBitsetEntryIndex(cpuIndex)] |= GetBitsetEntryMask(cpuIndex);
}
src/coreclr/gc/env/gcenv.os.h:224
- AffinitySet::Remove relies on assert-only bounds checking. In release builds, a bad cpuIndex would become an OOB write and could corrupt memory. Add a defensive bounds check (keep the assert for debug).
void Remove(size_t cpuIndex)
{
assert(GetBitsetEntryIndex(cpuIndex) < m_bitsetDataSize);
m_bitset[GetBitsetEntryIndex(cpuIndex)] &= ~GetBitsetEntryMask(cpuIndex);
}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
src/coreclr/gc/unix/gcenv.unix.cpp:1121
CPU_ALLOC(g_configuredCpuCount)can return null; the current code dereferencespCpuSetunconditionally (CPU_ZERO_S / CPU_SET_S), which would crash under memory pressure or ifg_configuredCpuCountis invalid. Add a null check and fail gracefully before using the buffer.
cpu_set_t* pCpuSet = CPU_ALLOC(g_configuredCpuCount);
size_t cpuSetSize = CPU_ALLOC_SIZE(g_configuredCpuCount);
CPU_ZERO_S(cpuSetSize, pCpuSet);
CPU_SET_S((int)procNo, cpuSetSize, pCpuSet);
src/coreclr/gc/unix/gcenv.unix.cpp:187
g_configuredCpuCountis only used within this translation unit (gcenv.unix.cpp). As a non-static global it gets external linkage unnecessarily; make itstaticlike the other file-scope state here to avoid exporting an unintended symbol.
uint32_t g_configuredCpuCount = 0;
src/native/minipal/cpucount.c:48
- The fallback
sysconf(_SC_NPROCESSORS_CONF)return value can be 0 or -1 on failure; returning it directly may lead toCPU_ALLOC(0)call sites. Normalize invalid values to -1 so callers can reliably trigger their fallback paths.
return (int)sysconf(_SC_NPROCESSORS_CONF);
|
Servicing approved. Please check test failures and get code review. |
|
Hi, the code complete date for 10.0.12 (the September 2026 release) is Monday 10 August. Make sure to merge this PR on that date at the latest, or it won't make it into that release. As a reminder, if this is a product change, you also need Tactics approval before merging this PR (test-only or infra-only changes don't require Tactics approval). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/coreclr/gc/unix/gcenv.unix.cpp:1121
- GCToOSInterface::SetThreadAffinity dereferences the result of CPU_ALLOC without checking for allocation failure. If CPU_ALLOC returns nullptr, CPU_ZERO_S/CPU_SET_S will crash; other call sites in this PR handle this case explicitly.
cpu_set_t* pCpuSet = CPU_ALLOC(g_configuredCpuCount);
size_t cpuSetSize = CPU_ALLOC_SIZE(g_configuredCpuCount);
CPU_ZERO_S(cpuSetSize, pCpuSet);
CPU_SET_S((int)procNo, cpuSetSize, pCpuSet);
src/coreclr/gc/unix/gcenv.unix.cpp:187
- g_configuredCpuCount is a file-local implementation detail but currently has external linkage, which can lead to accidental symbol collisions across the runtime. It should be file-static like the other state in this translation unit.
// The number of CPUs that are configured in the OS.
uint32_t g_configuredCpuCount = 0;
src/native/minipal/cpucount.c:49
- minipal_get_cpu_max_possible_count can return 0 from sysconf (or a negative value on error). Several call sites treat only -1 as failure and will pass the value directly to CPU_ALLOC/CPU_ALLOC_SIZE, so normalizing non-positive values to -1 here avoids potential zero-sized allocations and keeps failure signaling consistent.
#endif
return (int)sysconf(_SC_NPROCESSORS_CONF);
}
|
Github didn't pick my commit with build break fix and an attempt to close and reopen the PR doesn't work - it doesn't allow me to reopen it complaining with completely bogus error message: "There was a problem saving your comment. Please try again." |
|
I've managed to reopen it using the "gh" tool. |
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Backport of #126763 and #127572 to release/10.0
Customer Impact
.NET runtime fails to initialize on Linux on machines that have more than 1024 CPUs due to
sched_getaffinitybeing passed the default instance ofcpu_set_tthat supports max 1024 CPUs and fails if there are more CPUs on the current machine.This occurs also in case of containers limited to a small number of CPUs running on a host with more than 1024 CPUs. So it is impossible to use .NET on such machines.
Regression
Testing
Directed test on Azure VM with 1792 CPU cores, CI testing coreclr and libraries tests.
Risk
Low. The change has been in main since the beginning of April and no issues were discovered.