os.process_cpu_count(): get the number CPUs usable by the current process #109649

vstinner · 2023-09-21T14:14:27Z

Feature or enhancement

os.cpu_count() returns the total number of CPUs that the current machine has. On Unix, sched_setaffinity() can reduce the number of CPU "usable" by a process. Currently, os.cpu_count() docstring contains a recipe how to get the number of usable CPUs:

>>> import os; help(os.cpu_count)
cpu_count()
    Return the number of CPUs in the system; return None if indeterminable.
    
    This number is not equivalent to the number of CPUs the current process can
    use.  The number of usable CPUs can be obtained with
    ``len(os.sched_getaffinity(0))``

I propose to add an affinity optional keyword-only parameter to os.cpu_count() which would return len(os.sched_getaffinity(0)).st

UPDATE: Rename usable parameter to affinity.

Linked PRs

The text was updated successfully, but these errors were encountered:

vstinner · 2023-09-21T14:18:14Z

os.cpu_count() issues (open and closed):

issue add os.cpu_count() #62114: add os.cpu_count()
issue cpu_count() unreliable on Windows #67226: cpu_count() unreliable on Windows
issue os.cpu_count is problematic on sparc/solaris #73451: os.cpu_count is problematic on sparc/solaris
issue os.cpu_count() returns wrong number of processors on system with > 64 logical processors #74766: Windows: os.cpu_count() returns wrong number of processors on system with > 64 logical processors
issue os.cpu_count() should return a count assigned to a container or that the process is restricted to #80235: On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)
issue Support python -Xcpu_count=<n> feature for container environment. #109595: Support python -Xcpu_count= feature for container environment

Previously, the function was known as multiprocessing.cpu_count():

issue multiprocessing.cpu_count() should use hw.availcpu on Mac OS X #61646: multiprocessing.cpu_count() should use hw.availcpu on Mac OS X

Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes!

vstinner · 2023-09-21T14:33:30Z

On Windows, maybe it can be implemented with GetProcessGroupAffinity(), GetProcessAffinityMask(), GetActiveProcessorGroupCount() and GetActiveProcessorCount().

GetProcessGroupAffinity():
https://learn.microsoft.com/en-us/windows/win32/api/processtopologyapi/nf-processtopologyapi-getprocessgroupaffinity

vstinner · 2023-09-21T14:35:02Z

On Windows, a CPU affinity can be run with start /affinity command. Example to only run Python on the firstCPU:

start /affinity 0x0000000000000001 python.exe

Implement cpu_count(affinity=True) with sched_getaffinity() on Unix and GetProcessAffinityMask() on Windows. Changes: * Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes! * Doc: Specify that os.cpu_count() counts *logicial* CPUs and mention that Linux cgroups are ignored. * _Py_popcount32() uses UINT32_C() for M1, M2 and M4 constants. * Add _Py_popcount64(). Add tests on _Py_popcount64().

Implement cpu_count(affinity=True) with sched_getaffinity(). Changes: * Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes! * Doc: Specify that os.cpu_count() counts *logicial* CPUs. Mention that Linux cgroups are ignored.

* Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes! * Doc: Specify that os.cpu_count() counts *logicial* CPUs.

* Refactor os_sched_getaffinity_impl(): move variable definitions to their first assignment. * Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes! * Doc: Specify that os.cpu_count() counts *logicial* CPUs. * Doc: Specify that os.sched_getaffinity(0) is related to the calling thread.

Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count() * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.

Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.

* Doc: Specify that os.cpu_count() counts *logicial* CPUs. * Doc: Specify that os.sched_getaffinity(0) is related to the calling thread. * Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes!

Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.

* gh-109649: Enhance os.cpu_count() documentation * Doc: Specify that os.cpu_count() counts *logicial* CPUs. * Doc: Specify that os.sched_getaffinity(0) is related to the calling thread. * Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes! * Restore removed text

…H-110169) * pythongh-109649: Enhance os.cpu_count() documentation * Doc: Specify that os.cpu_count() counts *logicial* CPUs. * Doc: Specify that os.sched_getaffinity(0) is related to the calling thread. * Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes! * Restore removed text (cherry picked from commit 5245b97) Co-authored-by: Victor Stinner <vstinner@python.org>

) (#110226) [3.12] gh-109649: Enhance os.cpu_count() documentation (GH-110169) * gh-109649: Enhance os.cpu_count() documentation * Doc: Specify that os.cpu_count() counts *logicial* CPUs. * Doc: Specify that os.sched_getaffinity(0) is related to the calling thread. * Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes! * Restore removed text (cherry picked from commit 5245b97) Co-authored-by: Victor Stinner <vstinner@python.org>

When CPUs are isolated on Linux, os.process_cpu_count() is smaller than os.cpu_count(). Fix the test for this case. Example with "isolcpus=5,11 rcu_nocbs=5,11" options passed to a Linux command line to isolated two logical CPUs: $ ./python -c 'import os; print(os.process_cpu_count(), "/", os.cpu_count())' 10 / 12

…n#111689) When CPUs are isolated on Linux, os.process_cpu_count() is smaller than os.cpu_count(). Fix the test for this case. Example with "isolcpus=5,11 rcu_nocbs=5,11" options passed to a Linux command line to isolated two logical CPUs: $ ./python -c 'import os; print(os.process_cpu_count(), "/", os.cpu_count())' 10 / 12

vstinner added the type-feature A feature request or enhancement label Sep 21, 2023

vstinner added a commit to vstinner/cpython that referenced this issue Sep 21, 2023

pythongh-109649: Add usable parameter to os.cpu_count()

477d57b

vstinner added a commit to vstinner/cpython that referenced this issue Sep 21, 2023

pythongh-109649: Add usable parameter to os.cpu_count()

0f769ec

vstinner added a commit to vstinner/cpython that referenced this issue Sep 21, 2023

pythongh-109649: Add usable parameter to os.cpu_count()

df887a8

Fix test_posix.test_sched_getaffinity(): restore the old CPU mask when the test completes!

bedevere-app bot mentioned this issue Sep 21, 2023

gh-109649: Add affinity parameter to os.cpu_count() #109652

Closed

vstinner mentioned this issue Sep 21, 2023

Support python -Xcpu_count=<n> feature for container environment. #109595

Closed

vstinner changed the title ~~os.cpu_count(): add "usable" parameter to get the number of "usable" CPUs~~ os.cpu_count(): add "affinity" parameter to get the number of logical CPUs usable by the current process Sep 21, 2023

bedevere-app bot mentioned this issue Sep 26, 2023

gh-109649: Add os.process_cpu_count() function #109907

Merged

vstinner changed the title ~~os.cpu_count(): add "affinity" parameter to get the number of logical CPUs usable by the current process~~ os.process_cpu_count(): get the number CPUs usable by the current process Sep 30, 2023

vstinner closed this as completed Sep 30, 2023

bedevere-app bot mentioned this issue Sep 30, 2023

gh-109649: Use os.process_cpu_count() #110165

Merged

bedevere-app bot mentioned this issue Oct 1, 2023

[3.12] gh-109649: Enhance os.cpu_count() documentation #110169

Merged

miss-islington mentioned this issue Oct 2, 2023

[3.11] [3.12] gh-109649: Enhance os.cpu_count() documentation (GH-110169) #110226

Merged

corona10 mentioned this issue Oct 10, 2023

Add -Xcpu_count=process cmdline mode to redirect os.cpu_count as os.process_cpu_count #110649

Closed

bedevere-app bot mentioned this issue Nov 3, 2023

gh-109649: Fix test_os.test_process_cpu_count_affinity() #111689

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os.process_cpu_count(): get the number CPUs usable by the current process #109649

os.process_cpu_count(): get the number CPUs usable by the current process #109649

vstinner commented Sep 21, 2023 •

edited by bedevere-app bot

vstinner commented Sep 21, 2023 •

edited

vstinner commented Sep 21, 2023

vstinner commented Sep 21, 2023 •

edited

os.process_cpu_count(): get the number CPUs usable by the current process #109649

os.process_cpu_count(): get the number CPUs usable by the current process #109649

Comments

vstinner commented Sep 21, 2023 • edited by bedevere-app bot

Feature or enhancement

Linked PRs

vstinner commented Sep 21, 2023 • edited

vstinner commented Sep 21, 2023

vstinner commented Sep 21, 2023 • edited

vstinner commented Sep 21, 2023 •

edited by bedevere-app bot

vstinner commented Sep 21, 2023 •

edited

vstinner commented Sep 21, 2023 •

edited