Wrong number of CPU cores detected in Windows Server multi Processor Group environment #771

twisniewski78 · 2016-02-16T15:42:04Z

When running psutil.cpu_count(logical=False) I am getting 60 instead of 120 cores. Similarly when running psutil.cpu_percent(interval=1, percpu=True) I am getting utilization just for half of the CPU cores.
My system has two Processor Groups (each with 60 cores) and it seams that psutil is not processor group aware. On large systems processor resources are split into groups and it requires using special API to get correct numbers.

giampaolo · 2016-09-22T19:43:13Z

Sorry for such a big delay in responding. Can you provide more info about what kind of processors are these? Otherwise I really have no idea what kind of alternative API we could use.

twisniewski78 · 2016-09-22T19:54:17Z

I have eight Xeons E7-2890 v2 with 15 cores each. My first guess would be NUMA architecture and Processor Group awareness rather then type of CPU. I was having the same issues on different system with 80 cores and different CPU model.

giampaolo · 2016-11-25T16:54:50Z

Looking back at this: what does len(psutil.cpu_times(percpu=True)) return? 60 or 120?

giampaolo · 2016-12-01T23:19:48Z

The problem may be here:

psutil/psutil/_psutil_windows.c

Line 582 in 6c3e3e1

if (ptr->Relationship == RelationProcessorCore)

As per:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms686694(v=vs.85).aspx
...if (ptr->Relationship == RelationNumaNode) we may have to increment the counter by 1

Still, that would not match len(psutil.cpu_percent(interval=1, percpu=True)) which according to your report would keep returning half the cores.

cpu_times(percpu=True) internally relies on dwNumberOfProcessors:

psutil/psutil/_psutil_windows.c

Line 980 in 6c3e3e1

for (i = 0; i < si.dwNumberOfProcessors; i++) {

...which should be "correct", so I'm really not sure what to think.

twisniewski78 · 2016-12-02T13:22:16Z

Looking back at this: what does len(psutil.cpu_times(percpu=True)) return? 60 or 120?

60

twisniewski78 · 2016-12-02T13:47:34Z

This is probably nothing new but maybe will give some different perspective: I see this issue in many programs. They are all processor groups unaware recognizing only group0. This is how NUMA nodes look like in my case: One can see there are eight NUMA nodes, each has one CPU with 15 cores. The nodes 0,1,4 and 5 are grouped into GROUP0, the remaining nodes make GROUP1. The phython.exe instance which I run psutil from was assigned by OS (or picked itself) GROUP1: This means it will be unable using group2 resources and that’s why it returned len(psutil.cpu_times(percpu=True)) 60 instead of 120 One workaround would be to start second python.exe process making sure it is assigned to another group and gather statistics including CPUs count from this process. To make it generic algorithm would have to “ 1. count total number of groups (because there may and will be more groups on other systems. Mine is only 50% populated, I will add two more groups and another 120 CPUs in the future) 2. for each group start separate process enforcing different and unique affinity (assignment to specific group) 3. compute in parallel statistics in all processes 4. gather and integrate data from distributed processes into a single instance for reporting 5. shut down all the processes except master one

giampaolo · 2016-12-02T13:53:32Z

Indeed MS doc states:

dwNumberOfProcessors
The number of logical processors in the current group. 
Note  For information about the physical processors shared by logical processors, 
call GetLogicalProcessorInformationEx with the RelationshipType parameter set to 
RelationProcessorPackage (3).

It looks like GetLogicalProcessorInformationEx should give us what we want:

When this function is called with a relationship type of 
RelationProcessorCore, it returns a PROCESSOR_RELATIONSHIP 
structure for every active processor core in every processor group 
in the system.

giampaolo · 2016-12-02T13:56:36Z

Main problem is that I don't have the right hardware to test this against.

giampaolo · 2016-12-02T14:05:07Z

Note: os.cpu_count() on Python 3 relies on GetSystemInfo / dwNumberOfProcessors so it has the same problem.

twisniewski78 · 2016-12-02T14:19:18Z

I know it may be inefficient, but I can run code for you on my system and send back the output if this helps. --- Main problem is that I don't have the right hardware to test this against.

FJplant · 2016-12-03T13:53:29Z

Is this still on-going?

…

Sent from my iPhone4s

On 02 Dec 2016, at 23:19, twisniewski78 ***@***.***> wrote: I know it may be inefficient, but I can run code for you on my system and send back the output if this helps. From: giampaolo ***@***.*** Sent: Friday, December 2, 2016 8:57 AM To: giampaolo/psutil ***@***.***> Cc: twisniewski78 ***@***.***>; Author ***@***.***> Subject: Re: [giampaolo/psutil] Wrong number of CPU cores detected in Windows Server multi Processor Group environment (#771) Main problem is that I don't have the right hardware to test this against. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#771 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AQeYyM_tjkw3Nx69xlAxcHp4WH0JwvRrks5rECOWgaJpZM4HbQUI> . <https://github.com/notifications/beacon/AQeYyNEPTEV-rq9uOv91PQoRiCrRjIJhks5rECOWgaJpZM4HbQUI.gif> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

giampaolo · 2016-12-03T13:55:09Z

It is (still investigating and not fixed).

FJplant · 2016-12-03T13:58:42Z

Thank you Giampaolo.:-)

…

On Dec 3, 2016 10:55 PM, "giampaolo" ***@***.***> wrote: It is (still investigating and not fixed). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#771 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJSP__R6JtPJ7ZAXv-ZBw6PzYWkKomWYks5rEXS_gaJpZM4HbQUI> .

waterkirin · 2017-01-17T06:17:45Z

I may got the same issue. I have dual E5 2686 v3 with 18 cores each(36 cores and 72 logical processors ). When I ran psutil.cpu_count(logical = False) , I got 18. When I ran psutil.cpu_count(logical = True), i got 32. This is confusing.

giampaolo · 2017-01-17T12:23:20Z

Can you confirm os.cpu_count() has the same issue?

giampaolo · 2017-01-19T16:12:16Z

Here it is suggested that this may be due to 32 bit processes. Have you tried using Python 64 bit?

FJplant · 2017-01-19T16:24:31Z

Not completed... ^^

giampaolo · 2017-02-14T10:54:38Z

Another useful link:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx

robbuckley · 2017-04-28T16:32:47Z

hi, i have the same issue. I am running 64 bit python 3.6.0 (anaconda) on 2 similar platforms
platform1 - 2x Xeon E5-2698v4. 20 cores/CPU = total 80 logical cpus with hyperthreading
platform2 - 2x Xeon E5-2697v3. 14 cores/CPU = total 56 logical cpus with hyperthreading

os.cpu_count(), multiprocessing.cpu_count() and psutil.cpu_count() all report the same thing: 40 cores on platform1 and 56 on platform2.

sysinternals coreinfo shows that platform1 has 2 processor groups and platform2 has 1 group.

i have a simple python program using a threadpool to launch multiple subprocess of an external tool which is entirely CPU bound. On platform 2 i can get it to 100% CPU for sustained periods. on Platform1 it tops out at ~50%

i would really like to be able to get this using 100% of platform1.

Im happy to run any test to help with this issue

giampaolo · 2017-04-28T16:35:48Z

@robbuckley can you confirm os.cpu_count() has the same issue? That has been introduced in Python 3.4. If you're on 2.7 you can use multiprocessing.cpu_count().

robbuckley · 2017-04-28T16:56:39Z

@giampaolo yes I confirm. I didn't realise they are the same. Im new to multiprocessing in python :-)

giampaolo · 2017-04-28T17:03:52Z

Could you please try Python 3.7 + os.cpu_count(). I suppose we should file a ticket on the Python bug tracker. Hopefully we can get some insight from there. The main problem here is that I don't have the right hardware to test this against.

robbuckley · 2017-05-03T14:35:22Z

How do I install python 3.7? according to https://www.python.org/dev/peps/pep-0537/ there doesnt seem to be an 3.7.0 alpha 1 yet, and i couldnt find anything in the conda channels including conda-forge.

I'm happy to help but dont have a lot of time to spend on it. Surely reporting a bug against the latest released version (3.6.1) should be OK?

giampaolo · 2017-05-03T14:36:31Z

Yeah, 3.6 is also OK.

robbuckley · 2017-05-04T21:09:37Z

@giampaolo is there anything more you need from me now?

giampaolo · 2017-05-05T22:18:10Z

Is os.cpu_count() on python 3.6 broken in the same manner?

robbuckley · 2017-05-16T16:26:28Z

yes, i confirm. See attached python session with os,cpu_count() reporting 40 cores, and output of microsoft coreinfo tool showing 80 logical processors in 2 processor groups

ci.txt
osc.txt

giampaolo · 2017-05-16T16:27:23Z

OK, so can you please file an issue on http://bugs.python.org/?

robbuckley · 2017-06-06T12:50:29Z

http://bugs.python.org/issue30581 raised

giampaolo · 2018-03-28T13:10:20Z

Also related: https://bugs.python.org/issue33166

giampaolo · 2018-03-28T19:58:26Z

I created a psutil branch using GetLogicalProcessorInformation (not GetLogicalProcessorInformationEx) to determine both logical and physical CPUs:
#1257
@twisniewski78 / @robbuckley : can you try it?
If you don't have Visual Studio I can send you a .whl file for Python 2.7 32 bit.

twisniewski78 · 2018-03-28T20:11:19Z

Sadly I have no longer access to this environment or anything like it.

giampaolo · 2018-04-13T08:53:20Z

OK, this should finally be fixed as of #1257.

giampaolo · 2018-10-20T13:37:14Z

Closing out as fixed.

giampaolo added bug windows labels Sep 22, 2016

giampaolo mentioned this issue Sep 5, 2017

Add usable_cpu_count() function #1122

Open

giampaolo added the critical label Mar 28, 2018

giampaolo mentioned this issue Mar 28, 2018

771 Windows CPU count #1257

Merged

twisniewski78 closed this as completed Mar 28, 2018

giampaolo reopened this Mar 28, 2018

dependencies bot mentioned this issue Apr 14, 2018

psutil versions available: 5.4.5, 5.4.4 Harmon758/Harmonbot#76

Closed

giampaolo closed this as completed Oct 20, 2018

GermanAizek mentioned this issue Jan 29, 2024

Windows: incorrect detection of core count on NUMA hardware. rizinorg/cutter#3286

Open

GermanAizek mentioned this issue Mar 5, 2024

Does not using all threads on NUMA configuration (server motherboards 2, 4, 6 multisocket CPU) on Windows ollama/ollama#2936

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrong number of CPU cores detected in Windows Server multi Processor Group environment #771

Wrong number of CPU cores detected in Windows Server multi Processor Group environment #771

twisniewski78 commented Feb 16, 2016

giampaolo commented Sep 22, 2016

twisniewski78 commented Sep 22, 2016 •

edited by giampaolo

Loading

giampaolo commented Nov 25, 2016

giampaolo commented Dec 1, 2016

twisniewski78 commented Dec 2, 2016 via email •

edited by giampaolo

Loading

twisniewski78 commented Dec 2, 2016 via email •

edited by giampaolo

Loading

giampaolo commented Dec 2, 2016

giampaolo commented Dec 2, 2016

giampaolo commented Dec 2, 2016

twisniewski78 commented Dec 2, 2016 via email •

edited

Loading

FJplant commented Dec 3, 2016 via email

giampaolo commented Dec 3, 2016

FJplant commented Dec 3, 2016 via email

waterkirin commented Jan 17, 2017

giampaolo commented Jan 17, 2017

giampaolo commented Jan 19, 2017

FJplant commented Jan 19, 2017

giampaolo commented Feb 14, 2017

robbuckley commented Apr 28, 2017

giampaolo commented Apr 28, 2017

robbuckley commented Apr 28, 2017

giampaolo commented Apr 28, 2017

robbuckley commented May 3, 2017

giampaolo commented May 3, 2017

robbuckley commented May 4, 2017

giampaolo commented May 5, 2017

robbuckley commented May 16, 2017

giampaolo commented May 16, 2017 •

edited

Loading

robbuckley commented Jun 6, 2017

giampaolo commented Mar 28, 2018

giampaolo commented Mar 28, 2018

twisniewski78 commented Mar 28, 2018

giampaolo commented Apr 13, 2018

giampaolo commented Oct 20, 2018

Wrong number of CPU cores detected in Windows Server multi Processor Group environment #771

Wrong number of CPU cores detected in Windows Server multi Processor Group environment #771

Comments

twisniewski78 commented Feb 16, 2016

giampaolo commented Sep 22, 2016

twisniewski78 commented Sep 22, 2016 • edited by giampaolo Loading

giampaolo commented Nov 25, 2016

giampaolo commented Dec 1, 2016

twisniewski78 commented Dec 2, 2016 via email • edited by giampaolo Loading

twisniewski78 commented Dec 2, 2016 via email • edited by giampaolo Loading

giampaolo commented Dec 2, 2016

giampaolo commented Dec 2, 2016

giampaolo commented Dec 2, 2016

twisniewski78 commented Dec 2, 2016 via email • edited Loading

FJplant commented Dec 3, 2016 via email

giampaolo commented Dec 3, 2016

FJplant commented Dec 3, 2016 via email

waterkirin commented Jan 17, 2017

giampaolo commented Jan 17, 2017

giampaolo commented Jan 19, 2017

FJplant commented Jan 19, 2017

giampaolo commented Feb 14, 2017

robbuckley commented Apr 28, 2017

giampaolo commented Apr 28, 2017

robbuckley commented Apr 28, 2017

giampaolo commented Apr 28, 2017

robbuckley commented May 3, 2017

giampaolo commented May 3, 2017

robbuckley commented May 4, 2017

giampaolo commented May 5, 2017

robbuckley commented May 16, 2017

giampaolo commented May 16, 2017 • edited Loading

robbuckley commented Jun 6, 2017

giampaolo commented Mar 28, 2018

giampaolo commented Mar 28, 2018

twisniewski78 commented Mar 28, 2018

giampaolo commented Apr 13, 2018

giampaolo commented Oct 20, 2018

twisniewski78 commented Sep 22, 2016 •

edited by giampaolo

Loading

twisniewski78 commented Dec 2, 2016 via email •

edited by giampaolo

Loading

twisniewski78 commented Dec 2, 2016 via email •

edited by giampaolo

Loading

twisniewski78 commented Dec 2, 2016 via email •

edited

Loading

giampaolo commented May 16, 2017 •

edited

Loading