incorrect per-process CPU usage reported [Windows]

[giampaolo]

From francois...@gmail.com on February 11, 2014 00:42:48

What steps will reproduce the problem?  
1.Create a psutil.Process object for an existing PID
2.Call get_cpu_percent() on the Process object
3.Value returned is capped at 100 

What is the expected output?  
The value returned fits the per-process CPU % from Windows Performance 
monitor (perfmon.exe) "Process/% Processor Time" counter, which is not 
capped at 100 (the max is 100 * NB_CORES) 

What do you see instead?  
It is capped at 100 

What version of psutil are you using? What Python version?  
psutil 1.2.1
python 2.7.6 

On what operating system? Is it 32bit or 64bit version?  
Windows 8 Enterprise, 64-bit 

Please provide any additional information below.  
I already saw bug 194 that has been closed where it is seen as an error to 
report values higher than 100 on Unix platforms. I would not have any 
particular issue with that on Windows either, as long as the reported values 
are scaled accurately, which they are not.

I am currently running performance tests on a multithreaded program of mine, 
and I was surprised to see that the values reported by your tool did not 
correspond at all to the values I read in the task manager. I then investigated 
the issue and found that they rather correspond to the values reported by the 
aforementioned counter in Performance monitor (procmon.exe). The problem is, as 
soon as the value raises above 100 (which happens frequently on a multicore 
system), they are clipped to 100, which gives wrong values. 

In its current state, I cannot use this library for the intent I planned for it.

Thank you for looking into this. I can help solve the bug if it is approved.

Original issue: http://code.google.com/p/psutil/issues/detail?id=474

[giampaolo]

From g.rodola on February 13, 2014 16:52:48

According to issue 194 the reason why I decided to limit CPU percentage to 100% 
on Windows was because taskmgr.exe does the same thing.
Now it appears you're saying the opposite.
Maybe this is true on Windows 8 only? I don't know (I test psutil on Windows 7).

Question: if you manually remove the 100% limit I set in psutil code and 
compare psutil values with taskmgr's and procmon.exe do they look similar?

Personally I don't have any particular reason NOT to remove the limit as long 
as the return value is correct across different Windows versions  as it is on UNIX.
I'd say one way to investigate this would be to run a program spawning 2 
threads, each thread running a "while 1: pass" loop.
On a 10-cores system I would expect to see one process reporting 200% CPU usage.

Side note: such a test cannot be done in cPython because of the GIL so we'd 
have to cook that up by using another language or interpreter (IronPython or Jython).

[giampaolo]

From francois...@gmail.com on February 14, 2014 06:32:35

Hi,


The task manager indeed shows values normalized (not capped) on a 0-100% base. 
However, the value that Psutil gets from the system is not normalized (not 
divided by the number of cores in the system), so it makes no sense to cap it. 




I built locally a version where I just removed the cap and it seems to work 
perfectly fine. I can perform the test you asked for in the coming days to 
confirm it behaves as expected on more CPU-intensive workflows, as mine does 
not go much higher than 120%, but for political/legal reasons, the company 
where I work won't let me contribute directly to the project (I know it 
sucks...). Since the fix is only to remove the cap though, I'm sure it won't be 
too much of a hassle to do it yourself.




So I'll get back to you with test results to confirm my theories in the coming days.

[giampaolo]

From g.rodola on February 14, 2014 08:56:03

> the company where I work won't let me contribute directly to the
> project (I know it sucks...)

I'm sorry. That's just stupid.
I wish there would be a license not allowing such companies to use open source stuff.

[giampaolo]

From francois...@gmail.com on February 18, 2014 08:44:51

I agree, but I cannot change that. I am currently bending the rules opening 
this issue and performing tests to get this fixed instead of just forking it 
locally on our server...

As for the results:
1) I coded a tiny c++11 application that busy loops on 4 threads:

#include "stdafx.h"
#include <thread>
#include <iostream>

int _tmain(int argc, _TCHAR* argv[])
{
   std::thread* aThreads[4];
   bool bStop = false;

   for(int i = 0; i < 4; i++)
   {
      aThreads[i] = new std::thread([&bStop]()
      { 
         while(1)
         {
            // perform useless computation
            int x = 3 + 4;
            int y = x * 2;
            if(bStop)
               break;
         }            
      });
   }

   std::cin >> bStop;

   for(int i = 0; i < 4; i++)
   {
      if(aThreads[i])
      {
         aThreads[i]->join();
         delete aThreads[i];
      }
   }
    return 0;
}
--------------

2) I then use a python script to launch the process and grab its cpu_percent 
values with the 100 percent cap removed:

#!/usr/bin/python

import sys
import psutil
import subprocess

def go():
   # creating process to run application
   process = subprocess.Popen(["400percentbusy.exe"])

   p = psutil.Process(process.pid)
   if p.is_running():
      cpu_vals = []
      print "Grabbing results for the next 10 seconds..."
      for i in range(0,10):
         try:
            cpu_vals.append(p.get_cpu_percent(1))
            print 'cpu: {}'.format(str(cpu_vals[i]))
         except:
            print "Process ended while still taking measures! Accuracy might be affected."
            break
      print ''
      print 'SUMMARY:'
      print 'CPU     - MIN: {} MAX: {} AVG: {}'.format(str(min(cpu_vals)), 
str(max(cpu_vals)), str(sum(cpu_vals)/float(len(cpu_vals))))
   else:
      print 'invalid PID'

-----------------

3) I get the following output:

>>> test400percent.go()
Grabbing results for the next 1 minute...
cpu: 398.9
cpu: 402.0
cpu: 398.1
cpu: 396.5
cpu: 398.2
cpu: 410.1
cpu: 398.9
cpu: 398.2
cpu: 401.2
cpu: 398.9

SUMMARY:
CPU     - MIN: 396.5 MAX: 410.1 AVG: 400.1

---------------

This comfirms that the value grabbed by psutil is the sum of the percentages 
from all the cores in the system, not normalized and as such, it should not be 
capped to 100.

I hope you can overlook your frustration and perform the fix, as I think I did 
the best I could considering the circumstances and I think your project can 
benefit from it. If not, well I wish you good luck in your project(s), as you 
made a really useful piece of software.

Thanks for your help

[giampaolo]

From g.rodola on February 23, 2014 17:34:18

Fixed in revision fac74cc73582 .
Thanks for detailed insights.

Status: FixedInHG
Labels: OpSys-Windows Milestone-2.0.0

[giampaolo]

From g.rodola on March 10, 2014 04:36:50

Closing out as fixed as 2.0.0 version is finally out.

Status: Fixed

[fx86]

How does one normalize the cpu percent consumed if it goes above 100% ?

[giampaolo]

What do you mean?

[fx86]

Meaning, should we divide the CPU % given with percpu=False by the number of cores OR should it be average of values given by percpu=True ?

[giampaolo]

From psutil doc: https://pythonhosted.org/psutil/#psutil.Process.cpu_percent

<<Note the returned value is explicitly not split evenly between all available CPUs (differently from psutil.cpu_percent()). This means that a busy loop process running on a system with 2 logical CPUs will be reported as having 100% CPU utilization instead of 50%. This was done in order to be consistent with top UNIX utility and also to make it easier to identify processes hogging CPU resources independently from the number of CPUs. It must be noted that taskmgr.exe on Windows does not behave like this (it would report 50% usage instead). To emulate Windows taskmgr.exe behavior you can do: p.cpu_percent() / psutil.cpu_count().>>

[fx86]

Thanks much!
Had missed that line.