pid_exists() on Windows is 35x slower than Linux and OS X implementations

[giampaolo]

From billiej...@gmail.com on February 23, 2009 16:59:27

Linux:

root@ubuntu:/home/user# python -m timeit -s "import psutil" -c
"psutil.pid_exists(1)"
100000 loops, best of 3: 4.5 usec per loop


Windows:

C:\>C:\python26\python.exe -m timeit -s "import psutil" -c
"psutil.pid_exists(1)"
10000 loops, best of 3: 161 usec per loop


This is due to the underlying Windows implementation using "pid in
get_all_pids()" which is much slower than Linux and OS X implementations
relying on "os.kill(pid, 0)".

Original issue: http://code.google.com/p/psutil/issues/detail?id=20

[giampaolo]

From billiej...@gmail.com on February 23, 2009 08:04:53

Don't know if it's actually useful but we could try IsProcessInJob: 

[giampaolo]

From daver...@rsaisp.com on February 23, 2009 08:51:20

I think a simple OpenProcess and check the error code would suffice

[giampaolo]

From jlo...@gmail.com on February 23, 2009 09:56:10

OpenProcess was actually allowing a process handle to be retrieved on a dead 
during my initial testing. I'll try a simple C console app with a bogus PID 
OpenProcess to see if it errors out, but at the moment it looks like OpenProcess
isn't sufficient to detect when a process has gone away on us.

Status: Accepted
Owner: jloden

[giampaolo]

From jlo...@gmail.com on February 23, 2009 17:45:54

Well, as I had initially observed, evidently it is the case that OpenProcess 
on invalid PIDs in some cases. See 
http://blogs.msdn.com/oldnewthing/archive/2008/06/06/8576557.aspx for details, 
basically the bottom two bits of the PID are ignored so we can't reliably use 
for determining if the PID was valid. 

For now I'll implement pid_exists() in C and expose it to Python.

[giampaolo]

From jlo...@gmail.com on February 23, 2009 18:38:51

Implemented in C for r150 so it's considerably faster, though still quite a bit
slower than OS X and Linux, unfortunately: 

python2.6.exe -m timeit -s "import psutil" -c "psutil.pid_exists(1)"
10000 loops, best of 3: 70.3 usec per loop

Tried it a few times and it hovers somewhere around 70-75 usec per loop, which 
quit a bit faster but still unfortunately slow. Further analysis shows the slow 
is actually the get_pids() function which calls EnumProcesses to get a list of
process PIDs. So, bottom line is that the collection of PIDs with the Win32 API 
where we are taking the big speed hit here. Unless we read the kernel the PIDs 
EPROCESS structs directly or something similar I doubt we can make much of a 
the current speed.

I'll do some more digging and see if there's some other way we check if a 
valid.

[giampaolo]

From jlo...@gmail.com on February 23, 2009 19:52:51

This is also possible using OpenProcess() and GetExitCodeProcess() but,
unbelievably, this is even SLOWER:

python2.5 -m timeit -s "import psutil" -c "psutil.pid_exists(268)"
1000 loops, best of 3: 354 usec per loop


python2.6 -m timeit -s "import psutil" -c "psutil.pid_exists(268)"
1000 loops, best of 3: 331 usec per loop

In light of that I'll leave r150 in place and not bother checking this in. I
tried removing GetExitCodeProcess() from the equation, and even *just*
OpenProcess() provides almost the same (slow) results.

[giampaolo]

From jlo...@gmail.com on February 26, 2009 12:36:57

Asked this on StackOverflow as well: http://stackoverflow.com/questions/592256/

Status: Started

[giampaolo]

From jlo...@gmail.com on March 01, 2009 10:11:53

Ok, I have no idea what I did wrong before, but OpenProcess() / 
definitely not slower than EnumProcesses()

On a hunch I wrote two small standalone apps in C to test each in a loop of 
iterations and OpenProcess with GetExitCodeProcess came up much faster, so i 
the C code for _psutil_mswindows to use this method and tested again: 

python -m timeit -s "import psutil" -c "psutil.pid_exists(0)"
1000000 loops, best of 3: 1.2 usec per loop

python -m timeit -s "import psutil" -c "psutil.pid_exists(1)"
100000 loops, best of 3: 3.34 usec per loop

python -m timeit -s "import psutil" -c "psutil.pid_exists(1280)"
100000 loops, best of 3: 6.24 usec per loop

Note that PID 0 returns immediately as a special case, so that's why the fast 
there. PID 1 is near the beginning of the process list so is found faster, but 
with a high PID we're looking at around 6.5 usec per loop, which puts us on par 
the other platforms for performance.

Checked in as r194 .

Status: Fixed

[giampaolo]

From g.rodola on March 02, 2013 03:44:02

Updated csets after the SVN -> Mercurial migration: r150 == revision