get_open_files might hang forever

[giampaolo]

From amo...@google.com on November 24, 2012 15:39:51

What steps will reproduce the problem?  


1. Create a process_iter()
2. Call get_open_files on each of the processes 



What is the expected output?  
The list of open files for each process. 



What do you see instead?  
The program hangs forever.

This has already been encountered, there are some exception in the code for 
exactly this problem. I have found on more access mask where the problem arises 
- please add it to the list of exceptions. Patch is attached.

Attachment: patch

Original issue: http://code.google.com/p/psutil/issues/detail?id=340

[giampaolo]

From g.rodola on November 24, 2012 11:48:21

Could you provide some info as to where 0x00120089 comes from?

[giampaolo]

From amo...@google.com on November 24, 2012 11:52:36

sadly, I can't. We ran it, it hung on a couple machines, this value was the
access mask.

Do you have any idea where the other values come from? Maybe we can explain
this one then. I looked at the others but the flags they represent are
completely independent, I can't see a pattern.

Sorry, that I don't have more info.

[giampaolo]

From g.rodola on November 24, 2012 12:04:55

What I'd like to know is how exactly you came up with that value (you HAVE to know =)).
Did you debug the problem by printing handle.GrantedAccess and then you noticed 
that when it was ==  0x00120089 the function would hang?

If that's the case then I'm ok with applying the patch.
It's just that I don't feel comfortable in blindly accepting patches for an 
area of the code which is already pretty messy, that's all.

[giampaolo]

From amo...@google.com on November 24, 2012 12:09:07

yeah, exactly. I just printed the value and it hung after getting
0x00120089.

I don't really like the code there either to be honest and I have disabled
this completely locally. There might be more values that hang that we don't
know about. It would be really good to come up with a pattern to determine
if the calls are dangerous...

[giampaolo]

From g.rodola on November 25, 2012 14:38:39

Committed in r1548 .
I agree it would be great if that part was refactored.
Unfortunately it was borrowed from somewhere I can't even remember.
Anyway, closing this out as fixed for now and thanks for the patch.

Status: FixedInSVN
Labels: OpSys-Windows Milestone-0.6.2

[giampaolo]

From wj32...@gmail.com on November 25, 2012 14:42:42

This limits the usefulness of the get_open_files function, since it won't show 
a lot of open files/documents. The only way to completely fix this is to use a 
kernel-mode driver. There might also be a new API in Windows 8 that allows you 
to query the file name properly, but I can't remember what it is...

[giampaolo]

From g.rodola on November 25, 2012 14:56:01

Status: ReOpened

[giampaolo]

From g.rodola on November 26, 2012 03:22:54

wj32.64, thanks for signaling.
I will revert r1548 for now.
If the prospect is to skip files which might be valid then I'd rather risk it 
and signal in the doc that get_open_files() on Windows is not fully reliable.

[giampaolo]

From amo...@google.com on November 26, 2012 12:53:11

So I don't have a strong opinion on this issue since I have just stopped
using get_open_files altogether for now. I just want you to know that I ran
a simple get_open_files for all processes on 50 boxes and it locked up
exactly 20 of them so it's not a rare occurrence.

[giampaolo]

From g.rodola on November 26, 2012 13:54:30

That's weird as despite I have your same configuration (Win7 64bit) I've never 
bumped into this issue before.
Just a thought which occurred to me just now: might it be the case that 
0x00120089 refers to a file living on a network filesystem (e.g. SMB)? In that 
case it might be ok to just skip it.

I still couldn't find the original blog entry where I borrowed that code, nor I 
was able to figure out the the meaning of those codes (0x00120089, etc.).
This is one of those cases where I regret about copying and pasting code 
without understanding it but I had no chance. For things such as this one 
Windows requires to go so close to the kernel that the resulting code is just a 
mess which makes no sense.

[giampaolo]

From jlo...@gmail.com on November 28, 2012 06:17:32

Given what wj32.64 noted above, I would recommend pulling the change back in to 
eliminate the known hang at this point. 

There's not a whole lot we can do to avoid skipping some open files unless 
we're willing to create a kernel-mode driver component for psutil. That's 
something I'm not really eager to do; it would provide more power and 
functionality but it would also be a bunch of added work and complexity to the 
project that I'm not sure is worth the trade-off. 

For now I'd say we skip those we know cause a hang, and note in the 
documentation that due to access rights some files may not show up on Windows.

[giampaolo]

From wj32...@gmail.com on December 12, 2012 19:56:13

You can use Process Hacker's driver which is signed (for 64-bit). But users of 
your library might not like a Python module loading drivers without permission.

[giampaolo]

From g.rodola on December 13, 2012 06:10:05

I reverted the original r1548 commit because it turns out that that change 
basically skips *all* regular (legitimate) files, making get_open_files() 
basically useless on Windows.

@amoser@google.com: I'd like to know a bit more about the circumstances causing the hang.
For which processes do you have this problem?
The ones not owned by your user?
If that's the case we might check permissions up front and raise AccessDenied 
preemptively.

[giampaolo]

From g.rodola on February 24, 2013 11:49:00

@amoser: 
Up. 
Can you tell whether the problem occurs only for processes owned by another user?

Labels: -Milestone-0.6.2

[giampaolo]

From amo...@google.com on February 26, 2013 07:54:45

Sorry for the late response.

I have done some more testing and I have narrowed down the problem a bit. I can 
reproduce by running:

    > PsExec.exe -i -s C:\Windows\system32\cmd.exe

to get system and then

    for p in psutil.process_iter():
       with open("o", "a") as f:
             f.write(p.name)
       try:
          print p.get_open_files()
       except psutil.AccessDenied:
          pass

This will hang on the PsExec.exe process. All other processes including those 
owned by other users and my own cmd.exe work just fine. I hope this helps 
debugging?!