Process.environ() on Windows may raise WindowsError

[giampaolo]

https://ci.appveyor.com/project/giampaolo/psutil/build/806/job/fpxxbai3sg291i75

======================================================================
ERROR: test_proc_environ (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\psutil\psutil\tests\test_process.py", line 2054, in test_proc_environ
    encode_path(p.environ()['FUNNY_ARG']), funny_path)
  File "c:\projects\psutil\psutil\__init__.py", line 772, in environ
    return self._proc.environ()
  File "c:\projects\psutil\psutil\_pswindows.py", line 536, in wrapper
    return fun(self, *args, **kwargs)
  File "c:\projects\psutil\psutil\_pswindows.py", line 597, in environ
    return parse_environ_block(cext.proc_environ(self.pid))
WindowsError: [Error 299] Only part of a ReadProcessMemory or WriteProcessMemory request was comp

[giampaolo]

@fbenkstein do you mind taking a look at this or do you have any suggestion?
The error occurs here:

psutil/psutil/arch/windows/process_info.c

Line 663 in 7c36a63

if (!ReadProcessMemory(hProcess, src, buffer, size, NULL)) {

[fbenkstein]

@giampaolo how often does this occur? I assume it isn't reproducible reliably. Unfortunately I don't have a Windows dev environment at the moment so I can't debug this myself. Maybe it's possible that we're trying to read the process memory when it isn't fully created yet or otherwise the test machine was so busy that the test process exited while we were trying to read its memory.

[giampaolo]

I can reproduce it if I run that test in a loop. After a while (say 20-30 times) I bump into the exception. I suppose in that case we could realloc the buffer with a bigger size and retry. I will try that.

[fbenkstein]

I don't think it's the buffer size. If it's too small you'll simply read less. IIRC increasing the buffer will probably increase the likelihood of the error since you might try to read memory beyond what is actually valid in the process. If you want to handle the error, it's probably better to check if the process is still alive, though I think this should be left up to the caller of environ().

[giampaolo]

In that case (unit test) the process is alive for sure.

[giampaolo]

It turns out this also occurs for cmdline() (#1028). Since it all belongs to the same C code section cmdline(), cwd() and environ()all suffer the same issue.

[gozdal]

@giampaolo would you consider a workaround for ERROR_PARTIAL_COPY? The reasoning for the orkaround is in https://stackoverflow.com/questions/4457171/why-does-readprocessmemory-have-lpnumberofbytesread/4457745#4457745

[giampaolo]

I forgot about this one and I haven't seen it in a while meaning that (maybe) it got fixed but I forgot to update this issue. Can you reproduce it with latest version?

[gozdal]

Yes, it's reproducible given enough tries.

[jurcaua]

It can be reproduced with this:

import os
import psutil
import subprocess


def find_pid_by_name(name):
    "Return a list of processes matching 'name'."
    ls = []
    for p in list(psutil.process_iter(attrs=["name", "exe", "cmdline"])):
        if name == p.info['name'] or \
                p.info['exe'] and os.path.basename(p.info['exe']) == name or \
                p.info['cmdline'] and p.info['cmdline'][0] == name:
            ls.append(p)
    if len(ls) == 1:
        return ls[0].pid
    return -1

for i in range(10000):
    info = subprocess.STARTUPINFO()
    info.dwFlags = 1
    info.wShowWindow = 0
    process = subprocess.Popen(
        "notepad.exe",
        creationflags=subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_NEW_CONSOLE,
        startupinfo=info)

    while True:
        before = find_pid_by_name("notepad.exe")

        if before != -1:
            os.kill(before, 9)
        else:
            break

        after = find_pid_by_name("notepad.exe")
        print(before, after)
        if before != after and after == -1:
            break

[eight04]

Just saw this error in v5.6.3.

Traceback (most recent call last):
  File "pymemorymon.py", line 235, in <module>
    Main()
  File "pymemorymon.py", line 218, in __init__
    self.view()
  File "pymemorymon.py", line 230, in view
    self.monitor.event_loop()
  File "pymemorymon.py", line 134, in event_loop
    self.monitor()
  File "pymemorymon.py", line 106, in monitor
    process.update()
  File "pymemorymon.py", line 33, in update
    "pid", "name", "cpu_percent", "memory_percent", "cmdline", "exe"
  File "D:\Dev\pyMemoryMon\.venv\lib\site-packages\psutil\__init__.py", line 638
, in as_dict
    ret = meth()
  File "D:\Dev\pyMemoryMon\.venv\lib\site-packages\psutil\__init__.py", line 790
, in cmdline
    return self._proc.cmdline()
  File "D:\Dev\pyMemoryMon\.venv\lib\site-packages\psutil\_pswindows.py", line 7
18, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
  File "D:\Dev\pyMemoryMon\.venv\lib\site-packages\psutil\_pswindows.py", line 7
08, in convert_oserror
    raise exc
  File "D:\Dev\pyMemoryMon\.venv\lib\site-packages\psutil\_pswindows.py", line 7
16, in wrapper
    return fun(self, *args, **kwargs)
  File "D:\Dev\pyMemoryMon\.venv\lib\site-packages\psutil\_pswindows.py", line 7
97, in cmdline
    ret = cext.proc_cmdline(self.pid, use_peb=True)
OSError: [WinError 299] 僅完成部分的 ReadProcessMemory 或 WriteProcessMemory 要
求。: '(originated from ReadProcessMemory(ProcessParameters))'

[gozdal]

@giampaolo Thanks for your work on this issue. I wonder though if it would be more sensible to retry once or twice before returning AccessDenied? We're using a wrapper in our code which retries operations which call ReadProcessMemory and since we had merged this wrapper we didn't see ERROR_PARTIAL_COPY.

If psutil returns AccessDenied instead of OSError it still doesn't solve the root issue - Windows sometimes returns this error when you in fact have all the privileges to read process memory but the copy encounters page fault when copying.

[giampaolo]

I agree with you. Thanks for the insight.

[Courstick]

<function Process.environ at 0x0000026552DA0E18> retrieed 33 times, converted to AccessDenied as it's still returning OSError(22,'Only part of a ReadProcessMemory or WriteProcessMemory request was completed.')

I also got this error on Windows in psutil == 5.9.4， which looks similar to this problem