[Ubuntu 16.04] FileNotFoundError: [Errno 2] No such file or directory: '/proc/12825/stat'

[pseudotensor]

Platform

• Ubuntu 16.04

$ pip show psutil
Name: psutil
Version: 5.4.5
Summary: Cross-platform lib for process and system monitoring in Python.
Home-page: https://github.com/giampaolo/psutil
Author: Giampaolo Rodola
Author-email: g.rodola@gmail.com
License: BSD
Location: /home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages

Bug description

def find_procs_by_name(name):
    "Return a list of processes matching 'name'."
    ls = []
    for p in psutil.process_iter(attrs=['name']):
        try:
            if name in p.info['name'] or name in getprocname(p):
                ls.append(p)
        except psutil.NoSuchProcess:
            pass
        except psutil.AccessDenied:
            pass
    return ls

def getprocname(pp, joinchar=" "):
    # assume pp has pid that exists
    ppname1 = str(pp.name())
    cmdline = pp.cmdline()
    cmdline = [x for x in cmdline if x != '']
    ppname2 = str(joinchar.join(cmdline))
    if len(ppname1) > len(ppname2):
        return ppname1
    else:
        return ppname2

rarely (but shouldn't ever) gives:

Traceback (most recent call last):
  File "/data/jon/h2oai/h2oaicore/systemutils.py", line 3100, in find_procs_by_name
    for p in psutil.process_iter(attrs=['name']):
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 1440, in process_iter
    if proc.is_running():
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 569, in is_running
    return self == Process(self.pid)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 340, in __init__
    self._init(pid)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 367, in _init
    self.create_time()
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 698, in create_time
    self._create_time = self._proc.create_time()
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 1402, in wrapper
    return fun(self, *args, **kwargs)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 1584, in create_time
    values = self._parse_stat_file()
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_common.py", line 337, in wrapper
    return fun(self)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 1441, in _parse_stat_file
    with open_binary("%s/%s/stat" % (self._procfs_path, self.pid)) as f:
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 187, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/12825/stat'

I thought process_iter() and proc.is_running() were supposed to be robust against processes going away, etc.

I basically have to stop using psutil's process iter because it's not reliable. Instead, have to use "ps" on command line via subprocess.check_output().

[giampaolo]

Not sure how I messed up something so obvious. =)
Fixed.

[abditag2]

@giampaolo I am still seeing this problem with psutil==5.7.0. Is this fixed?

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/psutil/_common.py", line 340, in wrapper
    ret = self._cache[fun]
AttributeError: _cache

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/site-packages/petastorm/workers_pool/process_pool.py", line 327, in 
    main_process_alive = any([process.pid for process in process_iter() if process.pid == 
  File "/usr/lib/python3.6/site-packages/petastorm/workers_pool/process_pool.py", line 327, in <listcomp>
    main_process_alive = any([process.pid for process in process_iter() if process.pid == 
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 1569, in process_iter
    if proc.is_running():
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 695, in is_running
    return self == Process(self.pid)
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 447, in __init__
    self._init(pid)
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 474, in _init
    self.create_time()
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 824, in create_time
    self._create_time = self._proc.create_time()
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1517, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1723, in create_time
    ctime = float(self._parse_stat_file()['create_time'])
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1517, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/psutil/_common.py", line 343, in wrapper
    return fun(self)
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1559, in _parse_stat_file
    with open_binary("%s/%s/stat" % (self._procfs_path, self.pid)) as f:
  File "/usr/lib/python3.6/site-packages/psutil/_common.py", line 604, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/110/stat'

[jamilraichouni]

Same here with v5.7.0 of psutil

I call this function from within a Shell script:

def alive():
    '''
        Sub command to check, if VSCode is running.

        If VSCode is running the function will print 1 otherwise 0.
        The printed value can be understood in Shell scripts calling

        .. code-block:: bash

            python vscode.py alive

        Returns
        -------
        bool
            True if VSCode is running, else False

    '''
    if 'darwin' in PLATFORM_LOWER:
        vscode_exe_name_lw = 'visual studio code'
    elif 'windows' in PLATFORM_LOWER:
        vscode_exe_name_lw = 'Code.exe'
    elif 'linux' in PLATFORM_LOWER:
        vscode_exe_name_lw = 'code'
    for proc in psutil.process_iter(attrs=('exe',)):
        try:
            exe = proc.info['exe']
        except (psutil.AccessDenied, psutil.NoSuchProcess):
            exe = None
        if exe is not None and vscode_exe_name_lw in exe.lower():
            print(1)
            return True
    print(0)
    return False

and get a wrong result including exceptions:

$ bash /Users/USER/tmp/test.sh
Traceback (most recent call last):
  File "/Users/USER/repos/vscode/vscode.py", line 2358, in <module>
    cli()
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/USER/repos/vscode/vscode.py", line 1506, in alive
    for proc in psutil.process_iter(attrs=('exe',)):
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 1457, in process_iter
    yield add(pid)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 1434, in add
    proc.info = proc.as_dict(attrs=attrs, ad_value=ad_value)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 538, in as_dict
    ret = meth()
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 673, in exe
    exe = self._proc.exe()
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/_psosx.py", line 334, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/_psosx.py", line 413, in exe
    return cext.proc_exe(self.pid)
FileNotFoundError: [Errno 2] No such file or directory (originated from proc_pidpath())
/Users/USER/tmp/test.sh: line 4: [: ==: unary operator expected
Not alive

The exception occurs in the code line

for proc in psutil.process_iter(attrs=('exe',)):

[giampaolo]

It's fixed on master but there's no release yet.

[jamilraichouni]

Okay, thanks! It's weird, because when I run the exact same function code in Jupyter with the same conda env all runs through.

My function "alive" from above is a sub command of a command line tool where I use click for the argument processing.

[wookayin]

Is it fixed? @zhudd-hub has reported in wookayin/gpustat#95 that this issue still happens with psutil 5.8.0. A FileNotFoundError was thrown, not being translated. Please find a stacktrace in the linked issue.