Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ubuntu 16.04] FileNotFoundError: [Errno 2] No such file or directory: '/proc/12825/stat' #1447

Closed
pseudotensor opened this issue Mar 2, 2019 · 6 comments
Assignees

Comments

@pseudotensor
Copy link

pseudotensor commented Mar 2, 2019

Platform

  • Ubuntu 16.04
$ pip show psutil
Name: psutil
Version: 5.4.5
Summary: Cross-platform lib for process and system monitoring in Python.
Home-page: https://github.com/giampaolo/psutil
Author: Giampaolo Rodola
Author-email: g.rodola@gmail.com
License: BSD
Location: /home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages

Bug description

def find_procs_by_name(name):
    "Return a list of processes matching 'name'."
    ls = []
    for p in psutil.process_iter(attrs=['name']):
        try:
            if name in p.info['name'] or name in getprocname(p):
                ls.append(p)
        except psutil.NoSuchProcess:
            pass
        except psutil.AccessDenied:
            pass
    return ls

def getprocname(pp, joinchar=" "):
    # assume pp has pid that exists
    ppname1 = str(pp.name())
    cmdline = pp.cmdline()
    cmdline = [x for x in cmdline if x != '']
    ppname2 = str(joinchar.join(cmdline))
    if len(ppname1) > len(ppname2):
        return ppname1
    else:
        return ppname2

rarely (but shouldn't ever) gives:

Traceback (most recent call last):
  File "/data/jon/h2oai/h2oaicore/systemutils.py", line 3100, in find_procs_by_name
    for p in psutil.process_iter(attrs=['name']):
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 1440, in process_iter
    if proc.is_running():
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 569, in is_running
    return self == Process(self.pid)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 340, in __init__
    self._init(pid)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 367, in _init
    self.create_time()
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/__init__.py", line 698, in create_time
    self._create_time = self._proc.create_time()
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 1402, in wrapper
    return fun(self, *args, **kwargs)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 1584, in create_time
    values = self._parse_stat_file()
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_common.py", line 337, in wrapper
    return fun(self)
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 1441, in _parse_stat_file
    with open_binary("%s/%s/stat" % (self._procfs_path, self.pid)) as f:
  File "/home/jon/.pyenv/versions/3.6.4/lib/python3.6/site-packages/psutil/_pslinux.py", line 187, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/12825/stat'

I thought process_iter() and proc.is_running() were supposed to be robust against processes going away, etc.

I basically have to stop using psutil's process iter because it's not reliable. Instead, have to use "ps" on command line via subprocess.check_output().

@giampaolo
Copy link
Owner

giampaolo commented Mar 3, 2019

Not sure how I messed up something so obvious. =)
Fixed.

nlevitt added a commit to nlevitt/psutil that referenced this issue Apr 9, 2019
* origin/master: (150 commits)
  Linux / CPU freq, fixes giampaolo#1481
  improve pmap.py script
  reuse ps.py script in psutil.test()
  move get_terminal_size() in _compat.py
  improve ps.py script
  improve ps.py script
  move bytes2human() into psutil._common and reused it from scripts dir
  fix windows failure re. py 2 vs. 3
  fix linux tests
  fix giampaolo#1474: fix formatting of psutil.tests() which mimicks 'ps aux' output
  give CREDITS for giampaolo#1480
  remove outdated tests
  Fix read access violation in psutil.cpu_count(logical=False) (giampaolo#1480)
  update doc
  update doc
  [Win] Process IO priority constants + high priority (giampaolo#1479 / giampaolo#1476)
  don't fail if there are not prev failed tests
  fix giampaolo#1478: add make command to re-run tests failed on last run
  [Win] return value is not properly handled for undocumented NT* Windows APIs. (giampaolo#1477)
  fix error on py 2.7 where OSError doesn't always have winerror attribute
  update HISTORY for giampaolo#1475
  properly check OSError.winerror
  refactor ionice() on Linux
  refactor ionice() on Linux
  ionice test refactoring
  giampaolo#1404: fix regression not returning CPUs > 9
  give CREDITS to Daniel Beer for giampaolo#1471
  Fix spurious exception when iterating processes on Solaris (giampaolo#1471)
  give CREDITS for giampaolo#1470
  Fix corner case when /etc/mtab doesn't exist and procfs=/proc (giampaolo#1470)
  update DEVNOTES
  Typo fixed (giampaolo#1469)
  giampaolo#1458: implement colors on Windows
  update HISTORY / CREDITS, fix some C warnings
  Make uptime type consitent to fix boot time error. (giampaolo#1225)
  fix giampaolo#1463: cpu_distribution.py script is broken
  update issue template
  issue giampaolo#1404 / linux / phys CPUs count
  Big docfix (giampaolo#1464)
  Big docfix (giampaolo#1464)
  Make tests invariant to LANG setting (giampaolo#1462)
  test runner: show errors on KeyboardInterrupt
  test runner refactoring (avoid code duplication)
  Coloured tests (giampaolo#1459)
  pre-release
  [Windows] calculate USS memory by using NtQueryVirtualMemory (giampaolo#1453)
  fix version highlighting in docs/index (giampaolo#1455)
  fix version highlighting in README (giampaolo#1454)
  run win specific tests twice as fast
  test refactoring
  test: avoid failing at import time
  mention how to run tests in INSTALL guide
  giampaolo#1448: fix Wine support due to missing rtlIpv6AddressToStringA
  update HISTORY
  Fix giampaolo#1329: [AIX] disable some functions based on availability in libperfstat (giampaolo#1349)
  bump up version, fix some doc issues
  fix ResourceWarning
  pre-release
  fix giampaolo#1447: we weren't use @wrap_exceptions around oneshot() (doh\!)
  update doc + change git hook location
  update doc
  make pre-release checks/install src dist in a venv
  add new make command to check tar.gz sanity
  move doc; rephrase it a bit
  add issue templates for 'bug' and 'enhancement' types
  remove issue template commited by accident
  Update issue templates
  giampaolo#1291: (BACKWARD-INCOMPATIBLE) remove memory_maps() on OSX
  Restore Win-7 support on GIT master (5.5.1 was OK) (giampaolo#1446)
  try to fix ntext.h
  restore previous def
  fix compiler warning
  fix compiler warning
  fix compiler warning
  fix compiler warning
  take defs from PH
  set proper  SYSTEM_PROCESS_INFORMATION struct from PH
  fix compilation warnings
  giampaolo#1398 / win / cmdline: call NtQueryInformationProcess twice, the first time to get the right buf size (ProcessHacker does this)
  update doc
  better print formatting for print scripts
  fix giampaolo#1442: use python3 as Makefile default
  appveyor: run print scripts after tests
  highlight top 6 slowest calls
  add printerr() and exit() to shared utils module
  add arg parser for ad script
  introduce a new scriptsutils.py private module shared by all internal utils + refactor print_access_speed.py script
  giampaolo#1291 / OSX: mark memory_maps() as deprecated and make it alwats raise AccessDenied
  OSX memory_maps() - add error handling
  add script for to benchmark API calls
  move access_denied script
  _assert_alive() refactor (linux)
  refactor
  fix NetBSD: Process.connections() may return incomplete results if using oneshot() giampaolo#1439
  add win tests related to send_signal(CTRL_C_EVENT) giampaolo#1227
  fix win test
  fix win tests
  fix backslash warnings
  refactor README a bit
  #fix 1438: do not return any parent() for PID 0 + update doc
  ...
@abditag2
Copy link

abditag2 commented Mar 18, 2020

@giampaolo I am still seeing this problem with psutil==5.7.0. Is this fixed?

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/psutil/_common.py", line 340, in wrapper
    ret = self._cache[fun]
AttributeError: _cache

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/site-packages/petastorm/workers_pool/process_pool.py", line 327, in 
    main_process_alive = any([process.pid for process in process_iter() if process.pid == 
  File "/usr/lib/python3.6/site-packages/petastorm/workers_pool/process_pool.py", line 327, in <listcomp>
    main_process_alive = any([process.pid for process in process_iter() if process.pid == 
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 1569, in process_iter
    if proc.is_running():
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 695, in is_running
    return self == Process(self.pid)
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 447, in __init__
    self._init(pid)
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 474, in _init
    self.create_time()
  File "/usr/lib/python3.6/site-packages/psutil/__init__.py", line 824, in create_time
    self._create_time = self._proc.create_time()
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1517, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1723, in create_time
    ctime = float(self._parse_stat_file()['create_time'])
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1517, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/psutil/_common.py", line 343, in wrapper
    return fun(self)
  File "/usr/lib/python3.6/site-packages/psutil/_pslinux.py", line 1559, in _parse_stat_file
    with open_binary("%s/%s/stat" % (self._procfs_path, self.pid)) as f:
  File "/usr/lib/python3.6/site-packages/psutil/_common.py", line 604, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/110/stat'

@jamilraichouni
Copy link

jamilraichouni commented Jun 18, 2020

Same here with v5.7.0 of psutil

I call this function from within a Shell script:

def alive():
    '''
        Sub command to check, if VSCode is running.

        If VSCode is running the function will print 1 otherwise 0.
        The printed value can be understood in Shell scripts calling

        .. code-block:: bash

            python vscode.py alive

        Returns
        -------
        bool
            True if VSCode is running, else False

    '''
    if 'darwin' in PLATFORM_LOWER:
        vscode_exe_name_lw = 'visual studio code'
    elif 'windows' in PLATFORM_LOWER:
        vscode_exe_name_lw = 'Code.exe'
    elif 'linux' in PLATFORM_LOWER:
        vscode_exe_name_lw = 'code'
    for proc in psutil.process_iter(attrs=('exe',)):
        try:
            exe = proc.info['exe']
        except (psutil.AccessDenied, psutil.NoSuchProcess):
            exe = None
        if exe is not None and vscode_exe_name_lw in exe.lower():
            print(1)
            return True
    print(0)
    return False

and get a wrong result including exceptions:

$ bash /Users/USER/tmp/test.sh
Traceback (most recent call last):
  File "/Users/USER/repos/vscode/vscode.py", line 2358, in <module>
    cli()
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/USER/repos/vscode/vscode.py", line 1506, in alive
    for proc in psutil.process_iter(attrs=('exe',)):
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 1457, in process_iter
    yield add(pid)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 1434, in add
    proc.info = proc.as_dict(attrs=attrs, ad_value=ad_value)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 538, in as_dict
    ret = meth()
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/__init__.py", line 673, in exe
    exe = self._proc.exe()
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/_psosx.py", line 334, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/USER/conda_envs/jar/lib/python3.7/site-packages/psutil/_psosx.py", line 413, in exe
    return cext.proc_exe(self.pid)
FileNotFoundError: [Errno 2] No such file or directory (originated from proc_pidpath())
/Users/USER/tmp/test.sh: line 4: [: ==: unary operator expected
Not alive

The exception occurs in the code line

for proc in psutil.process_iter(attrs=('exe',)):

@giampaolo
Copy link
Owner

giampaolo commented Jun 18, 2020

It's fixed on master but there's no release yet.

@jamilraichouni
Copy link

jamilraichouni commented Jun 18, 2020

Okay, thanks! It's weird, because when I run the exact same function code in Jupyter with the same conda env all runs through.

My function "alive" from above is a sub command of a command line tool where I use click for the argument processing.

@wookayin
Copy link

wookayin commented Jan 13, 2021

Is it fixed? @zhudd-hub has reported in wookayin/gpustat#95 that this issue still happens with psutil 5.8.0. A FileNotFoundError was thrown, not being translated. Please find a stacktrace in the linked issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants