[Linux] Process.open_files() returns an empty list if a process is running inside a container

[NitroCao]

Summary

• OS: CentOS Linux release 7.2.1511 (Core).
• Kernel version: 3.10.0-693.21.1.el7.x86_64
• Architecture: 64bit
• Psutil version: 5.9.1
• Python version: 3.8.13
• Type: core

Description

The key point is isfile_strict() function in open_files() (L2209):

psutil/psutil/_pslinux.py

Lines 2183 to 2228 in 3795983

	def open_files(self):
	retlist = []
	files = os.listdir("%s/%s/fd" % (self._procfs_path, self.pid))
	hit_enoent = False
	for fd in files:
	file = "%s/%s/fd/%s" % (self._procfs_path, self.pid, fd)
	try:
	path = readlink(file)
	except (FileNotFoundError, ProcessLookupError):
	# ENOENT == file which is gone in the meantime
	hit_enoent = True
	continue
	except OSError as err:
	if err.errno == errno.EINVAL:
	# not a link
	continue
	if err.errno == errno.ENAMETOOLONG:
	# file name too long
	debug(err)
	continue
	raise
	else:
	# If path is not an absolute there's no way to tell
	# whether it's a regular file or not, so we skip it.
	# A regular file is always supposed to be have an
	# absolute path though.
	if path.startswith('/') and isfile_strict(path):
	# Get file position and flags.
	file = "%s/%s/fdinfo/%s" % (
	self._procfs_path, self.pid, fd)
	try:
	with open_binary(file) as f:
	pos = int(f.readline().split()[1])
	flags = int(f.readline().split()[1], 8)
	except FileNotFoundError:
	# fd gone in the meantime; process may
	# still be alive
	hit_enoent = True
	else:
	mode = file_flags_to_mode(flags)
	ntuple = popenfile(
	path, int(fd), int(pos), mode, flags)
	retlist.append(ntuple)
	if hit_enoent:
	self._assert_alive()
	return retlist

The definition of isfile_strict() function:

psutil/psutil/_common.py

Lines 475 to 487 in 3795983

	def isfile_strict(path):
	"""Same as os.path.isfile() but does not swallow EACCES / EPERM
	exceptions, see:
	http://mail.python.org/pipermail/python-dev/2012-June/120787.html
	"""
	try:
	st = os.stat(path)
	except OSError as err:
	if err.errno in (errno.EPERM, errno.EACCES):
	raise
	return False
	else:
	return stat.S_ISREG(st.st_mode)

The implemention of isfile_strict() decides a file won't be appended into the result of open_files() if the file doesn't exist, but for container processes, we cannot just use the value returned by os.readlink() function to check the existence of the file because of the mount namespace.

Solutions

One of the solution is we can use /proc/[pid]/root/ + [value of readlink] to check the existence of a file. The doc of /proc/[pid]/root is here https://man7.org/linux/man-pages/man5/proc.5.html. But it may not be a good idea to treat it as the return value of open_files().
In order to dereference /proc/[pid]/root, we need to find the actual rootfs of a process. It seems to be no better solution to finish it, but there's still a workaround by the third field in /proc/[pid]/mountinfo. For example, the following is the content of /proc/[pid]/mountinfo of a container process:

539 458 0:49 / / rw,relatime master:176 - overlay overlay rw,lowerdir=/data/docker/overlay2/l/TJ4XAVTH7AD3UFAT7LCQI6FJ5F:/data/docker/overlay2/l/J3O4XAMMPXGMIKIB3R2LF73V5U:/data/docker/overlay2/l/QIUBEKUBQ6Z3POD4CVRQV7N2GT:/data/docker/overlay2/l/4TERDLBXULE3337FK47A6ZX24N:/data/docker/overlay2/l/DNYYJ6I5SYNYIAG6KEHIL2GNG2:/data/docker/overlay2/l/IHX5RXV5LQESME3IGHYDLSOZ27,upperdir=/data/docker/overlay2/ad5011ed4627b5e8f1a1b2b39818c4b5753e3d25b8f863c689475973bfb35ea2/diff,workdir=/data/docker/overlay2/ad5011ed4627b5e8f1a1b2b39818c4b5753e3d25b8f863c689475973bfb35ea2/work
540 539 0:51 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
541 539 0:54 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
542 541 0:55 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
543 539 0:56 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro
544 543 0:57 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,mode=755
545 544 0:21 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/systemd ro,nosuid,nodev,noexec,relatime master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
546 544 0:23 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/cpu,cpuacct ro,nosuid,nodev,noexec,relatime master:10 - cgroup cgroup rw,cpuacct,cpu
547 544 0:24 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/cpuset ro,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,cpuset
548 544 0:25 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/pids ro,nosuid,nodev,noexec,relatime master:12 - cgroup cgroup rw,pids
549 544 0:26 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime master:13 - cgroup cgroup rw,memory
550 544 0:27 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/blkio ro,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,blkio
551 544 0:28 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/freezer ro,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,freezer
552 544 0:29 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/hugetlb ro,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,hugetlb
553 544 0:30 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/net_cls,net_prio ro,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,net_prio,net_cls
554 544 0:31 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/devices ro,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,devices
555 544 0:32 /system.slice/docker-a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a.scope /sys/fs/cgroup/perf_event ro,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,perf_event
556 541 0:50 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
557 541 0:58 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k
558 539 253:4 /docker/containers/a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/mapper/VolGroup00-data rw,data=ordered
559 539 253:4 /docker/containers/a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a/hostname /etc/hostname rw,relatime - ext4 /dev/mapper/VolGroup00-data rw,data=ordered
560 539 253:4 /docker/containers/a54f0cbe342103522d6c93c2e6bdfb7ed856f51666687f07ec5366100ec8ee5a/hosts /etc/hosts rw,relatime - ext4 /dev/mapper/VolGroup00-data rw,data=ordered
561 539 253:4 /docker/volumes/hooan_log_consumer_esdata/_data /usr/share/elasticsearch/data rw,relatime master:30 - ext4 /dev/mapper/VolGroup00-data rw,data=ordered
459 540 0:51 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw
460 540 0:51 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw
461 540 0:51 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw
462 540 0:51 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
463 540 0:51 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
464 540 0:64 / /proc/acpi ro,relatime - tmpfs tmpfs ro
465 540 0:54 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
466 540 0:54 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
477 540 0:54 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
478 540 0:54 /null /proc/timer_stats rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
479 540 0:54 /null /proc/sched_debug rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
480 540 0:65 / /proc/scsi ro,relatime - tmpfs tmpfs ro
481 543 0:66 / /sys/firmware ro,relatime - tmpfs tmpfs ro

We know the major and minor number of rootfs is 0:49. From /proc/1/mountinfo we find 0:49 corresponds to the following:

440 46 0:49 / /data/docker/overlay2/ad5011ed4627b5e8f1a1b2b39818c4b5753e3d25b8f863c689475973bfb35ea2/merged rw,relatime shared:176 - overlay overlay rw,lowerdir=/data/docker/overlay2/l/TJ4XAVTH7AD3UFAT7LCQI6FJ5F:/data/docker/overlay2/l/J3O4XAMMPXGMIKIB3R2LF73V5U:/data/docker/overlay2/l/QIUBEKUBQ6Z3POD4CVRQV7N2GT:/data/docker/overlay2/l/4TERDLBXULE3337FK47A6ZX24N:/data/docker/overlay2/l/DNYYJ6I5SYNYIAG6KEHIL2GNG2:/data/docker/overlay2/l/IHX5RXV5LQESME3IGHYDLSOZ27,upperdir=/data/docker/overlay2/ad5011ed4627b5e8f1a1b2b39818c4b5753e3d25b8f863c689475973bfb35ea2/diff,workdir=/data/docker/overlay2/ad5011ed4627b5e8f1a1b2b39818c4b5753e3d25b8f863c689475973bfb35ea2/work

[NitroCao]

BTW, is it necessary to check if the open files exist or not? I checked source code of gopsutil, OpenFiles() won't check the existence of files.
https://github.com/shirou/gopsutil/blob/3b417071a5835600ae0d654231cfb9ce226f2b6b/v3/process/process_linux.go#L608-L648