[Linux] Process.num_fds() always 1 more than expected

[jnrbsn]

Platform

• OS version: Debian 10 / Linux 4.19.76 (inside a docker container on macOS host)
• psutil version: 5.7.0
• python version: 3.8.2

Bug description

The result of Process.num_fds() is always 1 more than expected on Debian 10. I haven't tested on other Linux distros. It gives the expected result on macOS.

Test script:

import os
import psutil

proc = psutil.Process()

def print_open_fds():
    os.system(f'lsof -a -d0-65535 -p {proc.pid}')
    print('')
    if os.path.exists(f'/proc/{proc.pid}/fd'):
        os.system(f'ls -al /proc/{proc.pid}/fd')
        print('')
    print(proc.num_fds())
    print('')

reg_open_file = open('foo.txt', 'wb')
devnull_fd = os.open(os.devnull, os.O_RDONLY)

print_open_fds()

reg_open_file.close()
os.close(devnull_fd)

print_open_fds()

Output on Debian 10:

COMMAND PID   USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
python   10 docker    0u   CHR  136,0      0t0        3 /dev/pts/0
python   10 docker    1u   CHR  136,0      0t0        3 /dev/pts/0
python   10 docker    2u   CHR  136,0      0t0        3 /dev/pts/0
python   10 docker    3w   REG   0,87        0 20706825 /path/to/foo.txt
python   10 docker    4r   CHR    1,3      0t0    38969 /dev/null

total 0
dr-x------ 2 docker docker  0 Jul 14 14:32 .
dr-xr-xr-x 9 docker docker  0 Jul 14 14:32 ..
lrwx------ 1 docker docker 64 Jul 14 14:32 0 -> /dev/pts/0
lrwx------ 1 docker docker 64 Jul 14 14:32 1 -> /dev/pts/0
lrwx------ 1 docker docker 64 Jul 14 14:32 2 -> /dev/pts/0
l-wx------ 1 docker docker 64 Jul 14 14:32 3 -> /path/to/foo.txt
lr-x------ 1 docker docker 64 Jul 14 14:32 4 -> /dev/null

6

COMMAND PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python   10 docker    0u   CHR  136,0      0t0    3 /dev/pts/0
python   10 docker    1u   CHR  136,0      0t0    3 /dev/pts/0
python   10 docker    2u   CHR  136,0      0t0    3 /dev/pts/0

total 0
dr-x------ 2 docker docker  0 Jul 14 14:32 .
dr-xr-xr-x 9 docker docker  0 Jul 14 14:32 ..
lrwx------ 1 docker docker 64 Jul 14 14:32 0 -> /dev/pts/0
lrwx------ 1 docker docker 64 Jul 14 14:32 1 -> /dev/pts/0
lrwx------ 1 docker docker 64 Jul 14 14:32 2 -> /dev/pts/0

4

Output on macOS:

COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
python  11312 jonathan    0u   CHR   16,1 0t253324      651 /dev/ttys001
python  11312 jonathan    1u   CHR   16,1 0t253324      651 /dev/ttys001
python  11312 jonathan    2u   CHR   16,1 0t253324      651 /dev/ttys001
python  11312 jonathan    3w   REG    1,5        0 20706825 /path/to/foo.txt
python  11312 jonathan    4r   CHR    3,2      0t0      311 /dev/null

5

COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python  11312 jonathan    0u   CHR   16,1 0t253796  651 /dev/ttys001
python  11312 jonathan    1u   CHR   16,1 0t253796  651 /dev/ttys001
python  11312 jonathan    2u   CHR   16,1 0t253796  651 /dev/ttys001

3

[jnrbsn]

I think this is because num_fds() on Linux just uses os.listdir() on /proc/{pid}/fd, but os.listdir() actually opens a new file descriptor to read the /proc/{pid}/fd directory.

[giampaolo]

Yep, I agree that's the reason (and therefore it happens with os.getpid() only). I don't think there's nothing to fix here.

[jnrbsn]

it happens with os.getpid() only

I'm not sure I understand what you mean. It happens because os.listdir() opens a new file descriptor.

I don't think there's nothing to fix here.

Did you mean to use a double negative in this sentence? I'm not sure what you're trying to say.

---

This is certainly possible to fix. One way to fix it, for example, would be to use os.scandir() (although, that's only available on Python 3.5+), and then filter out the directory while it's open. Something like this:

    def num_fds(self):
        fd_dir = "%s/%s/fd" % (self._procfs_path, self.pid)
        num_fds = 0
        for entry in os.scandir(fd_dir):
            if os.readlink(entry.path) != fd_dir:
                num_fds += 1
        return num_fds

[giampaolo]

it happens with os.getpid() only
I'm not sure I understand what you mean.

I mean this problem occurs only with psutil.Process(os.getpid()). In order to count fds, the current process creates one fd itself.
The same thing happens on Windows with num_handles(): in order to count process handles, it creates one itself.
A similar thing can be said for memory (in order to calculate current process memory, you inevitably consume some additional memory), context switches and probably others.

[giampaolo]

Closing as "won't fix".