[Linux] process_iter ridiculously slow (like 240x slow than ps)

[petersilva]

Platform

• { ubuntu 16.04 , 18.04 }
• { psutil version: 5.4.2 }
• { python version: 3.6.9 }

Bug description

python psutil.process_iter runs 240 times slow than ps.
it's not usable.

sarra@ddsr1:/tmp$ time ps eaux | wc -l
1632

real	0m0.276s
user	0m0.139s
sys	0m0.131s
sarra@ddsr1:/tmp$
sarra@ddsr1:/tmp$ time python3 p.py

real	1m15.800s
user	0m28.375s
sys	0m38.346s
sarra@ddsr1:/tmp$ cat p.py

import psutil
for proc in psutil.process_iter( ['pid','cmdline','name', 'username' ] ):
     p = proc.as_dict()

sarra@ddsr1:/tmp$ 

**Test results**
{ output of `python -c psutil.tests` (failures only, not full result) }

[petersilva]

oh, and I tried selecting fields... it made no difference.

[petersilva]

the following is also an order of magnitude faster:

import psutil
import pwd
import os.path


for pid in psutil.pids(): 
            p={}
            p[ 'pid' ] = pid

            pidir= '/proc/%d' % pid 
            if not os.path.exists( pidir ):
               continue
            with open( pidir + '/cmdline', 'r' ) as cf:
                 p[ 'cmdline' ] = cf.read().split('\x00')
            with open( pidir + '/loginuid', 'r' ) as luf:
                 uid=luf.read()
            p[ 'name' ] = p['cmdline'][0] 
            print('.', end='' )           

It looks like psutil gathers many data structures that the user is not asking for, and it is very
expensive to get them, resulting in many hundred fold performance penalty.

[giampaolo]

You're using it wrong. It should be:

import psutil
for proc in psutil.process_iter( ['pid','cmdline','name', 'username' ] ):
-      p = proc.as_dict()
+      proc.info

[petersilva]

Thanks!

[Scylla2020]

@giampaolo I already had it like that in my case on windows and its still super slow, getting 6 attributes takes about 12 seconds for 180 processes. Any other ideas? The slowness is in the psutil.process_iter step. Im curious how other tools like Process hacker can handle many fields and thousands of processes super fast,