Faster Process.children(recursive=True) #1186

pitrou · 2017-12-01T16:12:51Z

Before:

$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))"
10 loops, best of 3: 29.6 msec per loop

After:

$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))"
100 loops, best of 3: 12.4 msec per loop

For reference, non-recursive:

$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children())"
100 loops, best of 3: 12.2 msec per loop

Issue #1183.

Before: ``` $ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))" 10 loops, best of 3: 29.6 msec per loop ``` After: ``` $ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))" 100 loops, best of 3: 12.4 msec per loop ``` For reference, non-recursive: ``` $ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children())" 100 loops, best of 3: 12.2 msec per loop ```

pitrou · 2017-12-01T16:17:11Z

@giampaolo

pitrou · 2017-12-01T16:24:30Z

Basically the idea is that instead of calling Process for all processes on the system and then selecting those that are candidate (grand)children of self, we prune the search space and only call Process for candidate (grand)children.

pitrou · 2017-12-01T16:25:37Z

(there seems to be an unrelated failure on AppVeyor)

pitrou · 2017-12-01T16:26:45Z

psutil/__init__.py

+            while stack:
+                pid = stack.pop()
+                if pid in seen:
+                    # Since pids can be reused while the ppid_map is


Note I'm not sure this check is necessary, as the create_time test may be sufficient to avoid such potential cycles.

giampaolo · 2017-12-01T16:35:45Z

Thanks, this is awesome. I'll try to take a look at this over the week end.

giampaolo · 2017-12-02T09:07:00Z

Looks good. Merging.

@wiggin15

* 'pslisten' of github.com:nlevitt/psutil: (922 commits) Update INSTALL.rst Pass python_requires argument to setuptools (giampaolo#1208) giampaolo#1152: fix doc to mention CLI command necessary to enable disk_io_counters() on win pre release pre release pre release pre-release fix giampaolo#1201: document that timeout kwarg is expressed in seconds Add mount points to disk_partitions() in Windows (giampaolo#775) (giampaolo#1192) add test for cpu_affinity what a stupid bug! (giampaolo#1190) update doc pre release pre-release; also get rid of PSUTIL_DEBUG doc instructions (it's kinda useless for the user after all) Use FutureWarning instead of DeprecationWarning (giampaolo#1188) fix test refactor environ() test Fix OSX pid 0 bug (giampaolo#1187) change assert in test refactor Process.__repr__ Faster Process.children(recursive=True) (giampaolo#1186) Speedup Process.children() (giampaolo#1185) update doc update HISTORY fix giampaolo#1179 / linux / cmdline: handle processes erroneously overwriting /proc/pid/cmdline by using spaces instead of null bytes as args separator set x bit to test_aix.py fix giampaolo#1181: raise AD if task_for_pid() fails with 5 and errno == ENOENT fix posix failure Arguments for NoSuchProcess and AccessDenied for the C ext (giampaolo#1180) fix travis failure https://travis-ci.org/giampaolo/psutil/jobs/306424509 be smarter in searching python exe do not test platf specific modules on wheelhouse try to fix travis failure fix travis failures try to use PYTHON_EXE instead of sys.executable giampaolo#1177: give credits to @wiggin15 OSX: implement sensors_battery (giampaolo#1177) improve error msg for old windows systems giampaolo#811 add debug messages do not mention apt-get as method of installation as it's not recommended syntax highlight in doc files syntax highlight in doc files fix doc indentation 1173 debug mode (giampaolo#1176) code style update MANIFEST giampaolo#1174: use TimeoutExpired in wait_pid() sort imports by name Move exceptions to separate file (giampaolo#1174) appveyor: enable python warnings when running tests refactor winmake.py use a C global variable to figure out whether we're in testing mode fix unicode err define a setup() function which is called on import by all C modules move PyUnicode compt fun definition up in the file rename C func re-enable test on appveyor; remove unused C code refactor PSUTIL_TESTING C APIs inspect PSUTIL_TESTING env var from C again giampaolo#1152: (DeviceIOControl), skip disk on ERROR_INVALID_FUNCTION and ERROR_NOT_SUPPORTED giampaolo#1152 / win / disk_io_counters(): DeviceIOControl errors were ignored; che return value and retry call on ERROR_INSUFFICIENT_BUFFER upgrade dist cmds change make cmds disable IPv6 tests if IPv6 is not supported travis / OSX: run py 3.6 instead of 3.4 fix giampaolo#1169: (Linux) users() hostname returns username instead update README, bump up version get rid of PSUTIL_TESTING env var: it must be necessarily set from cmdline, hence 'python -m psutil.tests' won't work out of the box try to set PSUTIL_TESTING env var from python before failing skip cpu_freq tests if not available (giampaolo#1170) update doc pre-release giampaolo#1053: drop python 3.3 support try to fix appveyor failure; also refactor generate_manifest.py giampaolo#1167 give CREDITS to @matray Including non-unicast packets in packet count calculation (giampaolo#1167) fix giampaolo#1166 (doc mistake) provide a 'make help' command ifconfig.py humanize bytes try to limit false positives on appveyor/windows reap_children() in a finally block in order to limit false positives unicode tests: use different name for test dir fix failure on osx/travis update Makefile fix test giampaolo#1164 give CREDITS to @wiggin15 AIX: implement num_ctx_switches (giampaolo#1164) use new PYTHON_EXE improve logic to determine python exe location add DEVNOTES file; move TODO.aix into _psutil_aix.c Fix test_emulate_energy_full_not_avail (giampaolo#1163) update README try to limit occasional appveyor failure Remove trove classifiers for untested and unsupported platforms (giampaolo#1162) Fix giampaolo#1154: remove 'threads' method on older AIX (giampaolo#1156) give CREDITS to @adpag for giampaolo#1159, giampaolo#1160 and giampaolo#1161 Fix test asserts due to leftover test subprocesses (giampaolo#1161) Fix network tests for newer ifconfig versions. (giampaolo#1160) Fix pre-commit hook for python 3.x. (giampaolo#1159) revert last commit ...

pitrou commented Dec 1, 2017

View reviewed changes

giampaolo merged commit f1d94cc into giampaolo:master Dec 2, 2017

giampaolo mentioned this pull request Dec 2, 2017

Process.children() is slow #1183

Closed

pitrou deleted the faster_children_recursive branch December 3, 2017 09:53

giampaolo added the performance label Nov 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Faster Process.children(recursive=True) #1186

Faster Process.children(recursive=True) #1186

pitrou commented Dec 1, 2017 •

edited

Loading

pitrou commented Dec 1, 2017

pitrou commented Dec 1, 2017

pitrou commented Dec 1, 2017

pitrou Dec 1, 2017

giampaolo commented Dec 1, 2017

giampaolo commented Dec 2, 2017

Faster Process.children(recursive=True) #1186

Faster Process.children(recursive=True) #1186

Conversation

pitrou commented Dec 1, 2017 • edited Loading

pitrou commented Dec 1, 2017

pitrou commented Dec 1, 2017

pitrou commented Dec 1, 2017

pitrou Dec 1, 2017

Choose a reason for hiding this comment

giampaolo commented Dec 1, 2017

giampaolo commented Dec 2, 2017

pitrou commented Dec 1, 2017 •

edited

Loading