Skip to content

Oneshot ctx manager #937

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 108 commits into from
Nov 5, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
5d1e6be
1st refactoring: parse /proc/pid/status in a single method
giampaolo Apr 29, 2016
86a9560
2nd refactoring: parse /proc/pid/stat in a single method
giampaolo Apr 29, 2016
a209a26
determine process status by using common parser method
giampaolo Apr 29, 2016
7efe52c
implement oneshot() ctx manager
giampaolo Apr 30, 2016
78c2a11
merge from master
giampaolo Apr 30, 2016
cc42f33
make as_dict() use oneshot() ctx manager
giampaolo Apr 30, 2016
80060c7
speedup code for when not using oneshot()
giampaolo Apr 30, 2016
2e653c4
speedup
giampaolo Apr 30, 2016
9853683
merge from master
giampaolo Apr 30, 2016
7fc29ae
fix some issues
giampaolo May 1, 2016
c39f0ba
merge from master
giampaolo May 1, 2016
388e82f
merge
giampaolo May 1, 2016
1b600a5
update doc and apply oneshot() where necessary
giampaolo May 1, 2016
f34770d
update doc
giampaolo May 2, 2016
4c102a5
merge from master
giampaolo Jun 27, 2016
cea803d
Merge branch 'master' into oneshot
giampaolo Jul 3, 2016
8bf7ab0
merge from master
giampaolo Aug 1, 2016
d3806e3
update doc
giampaolo Aug 2, 2016
50015c4
#799: BSD: use onectx() also for proc memory info
giampaolo Aug 3, 2016
f851be9
#799: speedup @memoize_when_activated deco from +1.9x to +2.6x
giampaolo Aug 3, 2016
c88a385
#799 add benchmark script
giampaolo Aug 3, 2016
db2bb6d
enhance benchmark script
giampaolo Aug 3, 2016
852c6c2
enhance benchmark script
giampaolo Aug 3, 2016
38ac193
fix failing test
giampaolo Aug 3, 2016
8fea6e3
#799: oneshot() BSD implementation
giampaolo Aug 3, 2016
355214d
update doc
giampaolo Aug 3, 2016
16b304b
update docs
giampaolo Aug 3, 2016
7fc6992
update doc
giampaolo Aug 3, 2016
3e9657f
fix typo
giampaolo Aug 3, 2016
d2aaf8c
Merge branch 'master' into oneshot
giampaolo Aug 3, 2016
fec353c
update memoize_when_activated docstring
giampaolo Aug 5, 2016
630b40d
#799 - oneshot: solaris implementation 1.37x speedup
giampaolo Aug 5, 2016
05defa0
#857: raise OSError instead of RuntimeError in case read() syscall fails
giampaolo Aug 5, 2016
8f27cbb
merge from master
giampaolo Aug 5, 2016
39165fb
#799, oneshot(), windows: expose C functions to OpenProcess and Close…
giampaolo Aug 6, 2016
6c55785
#799 / windows / cpu_times: add oneshot() support
giampaolo Aug 6, 2016
242f522
#799 / windows / cpu_times: add memory_info() support
giampaolo Aug 6, 2016
a6150cb
fix cache
giampaolo Aug 6, 2016
ddc7943
Merge branch 'master' into oneshot-win
giampaolo Aug 6, 2016
c8c8dfb
#799 / windows / cpu_times: add nice() support
giampaolo Aug 6, 2016
614ecab
#799 / windows / cpu_times: add ionice() support
giampaolo Aug 6, 2016
146df37
#799: use timeit module for doing benchmark
giampaolo Aug 6, 2016
b8564c7
fix benchmark script showing erroneous slowdown
giampaolo Aug 6, 2016
fb7da11
from __future__ import division
giampaolo Aug 6, 2016
86b6997
Merge branch 'master' into oneshot-win
giampaolo Aug 8, 2016
9af9621
#799 / windows / cpu_times: add io_counters() and cpu_affinity() support
giampaolo Aug 8, 2016
f293d90
merge oneshot branch
giampaolo Aug 8, 2016
b6a4f66
refactor benchmark script
giampaolo Aug 8, 2016
b0c6c42
Merge branch 'master' into oneshot
giampaolo Aug 8, 2016
4ed8d31
Merge branch 'oneshot' into oneshot-win
giampaolo Aug 8, 2016
ece3a48
update oneshot() win doc
giampaolo Aug 8, 2016
d632d2a
Merge branch 'master' into oneshot
giampaolo Aug 10, 2016
143d6a2
#799 cache cpu_times, memory_info and ppid a Process class level
giampaolo Aug 13, 2016
a69b7df
#799 add uids/userma,e
giampaolo Aug 13, 2016
da2f226
fix oneshot() ctx manager which was not deactivating cache decorator
giampaolo Aug 23, 2016
8a2505e
remove make cmd committed by accident
giampaolo Aug 23, 2016
16e11c0
#799 add a second script which uses perf module, which is supposed to…
giampaolo Aug 23, 2016
d46e922
merge from master
giampaolo Sep 22, 2016
a6fd47e
Merge branch 'master' into oneshot
giampaolo Oct 2, 2016
8a63928
fix conflicts
giampaolo Oct 7, 2016
8da5114
fix Makefile
giampaolo Oct 7, 2016
7b2a6b3
#799 / OSX: implement oneshot for kinfo_proc info
giampaolo Oct 7, 2016
cf21849
#799 / OSX: implement oneshot for PROC_PIDTASKINFO
giampaolo Oct 7, 2016
1e8cef9
#799 / OSX: also include proc.name() in the list of grouped oneshot info
giampaolo Oct 7, 2016
026afd5
update doc
giampaolo Oct 8, 2016
77b484c
#799 / BSD: also include the name() in the oneshot() info; adds furth…
giampaolo Oct 8, 2016
7ee1f4e
update doc
giampaolo Oct 9, 2016
f14ba13
update doc
giampaolo Oct 9, 2016
f2e9e7a
merge from master
giampaolo Oct 12, 2016
9873308
merge from oneshot branch
giampaolo Oct 19, 2016
4b134aa
merge from master
giampaolo Oct 25, 2016
23724f6
osx: fix memory leak
giampaolo Oct 25, 2016
3c8c729
bsd: fix mem leak
giampaolo Oct 25, 2016
52f3eb1
fix netbsd/openvsd compilation failure
giampaolo Oct 25, 2016
f923f45
adjust winmake script
giampaolo Oct 18, 2016
c8814c4
upgrade perf code
giampaolo Oct 26, 2016
bd0bbdd
try to adjust perf
giampaolo Oct 26, 2016
26f925f
Merge branch 'oneshot' into oneshot-win
giampaolo Oct 27, 2016
7ad634a
adjust bench2 script to new perf API
giampaolo Oct 27, 2016
0ccd137
Merge branch 'master' into oneshot
giampaolo Oct 27, 2016
04b1568
Merge branch 'oneshot' into oneshot-win
giampaolo Oct 27, 2016
bb56e9d
Merge branch 'oneshot-win' of https://github.com/giampaolo/psutil int…
giampaolo Oct 27, 2016
861bf2f
Merge branch 'master' into oneshot
giampaolo Oct 27, 2016
f2b8bb3
Merge branch 'master' into oneshot-win
giampaolo Oct 27, 2016
68ba191
Merge branch 'oneshot-win' of https://github.com/giampaolo/psutil int…
giampaolo Oct 27, 2016
ae7ceba
Merge branch 'oneshot' into oneshot-win
giampaolo Oct 27, 2016
7f51f00
bench script: add psutil ver
giampaolo Oct 27, 2016
c373045
update doc
giampaolo Oct 27, 2016
ed64f4a
Merge branch 'master' into oneshot
giampaolo Oct 27, 2016
0f98202
Merge branch 'oneshot' into oneshot-win
giampaolo Oct 27, 2016
0e760f6
update windmake script
giampaolo Oct 27, 2016
1a314d3
Merge branch 'oneshot' into oneshot-win
giampaolo Oct 27, 2016
677060e
Merge branch 'oneshot-win' of https://github.com/giampaolo/psutil int…
giampaolo Oct 27, 2016
a63974f
merge from master
giampaolo Oct 28, 2016
7818a81
Merge branch 'oneshot' into oneshot-win
giampaolo Oct 28, 2016
3efb6bf
#799 / win: use oneshot() around num_threads() and num_ctx_switches()…
giampaolo Oct 27, 2016
4a06a54
#799 / win: pass handle also to memory_maps() and username() functions
giampaolo Oct 28, 2016
d4a0717
799 onshot / win: no longer store the handle in python; I am now sure…
giampaolo Oct 28, 2016
c572c9d
update doc
giampaolo Oct 28, 2016
80c754f
Merge branch 'master' into oneshot
giampaolo Oct 28, 2016
d262c71
Merge branch 'master' into oneshot
giampaolo Oct 29, 2016
2bddd7b
Merge branch 'master' into oneshot
giampaolo Nov 4, 2016
4376fdf
Merge branch 'master' into oneshot
giampaolo Nov 5, 2016
5274e8a
#799 - oneshot / linux: speedup memory_full_info and memory_maps
giampaolo Nov 5, 2016
e2cacda
speedup fetch all process test by using oneshot
giampaolo Nov 5, 2016
cc81692
add simple test case for oneshot() ctx manager
giampaolo Nov 5, 2016
cd8d3d7
add simple test case for oneshot() ctx manager
giampaolo Nov 5, 2016
c123874
update version in doc
giampaolo Nov 5, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ DEPS = argparse \
ipaddress \
mock==1.0.1 \
pep8 \
perf \
pyflakes \
requests \
setuptools \
Expand Down Expand Up @@ -240,3 +241,11 @@ print-announce:

grep-todos:
git grep -EIn "TODO|FIXME|XXX"

# run script which benchmarks oneshot() ctx manager (see #799)
bench-oneshot: install
$(PYTHON) scripts/internal/bench_oneshot.py

# same as above but using perf module (supposed to be more precise)
bench-oneshot-2: install
$(PYTHON) scripts/internal/bench_oneshot_2.py
88 changes: 87 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,12 @@ Process class
and creation time). As such it can also be used with
`set()s <http://docs.python.org/2/library/stdtypes.html#types-set>`__.

.. note::

In order to efficiently fetch more than one information about the process
at the same time, make sure to use either :meth:`as_dict` or
:meth:`oneshot` context manager.

.. warning::

the way this class is bound to a process is via its **PID**.
Expand All @@ -760,9 +766,84 @@ Process class
:meth:`is_running()` before querying the process or use
:func:`process_iter()` in case you're iterating over all processes.

.. method:: oneshot()

Utility context manager which considerably speeds up the retrieval of
multiple process information at the same time.
Internally different process info (e.g. :meth:`name`, :meth:`ppid`,
:meth:`uids`, :meth:`create_time`, ...) may be fetched by using the same
routine, but only one data is returned and the others are discarded.
When using this context manager the internal routine is executed once (in
the example below on :meth:`name()`) and the other info are cached.
The subsequent calls sharing the same internal routine will return the
cached value.
The cache is cleared when exiting the context manager block.
The advice is to use this every time you retrieve more than one information
about the process. If you're lucky, you'll get a hell of a speedup.
Example:

>>> import psutil
>>> p = psutil.Process()
>>> with p.oneshot():
... p.name() # execute internal routine once collecting multiple info
... p.cpu_times() # return cached value
... p.cpu_percent() # return cached value
... p.create_time() # return cached value
... p.ppid() # return cached value
... p.status() # return cached value
...
>>>

Here's a list of methods which can take advantage of the speedup depending
on what platform you're on.
In the table below horizontal emtpy rows indicate what process methods can
be efficiently grouped together internally.
The last column (speedup) shows an approximation of the speedup you can get
if you call all the methods together (best case scenario).

+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| Linux | Windows | OSX | BSD | SunOS |
+==============================+===============================+==============================+==============================+==========================+
| :meth:`~Process.cpu_percent` | :meth:`~Process.cpu_percent` | :meth:`~Process.cpu_percent` | :meth:`~Process.cpu_percent` | :meth:`name` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`~Process.cpu_times` | :meth:`~Process.cpu_times` | :meth:`~Process.cpu_times` | :meth:`~Process.cpu_times` | :meth:`cmdline` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`create_time` | :meth:`io_counters()` | :meth:`memory_info` | :meth:`create_time` | :meth:`create_time` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`name` | :meth:`ionice` | :meth:`memory_percent` | :meth:`gids` | |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`ppid` | :meth:`memory_info` | :meth:`num_ctx_switches` | :meth:`io_counters` | :meth:`memory_info` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`status` | :meth:`nice` | :meth:`num_threads` | :meth:`name` | :meth:`memory_percent` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`terminal` | :meth:`memory_maps` | | :meth:`memory_info` | :meth:`nice` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| | :meth:`num_ctx_switches` | :meth:`create_time` | :meth:`memory_percent` | :meth:`num_threads` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`gids` | :meth:`num_handles` | :meth:`gids` | :meth:`num_ctx_switches` | :meth:`ppid` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`num_ctx_switches` | :meth:`num_threads` | :meth:`name` | :meth:`ppid` | :meth:`status` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`num_threads` | :meth:`username` | :meth:`ppid` | :meth:`status` | :meth:`terminal` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`uids` | | :meth:`status` | :meth:`terminal` | |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`username` | | :meth:`terminal` | :meth:`uids` | :meth:`gids` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| | | :meth:`uids` | :meth:`username` | :meth:`uids` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`memory_full_info` | | :meth:`username` | | :meth:`username` |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| :meth:`memory_maps` | | | | |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+
| *speedup: +2.5x* | *speedup: +1.8x / +6.5x* | *speedup: +1.9x* | *speedup: +2.0x* | *speedup: +1.3x* |
+------------------------------+-------------------------------+------------------------------+------------------------------+--------------------------+

.. versionadded:: 5.0.0

.. attribute:: pid

The process PID.
The process PID. This is the only (read-only) attribute of the class.

.. method:: ppid()

Expand Down Expand Up @@ -815,6 +896,8 @@ Process class
value which gets assigned to a dict key in case :class:`AccessDenied`
or :class:`ZombieProcess` exception is raised when retrieving that
particular process information.
Internally, :meth:`as_dict` uses :meth:`oneshot` context manager so
there's no need you use it also.

>>> import psutil
>>> p = psutil.Process()
Expand All @@ -825,6 +908,9 @@ Process class
3.0.0 *ad_value* is used also when incurring into
:class:`ZombieProcess` exception, not only :class:`AccessDenied`

.. versionchanged:: 4.5.0 :meth:`as_dict` is considerably faster thanks
to :meth:`oneshot` context manager.

.. method:: parent()

Utility method which returns the parent process as a :class:`Process`
Expand Down
Loading