ENH added optional ``tracemalloc`` backend by demiurg906 · Pull Request #117 · pythonprofilers/memory_profiler

demiurg906 · 2016-07-12T14:08:43Z

It is now possible to use tracemalloc to analyze memory usage Python
code on Python 3.4 and above. tracemalloc allows for more precise
measurements compared to psutil. However, it only works either for
pure Python code or for C extensions allocating memory via PyMem_Alloc.

To use the new backend code run memory_profiler with --backend
option, e.g.

$ python -m memory_profiler --backend=tracemalloc script.py

Also if you use memory_profiler with imported decorator you can specify
backend as an argument to the decorator function:

@profile(backend='tracemalloc')
def f(n):
    a = [0] * n
    return a

backend parameter to @profile has priority over --backend.

Note that using tracemalloc in mprof and IPython magic is not
supported at the moment.

@Profile

It is now possible to use ``tracemalloc`` to analyze memory usage Python code on Python 3.4 and above. ``tracemalloc`` allows for more precise measurements compared to ``psutil``. However, it only works either for pure Python code or for C extensions allocating memory via ``PyMem_Alloc``. To use the new backend code run ``memory_profiler`` with ``--backend`` option, e.g. $ python -m memory_profiler --backend=tracemalloc script.py Also if you use ``memory_profiler`` with imported decorator you can specify backend as an argument to the decorator function: @Profile(backend='tracemalloc') def f(n): a = [0] * n return a ``backend`` parameter to ``@profile`` has priority over ``--backend``. Note that using ``tracemalloc`` in ``mprof`` and IPython magic is not supported at the moment.

fabianp · 2016-07-13T08:19:38Z

+
+_backend_chosen = False
+_backend = 'psutil'
+


Do you need two global variables? I think you can do with only one (possibly initialized to None).

fabianp · 2016-07-13T08:21:21Z

Thanks for the contribution!!

Looks great. If I understood it correctly, the default backends by precedence are psutil -> posix (if on posix) -> tracemalloc, right? (this is perfect, just want to be sure I understood).

Also, what do you mean by all those "# TODO: add filename" ? could you remove them if they are not important?

superbobry · 2016-07-13T08:26:52Z

+            _backend = n_backend
+            break
+    if _backend == 'no_backend':
+        raise NotImplementedError('Tracemalloc or psutil module is required for non-unix '


A RuntimeError would be more appropriate here I think.

demiurg906 · 2016-07-14T10:58:59Z

Looks great. If I understood it correctly, the default backends by precedence are psutil -> posix (if on posix) -> tracemalloc, right? (this is perfect, just want to be sure I understood).

@fabianp, yes you understood correctly

Also, what do you mean by all those "# TODO: add filename" ? could you remove them if they are not important?

That TODO i added in places where necessary add some code to support tracemalloc, but i not invented now how realize that

fabianp · 2016-07-15T14:17:17Z

I don't understand your last statement. Do you mean that the support for tracemalloc is not complete?

demiurg906 · 2016-07-18T10:35:15Z

Yes, you right. tracemalloc now works only in profiling with @profile decorator with LineProfiler or TimeStamper.

Note that using tracemalloc in mprof and IPython magic is not supported at the moment.

fabianp · 2016-07-19T13:13:50Z

@demiurg906 let me know when you think this is ready to be merged

superbobry · 2016-08-04T16:16:25Z

Hi @fabianp, I wanted to quickly summarize the status of this PR.

The good part: tracemalloc produces more accurate results for pure Python code (see tests).

The not-so-good parts:

C allocations (including NumPy) aren't measured, which is a fundamental problem with tracemalloc at the moment.
It is much slower than psutil, because tracemalloc accumulates significantly more information than needed for memory_profiler.

The last issue can be partially "solved" by restarting tracing after each line of code. This brings tracemalloc up to speed with psutil, but makes it miss all deallocations, since it no longer tracks memory between lines.

Another way to approach the speed issue is to write a stripped down version of tracemalloc. Python3.4 introduced custom allocator API which is how tracemalloc does its bookkeeping. It shouldn't be very hard to do the same either in C or Cython.

I think deallocations are a must-have and therefore the only real option we have is to write a custom allocator. What do you think?

fabianp · 2016-08-09T12:50:18Z

thanks for the update @superbobry . For now I would like to avoid including any compiled code. For me, the benefit in speed would not compensate the deployment and maintenance burden.

I think the current code is good enough, and we can always improve upon. Is there something left to do?

superbobry · 2016-08-09T13:31:47Z

        self.include_children = kw.pop("include_children", False)

        # get baseline memory usage
+        # TODO: add filename


@demiurg906 is this (and the following) TODO still relevant?

demiurg906 · 2016-08-11T12:15:10Z

@fabianp, I fix all remaining comments, so I think PR is ready to merge

superbobry · 2016-08-11T13:44:21Z

-        return OrderedDict(items)
    if new_backend is not None:
-        backends = move_to_start(backends, new_backend)
+        backends.insert(0, backends.pop(backends_indices[new_backend]))


After this line backends_indices will become invalid, right?

Update: looks like backends_indices isn't used anywhere in the code.

right, backends_indices is used only in moving chosen backend

fabianp · 2016-08-13T17:08:29Z

Great work @demiurg906! .

I've added a commit on top of yours that removes the global variable _backend. Now that variable is passed along the classes to _get_memory, an approach that I prefer. The code is in PR #119 . Could you please take a look at the resulting code / try it on your problems? If it looks good to you, then we are all set!

demiurg906 added 6 commits July 12, 2016 17:05

FIX little fix for Python 3.3-

9e83dfc

FIX added a workaround for OrderedDict.move_to_end

30a1617

FIX added a choosing backend in memory_usage function

d0c8a39

FIX choosing backend in profile decorator fixed

361c751

TEST added test for tracemalloc

fc44641

fabianp reviewed Jul 13, 2016
View reviewed changes

superbobry reviewed Jul 13, 2016
View reviewed changes

FIX line length according to PEP8

e0b37a5

demiurg906 added 2 commits July 14, 2016 14:51

FIX global variable _backend_choosen removed

4565b8f

FIX string format changed to support Python 2

636b344

FIX _backend_chosen variable deleted from everywhere

4a26881

superbobry reviewed Aug 9, 2016
View reviewed changes

demiurg906 added 2 commits August 11, 2016 14:59

FIX TODO's removed

eba604c

FIX OrderedDict changed to list in choose_backend function

946aa5e

superbobry reviewed Aug 11, 2016
View reviewed changes

fabianp merged commit 946aa5e into pythonprofilers:master Aug 19, 2016

Conversation

demiurg906 commented Jul 12, 2016

Uh oh!

fabianp Jul 13, 2016

Choose a reason for hiding this comment

Uh oh!

fabianp commented Jul 13, 2016

Uh oh!

superbobry Jul 13, 2016

Choose a reason for hiding this comment

Uh oh!

demiurg906 commented Jul 14, 2016

Uh oh!

fabianp commented Jul 15, 2016

Uh oh!

demiurg906 commented Jul 18, 2016

Uh oh!

fabianp commented Jul 19, 2016

Uh oh!

superbobry commented Aug 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fabianp commented Aug 9, 2016

Uh oh!

superbobry Aug 9, 2016

Choose a reason for hiding this comment

Uh oh!

demiurg906 commented Aug 11, 2016

Uh oh!

superbobry Aug 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

demiurg906 Aug 13, 2016

Choose a reason for hiding this comment

Uh oh!

fabianp commented Aug 13, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

superbobry commented Aug 4, 2016 •

edited

Loading

superbobry Aug 11, 2016 •

edited

Loading