Skip to content

Commit

Permalink
Fixing bug in memory profiler (#704)
Browse files Browse the repository at this point in the history 
Upon trying to use memory_profiler.py to debug a problem a user reported it was broken. Indeed, it appeared to have a couple of bugs in it. Furthermore, at least in Python 3.x, the third-party tool pympler.asizeof() was so slow as to be entirely unusualble.

To this end, I have fixed a couple of bugs in memory_profiler.py and removed all references to pympler.

My guess is that the Python standard library sys.getsizeof() method will under-report memory usage a bit. BUT it does work, and it works fast, without failure. So that's still a big win.

Also, I was able to delete a bunch of unused code.
  • Loading branch information
@john-science
john-science committed Jun 9, 2022
1 parent 1ca96b6 commit 8c46f8c
Show file tree
Hide file tree
Showing 5 changed files with 55 additions and 144 deletions.
181 changes: 39 additions & 142 deletions armi/bookkeeping/memoryProfiler.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,35 +17,31 @@
There are many approaches to memory profiling.
1. You can ask psutil for the memory used by the process
from an OS perspective. This is great for top-down analysis. This module provides printouts
1. You can ask psutil for the memory used by the process from an OS perspective.
This is great for top-down analysis. This module provides printouts
that show info from every process running. This is very fast.
2. You can use ``asizeof`` (part of pympler) to measure the size of various individual objects. This will help
you pin-point your issue. But it's slow.
3. You can use ``gc.get_objects()`` to list all objects that the garbage collector is tracking. If you want, you
2. You can use ``gc.get_objects()`` to list all objects that the garbage collector is tracking. If you want, you
can filter it down and get the counts and sizes of objects of interest (e.g. all armi objects).
This module has tools to do all of this. It should help you out.
Note that if psutil is reporting way more memory usage than the asizeof reports, then you probably are dealing
with a garbage collection queue. If you free a large number of objects, call `gc.collect()` to force a
garbage collection and the psutil process memory should fall by a lot.
Also, it seems that even if your garbage is collected, Windows does not de-allocate all the memory. So if
you are a worker and you just got a 1.6GB reactor but then deleted it, Windows will keep you at 1.6GB for a while.
NOTE: Psutil and sys.getsizeof will certainly report slightly different results.
NOTE: In Windows, it seems that even if your garbage is collected, Windows does not de-allocate all the memory.
So if you are a worker and you just got a 2GB reactor but then deleted it, Windows will keep you at 2GB for a while.
See Also:
http://packages.python.org/Pympler/index.html
https://pythonhosted.org/psutil/
https://docs.python.org/2/library/gc.html#gc.garbage
https://docs.python.org/3/library/gc.html#gc.garbage
"""
from typing import Optional
import copy
import gc
import logging
import sys
import tabulate
from typing import Optional

from armi import context
from armi import interfaces
Expand All @@ -65,15 +61,8 @@
_havePsutil = False


# disable the import warnings (Issue #88)
logging.disable(logging.CRITICAL)
from pympler.asizeof import asizeof

# This is necessary to reset pympler's changes to the logging configuration
logging.disable(logging.NOTSET)


ORDER = interfaces.STACK_ORDER.POSTPROCESSING
REPORT_COUNT = 100000


def describeInterfaces(cs):
Expand Down Expand Up @@ -123,8 +112,6 @@ def displayMemoryUsage(self, timeDescription):
r"""
Print out some information to stdout about the memory usage of ARMI.
Makes use of the asizeof utility.
Useful when the debugMem setting is set to True.
Turn these on as appropriate to find all your problems.
Expand All @@ -143,23 +130,22 @@ def displayMemoryUsage(self, timeDescription):
def _reactorAssemblyTrackingBreakdown(self):
runLog.important("Reactor attribute ArmiObject tracking count")
for attrName, attrObj in self.r.core.__dict__.items():
if (
isinstance(attrObj, list)
and attrObj
and isinstance(attrObj[0], ArmiObject)
):
if not attrObj:
continue

if isinstance(attrObj, list) and isinstance(attrObj[0], ArmiObject):
runLog.important(
"List {:30s} has {:4d} assemblies".format(attrName, len(attrObj))
"List {:30s} has {:4d} ArmiObjects".format(attrName, len(attrObj))
)
if (
isinstance(attrObj, dict)
and attrObj
and isinstance(list(attrObj.values())[0], ArmiObject)

if isinstance(attrObj, dict) and isinstance(
list(attrObj.values())[0], ArmiObject
):
runLog.important(
"Dict {:30s} has {:4d} assemblies".format(attrName, len(attrObj))
"Dict {:30s} has {:4d} ArmiObjects".format(attrName, len(attrObj))
)
runLog.important("SFP has {:4d} assemblies".format(len(self.r.core.sfp)))

runLog.important("SFP has {:4d} ArmiObjects".format(len(self.r.core.sfp)))

def checkForDuplicateObjectsOnArmiModel(self, attrName, refObject):
"""Scans thorugh ARMI model for duplicate objects"""
Expand Down Expand Up @@ -210,14 +196,12 @@ def _printFullMemoryBreakdown(
"""
looks for any class from any module in the garbage collector and prints their count and size
Very powerful. Also very slow if you reportSize
Parameters
----------
startsWith : str, optional
limit to objects with classes that start with a certain string
reportSize : bool, optional
calculate size as well as counting individual objects. SLLOOOWW.
calculate size as well as counting individual objects.
Notes
-----
Expand All @@ -229,16 +213,15 @@ def _printFullMemoryBreakdown(
reactor = self.r

if reportSize:
self.r.detach()
self.o.detach()

gc.collect()
allObjects = gc.get_objects()
instanceCounters = KlassCounter(reportSize)

runLog.info("GC returned {} objects".format(len(allObjects)))

instanceCounters = KlassCounter(reportSize)
instanceCounters.countObjects(allObjects)

for counter in sorted(instanceCounters.counters.values()):
runLog.info(
"UNIQUE_INSTANCE_COUNT: {:60s} {:10d} {:10.1f} MB".format(
Expand Down Expand Up @@ -273,6 +256,13 @@ def getReferrers(obj):


class KlassCounter:
"""
Helper class, to allow us to count instances of various classes in the
Python standard library garbage collector (gc).
Counting can be done simply, or by memory footprint.
"""

def __init__(self, reportSize):
self.counters = dict()
self.reportSize = reportSize
Expand All @@ -283,28 +273,19 @@ def __getitem__(self, classType):
self.counters[classType] = InstanceCounter(classType, self.reportSize)
return self.counters[classType]

def __iadd__(self, item):
klass = type(item)
if klass in self.counters:
counter = self.counters[klass]
else:
counter = InstanceCounter(klass, self.reportSize)
counter.first = item # done here for speed
self.counters[klass] = counter
counter += item

def countObjects(self, ao):
"""
Recursively find non-list,dict, tuple objects in containers.
Recursively find objects inside arbitrarily-deeply-nested containers.
Essential for traversing the garbage collector
This is designed to work with the garbage collector, so it focuses on
objects potentially being held in dict, tuple, list, or sets.
"""
itemType = type(ao)
counter = self[itemType]
counter = self[type(ao)]
if counter.add(ao):
self.count += 1
if self.count % 100000 == 0:
if self.count % REPORT_COUNT == 0:
runLog.info("Counted {} items".format(self.count))

if isinstance(ao, dict):
for k, v in ao.items():
self.countObjects(k)
Expand Down Expand Up @@ -335,7 +316,7 @@ def add(self, item):
self.ids.add(itemId)
if self.reportSize:
try:
self.memSize += asizeof(item)
self.memSize += sys.getsizeof(item)
except:
self.memSize = float("nan")
self.count += 1
Expand All @@ -351,89 +332,6 @@ def __gt__(self, that):
return self.count > that.count


def _getClsName(obj):
try:
return obj.__class__.__name__
except:
try:
return obj.__name__
except:
return repr(obj)[:20]


def _getModName(obj):
try:
return obj.__class__.__module__
except:
return None


class ObjectSizeBreakdown:
def __init__(
self,
name,
minMBToShowAttrBreakdown=30.0,
excludedAttributes=None,
initialZeroes=0,
):
# don't hold onto obj, otherwise we'll bloat like crazy!!
self.name = name
self.sizes = [0.0] * initialZeroes
self.minMBToShowAttrBreakdown = minMBToShowAttrBreakdown
self.excludedAttributes = excludedAttributes or []
self.attrSizes = {}

@property
def size(self):
return self.sizes[-1]

def calcSize(self, obj):
tempAttrs = {}
try:
for attrName in self.excludedAttributes:
if hasattr(obj, attrName):
tempAttrs[attrName] = getattr(obj, attrName, None)
setattr(obj, attrName, None)
self.sizes.append(asizeof(obj) / (1024.0 ** 2))
self._breakdownAttributeSizes(obj)
finally:
for attrName, attrObj in tempAttrs.items():
setattr(obj, attrName, attrObj)

def _breakdownAttributeSizes(self, obj):
if self.size > self.minMBToShowAttrBreakdown:
# make a getter where getter(obj, key) gives obj.key or obj[key]
if isinstance(obj, dict):
keys = obj.keys()
getter = lambda obj, key: obj.get(key)
elif isinstance(obj, list):
keys = range(len(obj))
getter = lambda obj, key: obj[key]
else:
keys = obj.__dict__.keys()
getter = getattr

for attrName in set(keys) - set(self.excludedAttributes):
name = " .{}".format(attrName)
if name not in self.attrSizes:
# arbitrarily, we don't care unless the attribute is a GB
self.attrSizes[name] = ObjectSizeBreakdown(
name, 1000.0, initialZeroes=len(self.sizes) - 1
)
attrSize = self.attrSizes[name]
attrSize.calcSize(getter(obj, attrName))

def __repr__(self):
message = []
name = self.name
if self.excludedAttributes:
name += " except(.{})".format(", .".join(self.excludedAttributes))
message.append("{0:53s} {1:8.4f}MB".format(name, self.size))
for attr in self.attrSizes.values():
message.append(repr(attr))
return "\n".join(message)


class ProfileMemoryUsageAction(mpiActions.MpiAction):
def __init__(self, timeDescription):
mpiActions.MpiAction.__init__(self)
Expand Down Expand Up @@ -505,7 +403,6 @@ def printUsage(self, description=None):
SYS_MEM HOSTNAME 13.9% RAM. Proc mem (MB): 474 473 472 471 460 461
SYS_MEM HOSTNAME ...
SYS_MEM HOSTNAME ...
"""
printedNodes = set()
prefix = description or "SYS_MEM"
Expand Down
15 changes: 15 additions & 0 deletions armi/bookkeeping/tests/test_memoryProfiler.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,21 @@ def test_displayMemoryUsage(self):
# do some basic testing
self.assertIn("End Memory Usage Report", mock._outputStream)

def test_printFullMemoryBreakdown(self):
with mockRunLogs.BufferLog() as mock:
# we should start with a clean slate
self.assertEqual("", mock._outputStream)
runLog.LOG.startLog("test_displayMemUsage")
runLog.LOG.setVerbosity(logging.INFO)

# we should start at info level, and that should be working correctly
self.assertEqual(runLog.LOG.getVerbosity(), logging.INFO)
self.memPro._printFullMemoryBreakdown(startsWith="", reportSize=True)

# do some basic testing
self.assertIn("UNIQUE_INSTANCE_COUNT", mock._outputStream)
self.assertIn(" MB", mock._outputStream)

def test_getReferrers(self):
with mockRunLogs.BufferLog() as mock:
# we should start with a clean slate
Expand Down
1 change: 0 additions & 1 deletion armi/tests/refTestCartesian.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ settings:
max2SigmaCladIDT: 630.0
maxFlowZones: 12
maxRegionDensityIterations: 5
outputCOBRAPinPeakingFactors: false
outputFileExtension: png
percentNaReduction: 10.0
power: 400000000.0
Expand Down
1 change: 1 addition & 0 deletions doc/release/0.2.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Bug fixes
#. Multiple bug fixes in ``growToFullCore``.
#. ``Block.getWettedPerim`` was moved to ``HexBlock``, as that was more accurate.
#. ``pathTools.cleanPath()`` is not much more linear, and handles the MPI use-case better.
#. Fixed bugs in the ARMI memory profiler.
#. TBD


Expand Down
1 change: 0 additions & 1 deletion setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@ def collectExtraFiles():
"pillow",
"pluggy",
"pyevtk",
"pympler",
"scipy",
"tabulate",
"voluptuous",
Expand Down

0 comments on commit 8c46f8c

Please sign in to comment.