-
-
Notifications
You must be signed in to change notification settings - Fork 313
ReduceMemory
This page describes techniques for reducing the memory footprint of a SCons build. The first part addresses SCons users while the second part discusses techniques that can be applied to reduce the memory consumption in SCons itself.
SCons implements some debugging facilities that can help to track down memory problems. If those are not sufficient, check out the SCons Heapmonitor branch.
scons --debug=memory --debug=count
If you got the impression that SCons allocates too much memory to build your project you should first make sure it is really SCons and not a build tool that is used. The easiest way is to run an up-to-date check of your project after a full build was performed. If you still think that SCons takes up too much memory run SCons with the above mentioned debug flags.
The Node and Executor objects in the --debug=count
output should correspond to the number of files (or other Node types). There is not much you can do about it. If you see a high number of Environment objects, try to reduce those in your SConscript files.
Environment objects have a high memory footprint. Try to reuse existing environments if you can. Also note that you can override flags when invoking the builder:
env.Program('fastfoo.c', CFLAGS='-O3')
If you can formulate invariants stating that when reaching a specific condition an object is not needed anymore, delete it.
Caching is commonly employed to speed up access to values which are used more then once. Therefore, there is always a trade-off between memory consumption and runtime performance. SCons used _memo
dictionaries attached to each object which uses caching. These dictionaries allocate a considerable share of memory. It should always be asked if caching a specific object actually speeds up the build enough to sacrifice the additional memory used by the slot in the cache.
Empty sequences, dictionaries and sets consume memory. If such an attribute is only needed for a small subset of the instances of a class, lazy initialization can be used. The ignore
and depends
attributes of Node objects would be a classic example. Instead of:
def __init__(self):
self.rarely_used = {}
def append(self, k, v):
self.rarely_used[k] = v
The rarely_used
dictionary can be assigned to an object only if it's used:
def __init__(self):
pass
def append(self, k, v):
try:
self.rarely_used[k] = v
except AttributeError:
self.rarely_used = {k: v}
If the append
method is not called for most of the existing objects, the alternative above might be used. However, every attempt to access the object must then be protected to not raise an AttributeError
.
Pros | Cons |
---|---|
Does not require special language features | Reduces readability of the code |
New-styles classes with slots might be used for helper classes which contain a fixed set of attributes. Some memory overhead can be avoided by using slots because it doesn't create a __dict__
for the object.
class Color(object):
__slots__ = ('red', 'green', 'blue')
Pros | Cons |
---|---|
Few code changes needed, faster instantiation | Requires Python 2.2; pickling must be done manually; does not work with morphing classes |
String objects are not reused in general. Assigning the same string to another string actually makes a copy. This can be avoided by using the built-in function intern(). Using intern()
applies the Fly-weight pattern: It's most useful when a huge number of instances share a small number of unique attributes. For example, in a huge address database for a specific state, the city
attribute might be interned as it will most likely be reused by a large number of other address instances:
class Citizen:
def __init__(self, name, address, city):
self.name = name
self.address = address
self.city = intern(city)
In SCons, intern strings could be used for filenames, suffixes, any string that has a good chance of being reused.
Pros | Cons |
---|---|
Few code changes needed | interned strings are immortal in Python 2.2 and before |
Don't create a unique object if a singleton can be used instead.