RAM consumption

[chrisk280]

I'm having an issue with RAM consumption. It seems that when i process a lot of files (several thousand but one after another) im running out of RAM and i think it shouldn't happen in this situation.
When using the following code eventually my program gets killed because the OS is running out of RAM.
I have also tried to implement an easy way to monitor the storage usage. You can see that more and more dict objects are created (and not deleted i guess) which take more and more RAM.
If i run the xmltodict parser instead everything works just fine (but it is harder to access the data).

from stix.core import STIXPackage
from pympler import summary
from pympler import muppy
#import xmltodict

# define a list of paths
paths = ...

n = 0
for path in paths:
    if n%10 == 0:
        sum = summary.summarize(muppy.get_objects())
        summary.print_(sum)
        n = 0
    # I've got no issues when parsing with xmltodict
    #with open(path, "rb") as file:
        #data_dicts = xmltodict.parse(file)
    stix_package = STIXPackage.from_xml(path)
    data_dicts = stix_package.to_dict()
    n += 1

The short version of my output is something like this:

types	objects	total size
<class 'dict	6680	3.45 MB
...	...	...
<class 'dict	40947	11.29 MB
...	...	...
<class 'dict	100243	28.67 MB
and	so	on
<class 'dict	176329	50.51 MB

The number of objects and the storage consumption keeps rising.

Thanks for your help!

[gtback]

Hi, @chrisk280 . Thanks for reporting this!

It's possible that the issue is coming from python-cybox (which python-stix uses internally). There is a workaround in this issue

Basically, python-cybox caches references to created objects so they can be looked up later, which isn't good for long-running processes or ones where a lot of data is processed.

If that doesn't fix the issue, let us know and I'll look into it more.

[chrisk280]

Thanks for this workaround. I solved the problem with xmltodict in the meantime.
Today I tested your solution at work anyway and it fixed the problem.
Everything i had to add at the beginning of my script was:

from cybox.core import Object

Object.id_.postset_hook = None

I think this workaround should be mentioned somewhere in the README. I think it's likely that other people will stumble across this issue too when working with many and/or big files.

[emmanvg]

@chrisk280, I think cybox.utils.caches.cache_clear() would work equally as good. You just need to place it in the most convenient place in your code. (e.g., every time you parse a STIX package or after a certain threshlod is met.

Notes: CybOXProject/python-cybox#290 (comment)
Code Documentation: https://github.com/CybOXProject/python-cybox/blob/master/cybox/core/object.py#L64
ReadTheDocs: http://cybox.readthedocs.io/en/stable/api/cybox/core/object.html

[chrisk280]

Thank you for the additional information. Apparently i didn't dig deep enough to learn about the caching mechanism.

[emmanvg]

Is there anything to do here or is it OK to close?

[gtback]

We can close this issue. We should add documentation to python-cybox about how to avoid this, and probably add a note in python-stix documentation as well, since that's probably how people are most likely to run across it.

(edit: just realized there's already documentation of this here: https://cybox.readthedocs.io/en/stable/api/cybox/core/object.html#cybox.core.object.Object)