Huge memory usage by MISP objects

[mback2k]

Hello everyone,

we are running into issues, because creating a lot of MISPObjects (and other similar types) requires a lot of memory at the moment. We had an example here with roughly 60 GB of Python memory usage in our toolchain.

This is caused by the fact that each instance of MISPObject loads the JSON template files for itself instead of using a global cache (based on filename and hash maybe):

PyMISP/pymisp/mispevent.py

Lines 1206 to 1218 in de6a64b

	def _load_template_path(self, template_path):
	if not os.path.exists(template_path):
	return False
	with open(template_path, 'rb') as f:
	if OLD_PY3:
	self._definition = json.loads(f.read().decode())
	else:
	self._definition = json.load(f)
	setattr(self, 'meta-category', self._definition['meta-category'])
	self.template_uuid = self._definition['uuid']
	self.description = self._definition['description']
	self.template_version = self._definition['version']
	return True

I can try to implement such caching, but would you be interested in that?

Best regards,
Marc

[Rafiot]

Hello Marc, yes, definitely, that would be very interesting.

The other thing I was thinking about was to auto-generate enums with types and categories and use that in the attributes instead of the strings every time. I'd assume it will be nicer on the memory usage, but your idea is probably also a good call, at least for the templates bundled-in the package.

[Rafiot]

I merged a improvement, it might be only a partial fix, but please open a new issue if needed.

[mback2k]

@Rafiot Just for your information: I found our additional memory issue in our code and was finally able to solve it. Under some circumstances a list was being appended to itself a few hundred times and that caused exponential memory usage. Thanks again for your support with the PyMISP improvements, those also helped a lot.

[Rafiot]

Thank you for opening the issue, it helped a lot to pinpoint the problem. Do not hesitate to get in touch again if you have an other one.