Don't build whole xml tree for stanza payload #65

Merged
merged 6 commits into from Dec 23, 2016

Projects

None yet

2 participants

@olruzhytskyi

Some stanzas (with roster for example) can contain huge amount of xml, which is all parsed into xml tree even if use_raw=True. Because of this punjab can consume large amount of RAM (possibly because of python memory fragmentation http://www.gossamer-threads.com/lists/python/python/1162114) and CPU (to create domish.Element() instances).

ShallowExpatElementStream overrides _onStartElement() and _onEndElement() methods of regular parser to avoid building whole tree, which is done by default in domish.py:ExpatElementStream.

@twonds
Owner
twonds commented Dec 21, 2016

Thanks for this, I will look at it as soon as I can.

@twonds

This looks good. The changes to .travis.yml are not related but that is ok. Just need the small change to handle the possibility of not having the expat dependency.

# need this to be in xmlstream
- stream = domish.elementStream()
@twonds
twonds Dec 21, 2016 Owner

elementStream returns an ExpatElementStream or an SuxElementStream depending if expat is available or not. If someone is running punjab without that dependency it will fail.

If you could mimic this function, I think that would be ok.
https://github.com/twisted/twisted/blob/trunk/src/twisted/words/xish/domish.py#L591

@olruzhytskyi
olruzhytskyi Dec 22, 2016

Thanks for noticing this. I've added elementStream() func, almost the same as in domish.py. I can also make ShallowSaxElementStream, but not sure if it is necessary.

@twonds
twonds approved these changes Dec 23, 2016 View changes
@twonds twonds merged commit 30027b3 into twonds:master Dec 23, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment