lxmlrpc - XMLRPClib patch

This module monkeypatch python's xmlrpclib to use lxml based parser to reduce memory consumption on big xmlrpc requests / responses (100+ Mb)

NOTE

This module is useful only for *python2.7*.

Use it only if you suffer from high memory consumption of xmlrpclib

Install

This module is hosted on PyPI so it could be easily installed via pip:

pip install lxmlrpc_monkey

Usage

To use this module just do following

import xmlrpclib
from lxmlrpc_monkey import patch_xmlrpclib
# This line will monkey-patch xmlrpclib to use lxml for parser
patch_xmlrpclib()

Benchmarks (how to run)

To run benchmarks:

1. install memory_profiler
2. prepare data with python generate_data.py --path <demo data path> --size 50000000
3. run benchmarks with python benchmark.py --path <demo data path>

Benchmark results (50 Mb (real 65 Mb) data file)

look at `p.feed(data) call in loads` function of xmlrpclib

*Running unpatched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line #	Mem usage	Increment	Line Contents
1134 1135 1136 1137 1138 1139 1140 1141 1142	104.7 MiB	0.0 MiB	def loads(data, use_datetime=0): """data -> unmarshalled data, method name Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present). If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """
1143	104.7 MiB	0.0 MiB	p, u = getparser(use_datetime=use_datetime)
>1144	622.4 MiB	517.7 MiB	p.feed(data)
1145	558.0 MiB	-64.4 MiB	p.close()
1146	558.0 MiB	0.0 MiB	return u.close(), u.getmethodname()

*Running patched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line #	Mem usage	Increment	Line Contents
1134 1135 1136 1137 1138 1139 1140 1141 1142	106.9 MiB	0.0 MiB	def loads(data, use_datetime=0): """data -> unmarshalled data, method name Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present). If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """
1143	106.9 MiB	0.0 MiB	p, u = getparser(use_datetime=use_datetime)
>1144	235.9 MiB	129.0 MiB	p.feed(data)
1145	171.5 MiB	-64.4 MiB	p.close()
1146	171.5 MiB	0.0 MiB	return u.close(), u.getmethodname()

Filename: bechmark.py

Line #	Mem usage	Increment	Line Contents
13 14	104.7 MiB	0.0 MiB	@profile def bench_load(xmldata):
15	104.7 MiB	0.0 MiB	print ("Running unpatched loads")
16 17	106.9 MiB	2.2 MiB	loads(xmldata)
18 19	106.9 MiB	0.0 MiB	lxmlrpc.patch_xmlrpclib()
20	106.9 MiB	0.0 MiB	print ("Running patched loads")
21	107.1 MiB	0.2 MiB	loads(xmldata)

Benchmark results (100 Mb (real 129 Mb) data file)

*Running unpatched loads*

---

Filename: /usr/lib/python2.7/xmlrpclib.py

Line #	Mem usage	Increment	Line Contents
1134 1135 1136 1137 1138 1139 1140 1141 1142	169.2 MiB	0.0 MiB	def loads(data, use_datetime=0): """data -> unmarshalled data, method name Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present). If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """
1143	169.2 MiB	0.0 MiB	p, u = getparser(use_datetime=use_datetime)
>1144	1203.0 MiB	1033.8 MiB	p.feed(data)
1145	1074.2 MiB	-128.8 MiB	p.close()
1146	1074.2 MiB	0.0 MiB	return u.close(), u.getmethodname()

*Running patched loads*

---

Filename: /usr/lib/python2.7/xmlrpclib.py

Line #	Mem usage	Increment	Line Contents
1134 1135 1136 1137 1138 1139 1140 1141 1142	171.6 MiB	0.0 MiB	def loads(data, use_datetime=0): """data -> unmarshalled data, method name Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present). If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """
1143	171.6 MiB	0.0 MiB	p, u = getparser(use_datetime=use_datetime)
>1144	429.4 MiB	257.8 MiB	p.feed(data)
1145	300.6 MiB	-128.8 MiB	p.close()
1146	300.6 MiB	0.0 MiB	return u.close(), u.getmethodname()

Filename: bechmark.py

Line #	Mem usage	Increment	Line Contents
13 14	169.2 MiB	0.0 MiB	@profile def bench_load(xmldata):
15	169.2 MiB	0.0 MiB	print ("Running unpatched loads")
16 17	171.6 MiB	2.4 MiB	loads(xmldata)
18 19	171.6 MiB	0.0 MiB	lxmlrpc.patch_xmlrpclib()
20	171.6 MiB	0.0 MiB	print ("Running patched loads")
21	171.8 MiB	0.2 MiB	loads(xmldata)