Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

657 lines (498 sloc) 28.913 kB
====================
Benchmarks and Speed
====================
:Author:
Stefan Behnel
.. meta::
:description: Performance evaluation of lxml and ElementTree
:keywords: lxml performance, lxml.etree, lxml.objectify, benchmarks, ElementTree
As an XML library, lxml.etree is very fast. It is also slow. As with all
software, it depends on what you do with it. Rest assured that lxml is fast
enough for most applications, so lxml is probably somewhere between 'fast
enough' and 'the best choice' for yours.
This text describes where lxml.etree (abbreviated to 'lxe') excels, gives
hints on some performance traps and compares the overall performance to the
original ElementTree_ (ET) and cElementTree_ (cET) libraries by Fredrik Lundh.
The cElementTree library is a fast C-implementation of the original
ElementTree.
.. _ElementTree: http://effbot.org/zone/element-index.htm
.. _cElementTree: http://effbot.org/zone/celementtree.htm
.. contents::
..
1 How to read the timings
2 Bad things first
3 Parsing and Serialising
4 The ElementTree API
5 Tree traversal
6 XPath
7 lxml.objectify
General notes
=============
First thing to say: there *is* an overhead involved in having a DOM-like C
library mimic the ElementTree API. As opposed to ElementTree, lxml has to
generate Python representations of tree nodes on the fly when asked for them,
and the internal tree structure of libxml2 results in a higher maintenance
overhead than the simpler top-down structure of ElementTree. What this means
is: the more of your code runs in Python, the less you can benefit from the
speed of lxml and libxml2. Note, however, that this is true for most
performance critical Python applications. No one would implement fourier
transformations in pure Python when you can use NumPy.
The up side then is that lxml provides powerful tools like tree iterators,
XPath and XSLT, that can handle complex operations at the speed of C. Their
pythonic API in lxml makes them so flexible that most applications can easily
benefit from them.
How to read the timings
=======================
The statements made here are backed by the (micro-)benchmark scripts
`bench_etree.py`_, `bench_xpath.py`_ and `bench_objectify.py`_ that come with
the lxml source distribution. They are distributed under the same BSD license
as lxml itself, and the lxml project would like to promote them as a general
benchmarking suite for all ElementTree implementations. New benchmarks are
very easy to add as tiny test methods, so if you write a performance test for
a specific part of the API yourself, please consider sending it to the lxml
mailing list.
The timings cited below compare lxml 1.3 (with libxml2 2.6.27) to the
ElementTree and cElementTree versions shipped with CPython 2.5 (based on
ElementTree 1.2.6). They were run single-threaded on a 1.8GHz Intel Core Duo
machine under Ubuntu Linux 7.04 (Feisty).
.. _`bench_etree.py`: http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_etree.py
.. _`bench_xpath.py`: http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_xpath.py
.. _`bench_objectify.py`: http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_objectify.py
The scripts run a number of simple tests on the different libraries, using
different XML tree configurations: different tree sizes (T1-4), with or
without attributes (-/A), with or without ASCII string or unicode text
(-/S/U), and either against a tree or its serialised XML form (T/X). In the
result extracts cited below, T1 refers to a 3-level tree with many children at
the third level, T2 is swapped around to have many children below the root
element, T3 is a deep tree with few children at each level and T4 is a small
tree, slightly broader than deep. If repetition is involved, this usually
means running the benchmark in a loop over all children of the tree root,
otherwise, the operation is run on the root node (C/R).
As an example, the character code ``(SATR T1)`` states that the benchmark was
running for tree T1, with plain string text (S) and attributes (A). It was
run against the root element (R) in the tree structure of the data (T).
Note that very small operations are repeated in integer loops to make them
measurable. It is therefore not always possible to compare the absolute
timings of, say, a single access benchmark (which usually loops) and a 'get
all in one step' benchmark, which already takes enough time to be measurable
and is therefore measured as is. An example is the index access to a single
child, which cannot be compared to the timings for ``getchildren()``. Take a
look at the concrete benchmarks in the scripts to understand how the numbers
compare.
Parsing and Serialising
=======================
These are areas where lxml excels. The reason is that both parts are executed
entirely at the C level, without major interaction with Python code. The
results are rather impressive. Compared to cElementTree, lxml is about 20 to
40 times faster on serialisation::
lxe: tostring_utf16 (SATR T1) 21.9206 msec/pass
cET: tostring_utf16 (SATR T1) 461.9428 msec/pass
ET : tostring_utf16 (SATR T1) 486.8946 msec/pass
lxe: tostring_utf16 (UATR T1) 22.7508 msec/pass
cET: tostring_utf16 (UATR T1) 526.3446 msec/pass
ET : tostring_utf16 (UATR T1) 496.0767 msec/pass
lxe: tostring_utf16 (S-TR T2) 23.8452 msec/pass
cET: tostring_utf16 (S-TR T2) 537.9200 msec/pass
ET : tostring_utf16 (S-TR T2) 504.4273 msec/pass
lxe: tostring_utf8 (S-TR T2) 18.2550 msec/pass
cET: tostring_utf8 (S-TR T2) 528.3908 msec/pass
ET : tostring_utf8 (S-TR T2) 549.7071 msec/pass
lxe: tostring_utf8 (U-TR T3) 2.5497 msec/pass
cET: tostring_utf8 (U-TR T3) 49.8495 msec/pass
ET : tostring_utf8 (U-TR T3) 62.6927 msec/pass
For parsing, the difference between the libraries is smaller. The (c)ET
libraries use the expat parser, which is known to be extremely fast::
lxe: parse_stringIO (SAXR T1) 150.2380 msec/pass
cET: parse_stringIO (SAXR T1) 25.9311 msec/pass
ET : parse_stringIO (SAXR T1) 222.9431 msec/pass
lxe: parse_stringIO (S-XR T3) 5.9490 msec/pass
cET: parse_stringIO (S-XR T3) 5.4519 msec/pass
ET : parse_stringIO (S-XR T3) 76.4120 msec/pass
lxe: parse_stringIO (UAXR T3) 29.3601 msec/pass
cET: parse_stringIO (UAXR T3) 28.9941 msec/pass
ET : parse_stringIO (UAXR T3) 163.5361 msec/pass
The expat parser allows cET to be up to 80% faster than lxml on plain parser
performance. Similar timings can be observed for the ``iterparse()``
function. However, if you take a complete input-output cycle, the numbers
will look similar to these::
lxe: write_utf8_parse_stringIO (S-TR T1) 166.3210 msec/pass
cET: write_utf8_parse_stringIO (S-TR T1) 581.2099 msec/pass
ET : write_utf8_parse_stringIO (S-TR T1) 803.5331 msec/pass
lxe: write_utf8_parse_stringIO (UATR T2) 184.4249 msec/pass
cET: write_utf8_parse_stringIO (UATR T2) 671.5119 msec/pass
ET : write_utf8_parse_stringIO (UATR T2) 924.3481 msec/pass
lxe: write_utf8_parse_stringIO (S-TR T3) 9.1329 msec/pass
cET: write_utf8_parse_stringIO (S-TR T3) 77.9850 msec/pass
ET : write_utf8_parse_stringIO (S-TR T3) 157.0492 msec/pass
lxe: write_utf8_parse_stringIO (SATR T4) 1.3900 msec/pass
cET: write_utf8_parse_stringIO (SATR T4) 12.6081 msec/pass
ET : write_utf8_parse_stringIO (SATR T4) 16.2580 msec/pass
For applications that require a high parser throughput and do little
serialization, cET is the best choice. Also for iterparse applications that
extract small amounts of data from large XML data sets. If it comes to
round-trip performance, however, lxml tends to be 3-4 times faster in
total. So, whenever the input documents are not considerably bigger than the
output, lxml is the clear winner.
The ElementTree API
===================
Since all three libraries implement the same API, their performance is easy to
compare in this area. A major disadvantage for lxml's performance is the
different tree model that underlies libxml2. It allows lxml to provide parent
pointers for elements, but also increases the overhead of tree building and
restructuring. This can be seen from the tree setup times of the benchmark
(given in seconds)::
lxe: -- S- U- -A SA UA
T1: 0.1181 0.1080 0.1074 0.1088 0.1087 0.1099
T2: 0.1103 0.1109 0.1164 0.1241 0.1203 0.1231
T3: 0.0297 0.0309 0.0297 0.0716 0.0704 0.0703
T4: 0.0005 0.0004 0.0004 0.0014 0.0014 0.0014
cET: -- S- U- -A SA UA
T1: 0.0290 0.0271 0.0275 0.0297 0.0273 0.0274
T2: 0.0280 0.0280 0.0281 0.0285 0.0283 0.0286
T3: 0.0071 0.0072 0.0071 0.0113 0.0096 0.0096
T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001
ET : -- S- U- -A SA UA
T1: 0.1362 0.1985 0.2300 0.1344 0.2672 0.1335
T2: 0.3107 0.1386 0.3581 0.3886 0.1388 0.4277
T3: 0.0334 0.0332 0.0320 0.0367 0.3769 0.0375
T4: 0.0006 0.0005 0.0008 0.0007 0.0007 0.0006
While lxml is still faster than ET in most cases (30-60%), cET can be up to
three times faster than lxml here. One of the reasons is that lxml must
additionally discard the created Python elements after their use, when they
are no longer referenced. ET and cET represent the tree itself through these
objects, which reduces the overhead in creating them.
Child access
------------
The same reason makes operations like ``getchildren()`` more costly in lxml.
Where ET and cET can quickly create a shallow copy of their list of children,
lxml has to create a Python object for each child and collect them in a list::
lxe: root_getchildren (--TR T2) 0.1960 msec/pass
cET: root_getchildren (--TR T2) 0.0150 msec/pass
ET : root_getchildren (--TR T2) 0.0091 msec/pass
When accessing single children, however, e.g. by index, this handicap is
negligible::
lxe: first_child (--TR T2) 0.2289 msec/pass
cET: first_child (--TR T2) 0.2048 msec/pass
ET : first_child (--TR T2) 0.9291 msec/pass
lxe: last_child (--TR T1) 0.2310 msec/pass
cET: last_child (--TR T1) 0.2148 msec/pass
ET : last_child (--TR T1) 0.9191 msec/pass
... unless you add the time to find a child index in a bigger list, as ET and
cET use Python lists here, which are based on arrays. The data structure used
by libxml2 is a linked tree, and thus, a linked list of children::
lxe: middle_child (--TR T1) 0.2759 msec/pass
cET: middle_child (--TR T1) 0.2069 msec/pass
ET : middle_child (--TR T1) 0.9291 msec/pass
lxe: middle_child (--TR T2) 1.7111 msec/pass
cET: middle_child (--TR T2) 0.2089 msec/pass
ET : middle_child (--TR T2) 0.9360 msec/pass
Element creation
----------------
As opposed to ET, libxml2 has a notion of documents that each element must be
in. This results in a major performance difference for creating independent
Elements that end up in independently created documents::
lxe: create_elements (--TC T2) 3.7301 msec/pass
cET: create_elements (--TC T2) 0.1960 msec/pass
ET : create_elements (--TC T2) 1.4279 msec/pass
Therefore, it is always preferable to create Elements for the document they
are supposed to end up in, either as SubElements of an Element or using the
explicit ``Element.makeelement()`` call::
lxe: makeelement (--TC T2) 2.3680 msec/pass
cET: makeelement (--TC T2) 0.3128 msec/pass
ET : makeelement (--TC T2) 1.6940 msec/pass
lxe: create_subelements (--TC T2) 2.2051 msec/pass
cET: create_subelements (--TC T2) 0.2370 msec/pass
ET : create_subelements (--TC T2) 3.2189 msec/pass
So, if the main performance bottleneck of an application is creating large XML
trees in memory through calls to Element and SubElement, cET is the best
choice. Note, however, that the serialisation performance may even out this
advantage, especially for smaller trees and trees with many attributes.
Merging different sources
-------------------------
A critical action for lxml is moving elements between document contexts. It
requires lxml to do recursive adaptations throughout the moved tree structure.
The following benchmark appends all root children of the second tree to the
root of the first tree::
lxe: append_from_document (--TR T1,T2) 4.3468 msec/pass
cET: append_from_document (--TR T1,T2) 0.2608 msec/pass
ET : append_from_document (--TR T1,T2) 1.2310 msec/pass
lxe: append_from_document (--TR T3,T4) 0.0679 msec/pass
cET: append_from_document (--TR T3,T4) 0.0148 msec/pass
ET : append_from_document (--TR T3,T4) 0.0880 msec/pass
Although these are fairly small numbers compared to parsing, this easily shows
the different performance classes for lxml and (c)ET. Where the latter do not
have to care about parent pointers and tree structures, lxml has to deep
traverse the appended tree. The performance difference therefore increases
with the size of the tree that is moved.
This difference is not always as visible, but applies to most parts of the
API, like inserting newly created elements::
lxe: insert_from_document (--TR T1,T2) 6.3150 msec/pass
cET: insert_from_document (--TR T1,T2) 0.4039 msec/pass
ET : insert_from_document (--TR T1,T2) 1.4770 msec/pass
Or replacing the child slice by a new element::
lxe: replace_children_element (--TC T1) 0.2608 msec/pass
cET: replace_children_element (--TC T1) 0.0238 msec/pass
ET : replace_children_element (--TC T1) 0.1628 msec/pass
You should keep this difference in mind when you merge very large trees.
deepcopy
--------
Deep copying a tree is fast in lxml::
lxe: deepcopy_all (--TR T1) 11.0400 msec/pass
cET: deepcopy_all (--TR T1) 119.6141 msec/pass
ET : deepcopy_all (--TR T1) 451.2160 msec/pass
lxe: deepcopy_all (-ATR T2) 13.5410 msec/pass
cET: deepcopy_all (-ATR T2) 135.2482 msec/pass
ET : deepcopy_all (-ATR T2) 476.1350 msec/pass
lxe: deepcopy_all (S-TR T3) 4.2889 msec/pass
cET: deepcopy_all (S-TR T3) 36.0429 msec/pass
ET : deepcopy_all (S-TR T3) 113.4322 msec/pass
So, for example, if you have a database-like scenario where you parse in a
large tree and then search and copy independent subtrees from it for further
processing, lxml is by far the best choice here.
Tree traversal
--------------
Another area where lxml is very fast is iteration for tree traversal. If your
algorithms can benefit from step-by-step traversal of the XML tree and
especially if few elements are of interest or the target element tag name is
known, lxml is a good choice::
lxe: getiterator_all (--TR T2) 6.4790 msec/pass
cET: getiterator_all (--TR T2) 28.2831 msec/pass
ET : getiterator_all (--TR T2) 26.0720 msec/pass
lxe: getiterator_islice (--TR T2) 0.0892 msec/pass
cET: getiterator_islice (--TR T2) 0.2460 msec/pass
ET : getiterator_islice (--TR T2) 26.6550 msec/pass
lxe: getiterator_tag (--TR T2) 0.3850 msec/pass
cET: getiterator_tag (--TR T2) 9.3720 msec/pass
ET : getiterator_tag (--TR T2) 22.8221 msec/pass
lxe: getiterator_tag_all (--TR T2) 0.7222 msec/pass
cET: getiterator_tag_all (--TR T2) 27.2939 msec/pass
ET : getiterator_tag_all (--TR T2) 22.8271 msec/pass
This translates directly into similar timings for ``Element.findall()``::
lxe: findall (--TR T2) 6.8321 msec/pass
cET: findall (--TR T2) 28.8639 msec/pass
ET : findall (--TR T2) 27.1060 msec/pass
lxe: findall (--TR T3) 1.3590 msec/pass
cET: findall (--TR T3) 8.9881 msec/pass
ET : findall (--TR T3) 6.4890 msec/pass
lxe: findall_tag (--TR T2) 0.9229 msec/pass
cET: findall_tag (--TR T2) 27.2651 msec/pass
ET : findall_tag (--TR T2) 22.7208 msec/pass
lxe: findall_tag (--TR T3) 0.1700 msec/pass
cET: findall_tag (--TR T3) 6.4540 msec/pass
ET : findall_tag (--TR T3) 5.4770 msec/pass
Note that all three libraries currently use the same Python implementation for
``findall()``, except for their native tree iterator.
XPath
=====
The following timings are based on the benchmark script `bench_xpath.py`_.
This part of lxml does not have an equivalent in ElementTree. However, lxml
provides more than one way of accessing it and you should take care which part
of the lxml API you use. The most straight forward way is to call the
``xpath()`` method on an Element or ElementTree::
lxe: xpath_method (--TC T1) 1.0180 msec/pass
lxe: xpath_method (--TC T2) 20.3521 msec/pass
lxe: xpath_method (--TC T3) 0.1259 msec/pass
lxe: xpath_method (--TC T4) 1.0169 msec/pass
This is well suited for testing and when the XPath expressions are as diverse
as the trees they are called on. However, if you have a single XPath
expression that you want to apply to a larger number of different elements,
the ``XPath`` class is the most efficient way to do it::
lxe: xpath_class (--TC T1) 0.1891 msec/pass
lxe: xpath_class (--TC T2) 3.0179 msec/pass
lxe: xpath_class (--TC T3) 0.0570 msec/pass
lxe: xpath_class (--TC T4) 0.1910 msec/pass
Note that this still allows you to use variables in the expression, so you can
parse it once and then adapt it through variables at call time. In other
cases, where you have a fixed Element or ElementTree and want to run different
expressions on it, you should consider the ``XPathEvaluator``::
lxe: xpath_element (--TR T1) 0.4089 msec/pass
lxe: xpath_element (--TR T2) 5.9960 msec/pass
lxe: xpath_element (--TR T3) 0.1230 msec/pass
lxe: xpath_element (--TR T4) 0.3440 msec/pass
While it looks slightly slower, creating an XPath object for each of the
expressions generates a much higher overhead here::
lxe: xpath_class_repeat (--TC T1) 1.0259 msec/pass
lxe: xpath_class_repeat (--TC T2) 20.4861 msec/pass
lxe: xpath_class_repeat (--TC T3) 0.1280 msec/pass
lxe: xpath_class_repeat (--TC T4) 1.0269 msec/pass
A longer example
================
A while ago, Uche Ogbuji posted a `benchmark proposal`_ that would read in a
3MB XML version of the `Old Testament`_ of the Bible and look for the word
*begat* in all verses. Apparently, it is contained in 120 out of almost 24000
verses. This is easy to implement in ElementTree using ``findall()``.
However, the fastest way to do this is obviously ``iterparse()``, as most of
the data is not of any interest.
.. _`benchmark proposal`: http://www.onlamp.com/pub/wlg/6291
.. _`Old Testament`: http://www.ibiblio.org/bosak/xml/eg/religion.2.00.xml.zip
Now, Uche's original proposal was more or less the following::
def bench_ET():
tree = ElementTree.parse("ot.xml")
result = []
for v in tree.findall("//v"):
text = v.text
if 'begat' in text:
result.append(text)
return len(result)
which takes about one second on my machine today. The faster ``iterparse()``
variant looks like this::
def bench_ET_iterparse():
result = []
for event, v in ElementTree.iterparse("ot.xml"):
if v.tag == 'v':
text = v.text
if 'begat' in text:
result.append(text)
v.clear()
return len(result)
The improvement is about 10%. At the time I first tried (early 2006), lxml
didn't have ``iterparse()`` support, but the ``findall()`` variant was already
faster than ElementTree. This changes immediately when you switch to
cElementTree. The latter only needs 0.17 seconds to do the trick today and
only some impressive 0.10 seconds when running the iterparse version. And
even back then, it was quite a bit faster than what lxml could achieve.
Since then, lxml has matured a lot and has gotten much faster. The iterparse
variant now runs in 0.14 seconds, and if you remove the ``v.clear()``, it is
even a little faster (which isn't the case for cElementTree).
One of the many great tools in lxml is XPath, a swiss army knife for finding
things in XML documents. It is possible to move the whole thing to a pure
XPath implementation, which looks like this::
def bench_lxml_xpath_all():
tree = etree.parse("ot.xml")
result = tree.xpath("//v[contains(., 'begat')]/text()")
return len(result)
This runs in about 0.13 seconds and is about the shortest possible
implementation (in lines of Python code) that I could come up with. Now, this
is already a rather complex XPath expression compared to the simple "//v"
ElementPath expression we started with. Since this is also valid XPath, let's
try this instead::
def bench_lxml_xpath():
tree = etree.parse("ot.xml")
result = []
for v in tree.xpath("//v"):
text = v.text
if 'begat' in text:
result.append(text)
return len(result)
This gets us down to 0.12 seconds, thus showing that a generic XPath
evaluation engine cannot always compete with a simpler, tailored solution.
However, since this is not much different from the original findall variant,
we can remove the complexity of the XPath call completely and just go with
what we had in the beginning. Under lxml, this runs in the same 0.12 seconds.
But there is one thing left to try. We can replace the simple ElementPath
expression with a native tree iterator::
def bench_lxml_getiterator():
tree = etree.parse("ot.xml")
result = []
for v in tree.getiterator("v"):
text = v.text
if 'begat' in text:
result.append(text)
return len(result)
This implements the same thing, just without the overhead of parsing and
evaluating a path expression. And this makes it another bit faster, down to
0.11 seconds. For comparison, cElementTree runs this version in 0.17 seconds.
So, what have we learned?
* Python code is not slow. The pure XPath solution was not even as fast as
the first shot Python implementation. In general, a few more lines in
Python make things more readable, which is much more important than the last
5% of performance.
* It's important to know the available options - and it's worth starting with
the most simple one. In this case, a programmer would then probably have
started with ``getiterator("v")`` or ``iterparse()``. Either of them would
already have been the most efficient, depending on which library is used.
* It's important to know your tool. lxml and cElementTree are both very fast
libraries, but they do not have the same performance characteristics. The
fastest solution in one library can be comparatively slow in the other. If
you optimise, optimise for the specific target platform.
* It's not always worth optimising. After all that hassle we got from 0.12
seconds for the initial implementation to 0.11 seconds. Switching over to
cElementTree and writing an ``iterparse()`` based version would have given
us 0.10 seconds - not a big difference for 3MB of XML.
* Take care what operation is really dominating in your use case. If we split
up the operations, we can see that lxml is slightly slower than cElementTree
on ``parse()`` (both about 0.06 seconds), but more visibly slower on
``iterparse()``: 0.07 versus 0.10 seconds. However, tree iteration in lxml
is increadibly fast, so it can be better to parse the whole tree and then
iterate over it rather than using ``iterparse()`` to do both in one step.
Or, you can just wait for the lxml authors to optimise iterparse in one of
the next releases...
lxml.objectify
==============
The following timings are based on the benchmark script `bench_objectify.py`_.
Objectify is a data-binding API for XML based on lxml.etree, that was added in
version 1.1. It uses standard Python attribute access to traverse the XML
tree. It also features ObjectPath, a fast path language based on the same
meme.
Just like lxml.etree, lxml.objectify creates Python representations of
elements on the fly. To save memory, the normal Python garbage collection
mechanisms will discard them when their last reference is gone. In cases
where deeply nested elements are frequently accessed through the objectify
API, the create-discard cycles can become a bottleneck, as elements have to be
instantiated over and over again.
ObjectPath
----------
ObjectPath can be used to speed up the access to elements that are deep in the
tree. It avoids step-by-step Python element instantiations along the path,
which can substantially improve the access time::
lxe: attribute (--TR T1) 10.6189 msec/pass
lxe: attribute (--TR T2) 53.7431 msec/pass
lxe: attribute (--TR T4) 10.3359 msec/pass
lxe: objectpath (--TR T1) 5.8351 msec/pass
lxe: objectpath (--TR T2) 48.1579 msec/pass
lxe: objectpath (--TR T4) 5.6930 msec/pass
lxe: attributes_deep (--TR T1) 58.7430 msec/pass
lxe: attributes_deep (--TR T2) 63.0901 msec/pass
lxe: attributes_deep (--TR T4) 17.4620 msec/pass
lxe: objectpath_deep (--TR T1) 52.1719 msec/pass
lxe: objectpath_deep (--TR T2) 52.9201 msec/pass
lxe: objectpath_deep (--TR T4) 7.5650 msec/pass
Note, however, that parsing ObjectPath expressions is not for free either, so
this is most effective for frequently accessing the same element.
Caching Elements
----------------
A way to improve the normal attribute access time is static instantiation of
the Python objects, thus trading memory for speed. Just create a cache
dictionary and run::
cache[root] = list(root.iter())
after parsing and::
del cache[root]
when you are done with the tree. This will keep the Python element
representations of all elements alive and thus avoid the overhead of repeated
Python object creation. You can also consider using filters or generator
expressions to be more selective. By choosing the right trees (or even
subtrees and elements) to cache, you can trade memory usage against access
speed::
lxe: attribute_cached (--TR T1) 7.9739 msec/pass
lxe: attribute_cached (--TR T2) 50.9331 msec/pass
lxe: attribute_cached (--TR T4) 7.8540 msec/pass
lxe: attributes_deep_cached (--TR T1) 51.1391 msec/pass
lxe: attributes_deep_cached (--TR T2) 55.7129 msec/pass
lxe: attributes_deep_cached (--TR T4) 10.7968 msec/pass
lxe: objectpath_deep_cached (--TR T1) 47.6151 msec/pass
lxe: objectpath_deep_cached (--TR T2) 48.0802 msec/pass
lxe: objectpath_deep_cached (--TR T4) 4.0281 msec/pass
Things to note: you cannot currently use ``weakref.WeakKeyDictionary`` objects
for this as lxml's element objects do not support weak references (which are
costly in terms of memory). Also note that new element objects that you add
to these trees will not turn up in the cache automatically and will therefore
still be garbage collected when all their Python references are gone, so this
is most effective for largely immutable trees. You should consider using a
set instead of a list in this case and add new elements by hand.
Further optimisations
---------------------
Here are some more things to try if optimisation is required:
* A lot of time is usually spent in tree traversal to find the addressed
elements in the tree. If you often work in subtrees, do what you would also
do with deep Python objects: assign the parent of the subtree to a variable
or pass it into functions instead of starting at the root. This allows
accessing its descendents more directly.
* Try assigning data values directly to attributes instead of passing them
through DataElement.
* If you use custom data types that are costly to parse, try running
``objectify.annotate()`` over read-only trees to speed up the attribute type
inference on read access.
Note that none of these measures is guaranteed to speed up your application.
As usual, you should prefer readable code over premature optimisations and
profile your expected use cases before bothering to apply optimisations at
random.
Jump to Line
Something went wrong with that request. Please try again.