If you find this repo and consider using it, reconsider using xmltodict instead since it is more feature complete, well tested and much more actively maintained!
It can be problematic to handle large XML files (>> 10 MB) and using the
xml module in Python directly leads to huge memory overheads. Most
often, these large XML files are pure data files, storing highly
structured data that have no intrinsic need to be stored in XML.
This package provides iterative methods for dealing with them, reading
the XML documents into Python dict representation instead, according to
methodology specifed on the page Converting Between XML and JSON.
xmlr is inspired by the solutions
described in the blog posts High-performance XML parsing in Python with lxml and
Parsing large XML files, serially, in Python,
enabling the parsing of very large documents without problems with
overtaxing the memory.
This package generally provides a one way trip; there is not necessarily a bijectional relation with the XML source after parsing.
pip install xmlr
To parse an entire document, use the
from xmlr import xmlparse doc = xmlparse('very_large_doc.xml')
xmliter, yielding elements of a specified type as they
are parsed from the document is also present:
from xmlr import xmliter for d in xmliter('very_large_record.xml', 'Record'): print(d)
The desired parser can also be specified. Available methods are:
lxml.etreeas backend. Requires installation of the
These can then be used like this:
from xmlr import xmliter, XMLParsingMethods for d in xmliter('very_large_record.xml', 'Record', parsing_method=XMLParsingMethods.LXML_ELEMENTTREE): print(d)
No type conversion is performed right now. A value in the output
dictionary can have the type
dict (a subdocument),
array of similar documents),
str (a leaf or value) or
(empty XML leaf tag). All keys are of the type
Tests are run with
$ py.test tests/ ============================= test session starts ============================== platform linux2 -- Python 2.7.6, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 rootdir: /home/hbldh/Repos/xmlr, inifile: collected 50 items tests/test_iter.py ........................... tests/test_methods.py .. tests/test_parsing.py ..................... ========================== 50 passed in 0.50 seconds ===========================
The tests fetches some XML documents from W3Schools XML tutorials and also uses a bundled, slimmed down version of the document available at U.S. copyright renewal records available for download.