Python scripts for processing XML documents and converting to SQL, CSV, and JSON
Python
Latest commit 64168e2 Mar 23, 2016 @knadh version bump

README.md

xmlutils.py

xmlutils.py is a set of Python utilities for processing xml files serially
for converting them to various formats (SQL, CSV, JSON). The scripts use ElementTree.iterparse() to iterate through nodes in an XML document, thus not needing to load the entire DOM into memory. The scripts can be used to churn through large XML files (albeit taking long :P) without memory hiccups.

Simple table-representing XMLs can be converted to CSV using xmltable2csv. It assumes each entry is encapsulated in some tag, and successfuly tested on some XLSX files.

Blind conversion of XML to CSV and SQL is not recommended. It only works if the structure of the XML document is simple (flat). On the other hand, xml2json supports complex XML documents with multiple nested hierarchies. Lastly, the XML files are not validated at the time of conversion.

Installation

With pip or easy_install

pip install xmlutils or easy_install xmlutils

Or from the source

python setup.py install

Commandline utilities

Once the package is installed, the three bundled commandline utilities should be available from the terminal.

xml2csv

Convert an XML document to a CSV file.

xml2csv --input "samples/fruits.xml" --output "samples/fruits.csv" --tag "item"
Arguments
--input     Input XML document's filename*
--output    Output CSV file's filename*
--tag       The tag of the node that represents a single record (Eg: item, record)*
--delimiter     Delimiter for seperating items in a row. Default is , (a comma followed by a space)
--ignore    A space separated list of element tags in the XML document to ignore
--noheader  Exclude CSV fields header (first line). Off by default
--encoding  Character encoding of the document. Default is utf-8
--limit     Limit the number of records to be processed from the document to a particular number. Default is no limit (-1)
--buffer    The number of records to be kept in memory before it is written to the output CSV file. Helps reduce the number of disk writes. Default is 1000

xmltable2csv

Convert an XML table to a CSV file.

xmltable2csv --input "samples/fruits.xml" --output "samples/fruits.csv" --tag "Data"
Arguments
--input         Input XML table's filename*
--output        Output CSV file's filename*
--tag           The tag of the node that represents a single record (Eg: Data, record)*
--delimiter     Delimiter for seperating items in a row. Default is , (a comma followed by a space)
--header        Whether to print the header (first row of records in the XML) in the first line; 1=yes, 0=no. Default is 1.
--encoding      Character encoding of the document. Default is utf-8
--limit         Limit the number of records to be processed from the document to a particular number. Default is no limit (-1)
--buffer        The number of records to be kept in memory before it is written to the output CSV file. Helps reduce the number of disk writes. Default is 1000.

xml2sql

Convert an XML document to an SQL file.

xml2sql --input "samples/fruits.xml" --output "samples/fruits.sql" --tag "item" --table "myfruits"
Arguments
--tag           the record tag. eg: item
--table         table name
--ignore        list of tags to ignore
--limit         maximum number of records to process
--packet        maximum size of an insert query in MB (MySQL's max_allowed_packet)

xml2json

Convert XML to JSON. xml2json supports hierarchies nested to any number of levels.

xml2json --input "samples/fruits.xml" --output "samples/fruits.json"

Modules

xmlutils.xml2sql

from xmlutils.xml2sql import xml2sql

converter = xml2sql("samples/fruits.xml", "samples/fruits.sql", encoding="utf-8")
converter.convert(tag="item", table="table")
Arguments
tag     -- the record tag. eg: item
table   -- table name
ignore  -- list of tags to ignore
limit   -- maximum number of records to process
packet  -- maximum size of an insert query in MB (MySQL's max_allowed_packet)

Returns:
{   num: number of records converted,
    num_insert: number of sql insert statements generated
}

xmlutils.xml2csv

from xmlutils.xml2csv import xml2csv

converter = xml2csv("samples/fruits.xml", "samples/fruits.csv", encoding="utf-8")
converter.convert(tag="item")
Arguments
tag -- the record tag. eg: item
delimiter -- csv field delimiter
ignore  -- list of tags to ignore
limit   -- maximum number of records to process
buffer  -- number of records to keep in buffer before writing to disk

Returns:
number of records converted

xmlutils.xml2json

from xmlutils.xml2json import xml2json

converter = xml2json("samples/fruits.xml", "samples/fruits.json", encoding="utf-8")
converter.convert()

# to get a json string
converter = xml2json("samples/fruits.xml", encoding="utf-8")
print converter.get_json()
Arguments
pretty  -- pretty print?