Permalink
Fetching contributors…
Cannot retrieve contributors at this time
104 lines (84 sloc) 4.49 KB

libxmlb

Introduction

XML is slow to parse and strings inside the document cannot be memory mapped as they do not have a trailing NUL char. The libxmlb library takes XML source, and converts it to a structured binary representation with a deduplicated string table -- where the strings have the NULs included.

This allows an application to mmap the binary XML file, do an XPath query and return some strings without actually parsing the entire document. This is all done using (almost) zero allocations and no actual copying of the binary data.

As each node in the binary XML file encodes the 'next' node at the same level it makes skipping whole subtrees trivial. A 10Mb binary XML file can be loaded from disk and queried in less than a few milliseconds.

The binary XML is not supposed to be small. It's usually about half the size of the text XML data where a lot of the tag content is duplicated, but can actually be larger than the original XML file. This isn't important; the fast query speed and the ability to mmap strings without copies more than makes up for the larger on-disk size. If you want to compress your XML, this library probably isn't for you -- just use gzip -- its gives you an almost a perfect compression ratio for data like this.

For example:

$ xb-tool compile fedora.xmlb fedora.xml.gz

$ du -h fedora.xml*
12M         fedora.xmlb
3.6M        fedora.xml.gz

$ xb-tool query fedora.xmlb "components/component[@type=desktop]/id[text()=firefox.desktop]"
RESULT: firefox.desktop
real        0m0.011s
user        0m0.010s
sys         0m0.001s

XPath

This library only implements a tiny subset of XPath. See the examples for the full list, but it's basically restricted to element_name, attributes and text.

We will use the following XML document in the examples below.

<?xml version="1.0" encoding="UTF-8"?>
<bookstore>
  <book>
    <title lang="en">Harry Potter</title>
    <price>29.99</price>
  </book>
  <book percentage="99">
    <title lang="en">Learning XML</title>
    <price>39.95</price>
  </book>
</bookstore>

Selecting Nodes

XPath uses path expressions to select nodes in an XML document. The only thing that libxmlb can return are nodes.

Example Description Supported
/bookstore Returns the root bookstore element
/bookstore/book Returns all book elements
//book Returns books no matter where they are
bookstore//book Returns books that are descendant of bookstore
@lang Returns attributes that are named lang
/bookstore/. Returns the bookstore node
/bookstore/book/* Returns all title and price nodes of each book node
/bookstore/book/child::* Returns all title and price nodes of each book node
/bookstore/book/title/.. Returns the book nodes with a title
/bookstore/book/parent::* Returns bookstore, the parent of book
/bookstore/book/parent::bookstore Returns the parent bookstore of book

Predicates

Predicates are used to find a specific node or a node that contains a specific value. Predicates are always embedded in square brackets.

Example Description Supported
/bookstore/book[1] Returns the first book element
/bookstore/book[first()] Returns the first book element
/bookstore/book[last()] Returns the last book element
/bookstore/book[last()-1] Returns the last but one book element
/bookstore/book[position()<3] Returns the first two books
/bookstore/book[upper-case(text())=='HARRY POTTER'] Returns the first book
/bookstore/book[@percentage>=90] Returns the book with >= 90% completion
/bookstore/book/title[@lang] Returns titles with an attribute named lang
/bookstore/book/title[@lang='en'] Returns titles that have a langequal en
/bookstore/book/title[@lang!='en'] Returns titles that have a lang not equal en
/bookstore/book/title[@lang<='zz_ZZ'] Returns titles that lang <= zz_ZZ
/bookstore/book[price>35.00] Returns the books with a price greater than 35
/bookstore/book[price>35.00]/title Returns the titles that have a price greater than 35
/bookstore/book/title[text()='Learning XML'] Returns the book node with matching content