Use Case: Reading a Record List

Input

Consider the following XML:

<Data>
  <Record>
    <Id>1</Id>
    <Name>John Doe</Name>
  </Record>
  <!-- ... millions of records, same structure ...-->
</Data>

Problem

The main goal is to transfrorm xmlSource, a readable utf8 bytes stream representing the input xml, into records: a stream of objects like

//...
{
  Id: "1",
  Name: "John Doe",
}
//...

Basic Solution

const {XMLReader, XMLNode} = require ('xml-toolkit')

const records = new XMLReader ({

  filterElements : 'Record', 

  map            : XMLNode.toObject ({})

}).process (xmlSource)

// await someLoader.load (records)

Explanation

Here:

• an XMLReader is created;
  • with the filterElements that tells him to only handle Record elements (skipping the root Data node);
  • and the map option requiring the XMLNode.toObject transformation;
• the process method implicitly creates an XMLLexer instance, performs all the necessary piping and produces the desired object mode readable stream.

Q&A

How to alter field names?

With XMLNode.toObject's getName option. For example, by setting

map: XMLNode.toObject ({
  getName: s => s.toLowerCase (),
  //...
},

we'll obtain {id: "1", name: "John Doe"} instead of {Id: "1", Name: "John Doe"}.

Are XML namespaces supported?

Yes, but not by default. To calculate js object property names from XML local name and namespace uri, use the 2-argument getName, for example:

map: XMLNode.toObject ({
  getName: (localName, namespaceURI) => `{${namespaceURI}}${localName}`,
  //...
},

How to adjust output records content?

In general, this is what transform streams are for.

But in simple cases, using XMLNode.toObject's map option is more handy. For example, by setting

map: XMLNode.toObject ({
  map: r => {...r, ord: ++ord}
  //...
},

we'll add the record counter named ord:

{Id: "1",   Name: "John Doe", ord: 1},
{Id: "989", Name: "Mary Sue", ord: 2},
...

What if record fields are presented as attributes, not nested elements?

The input

<Data>
  <Record Id="1" Name="John Doe" />
  <!-- ... and so on ...-->
</Data>

can be processed with literally same code as shown above with the same result: XMLNode.toObject treats attributes and unique children equally.

However, for a flat namespaceless attribute-only XML like this, there is a faster parsing method described in the next subsection.

Using XMLLexer

Without any child elements in use, one don't have to build any fragment of DOM tree at all. Attribute only XML is very similar to CSV consisting of lines delimited with some char sequences. In this case, low level XMLLexer and SAXEvent can be used instead of XMLReader and XMLNode:

const {XMLLexer, SAXEvent} = require ('xml-toolkit')

const lex = new XMLLexer ()

lex.on ('data', s => {
  const e = new SAXEvent (data); if (!e.isSelfEnclosed) return
  const {attributes} = e
  // ...do whathever with attributes
  //      it's a Map, not a plain Object
})

xmlSource.pipe (lex)

This technique lets us skip creating some objects and thus outperform a bit saxophone (the fastest streaming xml parser for node.js known to author so far) when reading multi-gigabyte files.

The POJO creation is not shown here because that would probably be a wrong step on the way to the maximum productivity: Maps are faster, this is why SAXEvent provides attributes in this form.

What are type mapping rules for field values?

Simple: all XML content is presented as primitive js strings except for empty strings that are mapped to null. See XMLNode.toObject for details.

In most cases, that works fine for further load to database in ETL scenarios.

If you need Numbers, Dates etc, transform the output (see above).

Is it mandatory to fix the node name looked for as filterElements?

No, but it's a simple option, effective in most cases.

Recordsets in the form of XML like

<root>
  <Plant>
    <Id>0</Id>
    <Name>Oats</Name>
  </Plant>
  <Animal>
    <Id>A</Id>
    <Name>Ortolan</Name>
  </Animal>
</root>

are pretty rare. But one can want to have one procedure to read different flat XML documents, with differently named, but similarly arranged nodes.

The filterElements option, other than a string (corresponding to element's local name) can be any function mapping XMLNode to Boolean.

For example, by setting

filterElements: e => e.level === 1,

we'll obtain all root element's children, whatever their names are.

Tuning up filterElements, one can obtain most of XPath filter functionality available for a streaming parser.

Is XMLNode.toObject mandatory as map option?

Not at all, but it's designed to cover most of real life application needs.

Without any map set, the stream will provide parsed content in form of XMLNode instances instead of plain js objects. Application developer is welcome to explore technical details and to build custom mappers.

Is utf-8 the only supported encoding?

If you have XML presented as a binary readable stream compatible with the standard StringDecoder, you can set its encoding as process parameter (effectively, an XMLLexer option):

.process (xmlSource, {
  encoding: 'latin1',
})

For any other encoding, the source is to be preprocessed with something like iconv-lite. This is not an xml-toolkit's task.