Use Case: Reading a Record List

Input

Consider the following XML:

<Data>
  <Record>
    <Id>1</Id>
    <Name>John Doe</Name>
  </Record>
  <!-- ... millions of records, same structure ...-->
</Data>

Problem

The main goal is to transfrorm xmlSource, a readable utf8 bytes stream representing the input xml, into records: a stream of objects like

//...
{
  Id: "1",
  Name: "John Doe",
}
//...

Basic Solution

const {XMLReader, XMLNode} = require ('xml-toolkit')

const records = new XMLReader ({

  filterElements : 'Record', 

  map            : XMLNode.toObject ({})

}).process (xmlSource)

// await someLoader.load (records)

Explanation

Here:

• an XMLReader is created;
  • with the filterElements that tells him to only handle Record elements (skipping the root Data node);
  • and the map option requiring the XMLNode.toObject transformation;
• the process method implicitly creates an XMLLexer instance, performs all the necessary piping and produces the desired object mode readable stream.

Q&A

How to alter field names?

With XMLNode.toObject's getName option. For example, by setting

map: XMLNode.toObject ({
  getName: s => s.toLowerCase (),
  //...
},

we'll obtain {id: "1", name: "John Doe"} instead of {Id: "1", Name: "John Doe"}.

Are XML namespaces supported?

Yes, but not by default. To calculate js object property names from XML local name and namespace uri, use the 2-argument getName, for example:

map: XMLNode.toObject ({
  getName: (localName, namespaceURI) => `{${namespaceURI}}${localName}`,
  //...
},

How to adjust output records content?

In general, this is what Transform streams are for.

But in simple cases, using XMLNode.toObject's map option is more handy. For example, by setting

map: XMLNode.toObject ({
  map: r => {...r, ord: ++ord}
  //...
},

we'll add the record counter named ord:

{Id: "1",   Name: "John Doe", ord: 1},`
{Id: "989", Name: "Mary Sue", ord: 2},`
...

What are type mapping rules?

Simple: all XML content is presented as primitive js strings except for empty strings that are mapped to null. See XMLNode.toObject for details.

In most cases, that works fine for further load to database in ETL scenarios.

If you need Numbers, Dates etc, transform the output (see above).