Use Case: Reading a Record List

Input

Consider the following XML:

<Data>
  <Record>
    <Id>1</Id>
    <Name>John Doe</Name>
  </Record>
  <!-- ... millions of records, same structure ...-->
</Data>

Problem

The main goal is to transfrorm xmlSource, a readable utf8 bytes stream representing the input xml, into records: a stream of objects like

//...
{
  Id: "1",
  Name: "John Doe",
}
//...

Basic Solution

const {XMLReader, XMLNode} = require ('xml-toolkit')

const records = new XMLReader ({

  filterElements : 'Record', 

  map            : XMLNode.toObject ({})

}).process (xmlSource)

// await someLoader.load (records)

Explanation

Here:

• an XMLReader is created;
  • with the filterElements that tells him to only handle Record elements (skipping the root Data node);
  • and the map option requiring the XMLNode.toObject transformation;
• the process method implicitly creates an XMLLexer instance, performs all the necessary piping and produces the desired object mode readable stream.

Q&A

How to alter field names?

With XMLNode.toObject's getName option. For example, by setting

map: XMLNode.toObject ({
  getName: s => s.toLowerCase (),
  //...
},

we'll obtain {id: "1", name: "John Doe"} instead of {Id: "1", Name: "John Doe"}.

How to adjust output records content?

In general, this is what Transform streams are for.

But in simple cases, using XMLNode.toObject's map option is more handy. For example, by setting

map: XMLNode.toObject ({
  map: r => {...r, ord: ++ord}
  //...
},

we'll add the record counter named ord:

{Id: "1",   Name: "John Doe", ord: 1},`
{Id: "989", Name: "Mary Sue", ord: 2},`
...

What are type mapping rules?

Simple: all XML content is presented as primitive js strings except for empty strings that are mapped to null.

In most cases, that works fine for further load to database in ETL scenarios.

If you need Numbers, Dates etc, transform the output (see above).