A command-line application for processing UK election data.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Forward Ferret

Reverse ferret: “The moment when a newspaper, having reported an event with great conviction one week, says the opposite the following week with equal conviction.”
Roland White, The Sunday Times, July 6, 2008

Forward Ferret: A command-line application for processing UK election data. Built by developers at the Times of London and the Wall Street Journal specifically for converting the Press Association's XML feeds into JSON.

Table of Contents


Make sure you have Node.js v6.2.0 or higher installed. Then run:

npm install forward-ferret -g


Forward Ferret uses a plugin system which is configured using command-line flags.

For example, to convert XML on a remote FTP server to JSON, one could use the following:

ferret \
  --acquire-ftp \
  --ftp-user "YOUR_USERNAME" \
  --ftp-password "YOUR_PASSWORD" \
  --ftp-hostname "FTP_SERVER_ADDRESS" \
  --parse-xml2js \
  --transform-clean-xml \

In this example:

  • The acquire plugin is acquire-ftp, which requires the configuration flags ftp-user, ftp-password, ftp-hostname.
  • The parse plugin is parse-xml2js, which runs the XML files through xml2js.
  • The transform plugin is transform-clean-xml, which removes leftover cruft from the XML file.
  • The output plugin is output-stdout, which effectively logs the output to the command line.

Forward Ferret will not run without specified acquire and output plugins.

See below for a full list of plugins.

Built-in plugins


These plugins acquire data and convert them into arrays of streams.


Get the contents of files stored on a remote FTP server.

  • Flags:
    • ftp-user
    • ftp-password
    • ftp-hostname
  • Output: Array of streams, each containing content of a file


Get the contents of local files.

  • Flags:
    • local-path Glob / directory path, e.g. ~/Documents/Downloads/*.xml
      • Note: On the command line, this must be surrounded by double-quote marks
  • Output: Array of streams, each containing content of a file


These plugins parse arrays of streams and turns them into JSON (or, to be precise, in-memory JavaScript objects).


Convert XML strings to JS objects using the xml2js library.

  • Flags: None
  • Input: Array of streams, each containing raw text of file
  • Output: Array of JS objects


Convert array of streams to array of strings.

  • Flags: None
  • Input: Array of streams, each containing raw text of file
  • Output: Array of JS objects


These plugins transform JavaScript objects into a more desirable format.


Simplify output of xml2js, flattening data in '$' and simplifying single-item arrays.

  • Flags: None
  • Input: Array of streams, each containing a JS object
  • Output: Array of JS objects


A plugin written specifically for the UK's 2016 referendum on EU membership.

  • Flags: None
  • Input: Array of xml2js-created JS objects
  • Output: JS object


These plugins output the resulting data so it can be used elsewhere.


Print out the data to the terminal's 'standard output'. Basically like running console.log in Node.

  • Flags: None
  • Input: JS object
  • Output: JS object


Save an array of strings (or JS objects) as flat files, named sequentially (0.txt, 1.txt, etc).

  • Flags:
    • local-save Local directory to save files in.
  • Input: Array of strings or JS objects
  • Output: Array of strings or JS objects

Using Ferret programmatically

i.e. in a script, rather than on the command line.

npm install forward-ferret --save

Then in a JavaScript file:

const ferret = require('../lib/');
let f = ferret(cli, plugins);

ferret returns a promise, which resolves to the output of the plugin chain.

Developer guide

For those wanting to make their own plungs.


Install Node and npm. Then clone or download this repo and run:

npm install


To run tests:

npm test

Plugin development

Plugin structure

module.exports = {
  type: 'acquire', // see below for list of types
  priority: 0, // higher number means greater precedence
  function: function(inputPromise){
    // main plugin functionality goes here
      // do something with whatever data has been passed in
    // make sure to return a promise!
  // flags passed to the command line interface (using "meow")
  flags: {
    u: 'ftp-user',
    p: 'ftp-password',
    h: 'ftp-hostname'

Plugin types

All inputs and outputs are wrapped in promises.

Type Input Output Example usage
acquire n/a array of streams Fetch XML files from an FTP server (acquire-ftp)
parse stream js object Convert XML data into JS object (parseXml2Js)
transform js object js object 'Munge' data to desired format
output js object n/a Output to stdout (output-stdout)

Version history


  • Fix binary path in package.json


  • Initial public release