Parser generator based on node-atok
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

Parser builder


Writing parsers is quite a common but sometimes lengthy task. To ease this process atok-parser leverages the atok tokenizer and performs the basic steps to set up a streaming parser, such as:

  • Automatically instantiate a tokenizer with provided options
  • Provide a mechanism to locate an error in the input data
    • track([Boolean]): keep track of the line and column positions to be used when building errors. Note that when set, tracking incurs a performance penalty.
  • Proxy basic node.js streaming methods: write(), end(), pause() and resume()
  • Proxy basic node.js streaming events (note that [data] and [end] are not automatically proxied) and some of atok
    • [drain]
    • [debug]
  • Provide preset variables within the parser constructor
    • atok {Object}: atok tokenizer instance
    • self {Object}: this
  • Provide helpers that simplify parsing rules (see below for description)
    • whitespace()
    • number()
    • float()
    • word()
    • string()
    • utf8()
    • chunk()
    • stringList()
    • match()
    • noop()
    • wait()


It is published on node package manager (npm). To install, do:

npm install atok-parser


A silly example to illustrate the various pre defined variables and parser definition. It parses a flot number and returns the value via its #parse method.

function myParser (options) {
	function handler (num) {
		// The options are set from the myParser function parameters
		// self is already set to the Parser instance
		if ( options.check && !isFinite(num) )
			return self.emit('error', new Error('Invalid float: ' + num))

		self.emit('data', num)
	// the float() and whitespace() helpers are provided by atok-parser

var Parser = require('..').createParser(myParser)

// Add the #parse() method to the Parser
Parser.prototype.parse = function (data) {
	var res

	// One (silly) way to make parse() look synchronous...
	this.once('data', function (data) {
		res = data

	// ...write() is synchronous
	return res

// Instantiate a parser
var p = new Parser({ check: true })

// Parse a valid float
var validfloat = p.parse('123.456 ')
console.log('parsed data is of type', typeof validfloat, 'value', validfloat)

// The following data will produce an invalid float and an error
p.on('error', console.error)
var invalidfloat = p.parse('123.456e1234 ')


  • createParserFromFile(file[, parserOptions, parserEvents, atokOptions]): return a parser class (Function) based on the input file.

    • file {String}: file to read the parser from (.js extension is optional)
    • parserOptions {String}: coma separated list of parser options
    • parserEvents {Object}: events emitted by the parser with their arguments count
    • atokOptions {Object}: tokenizer options

    The following variables are made available to the parser javascript code:

    • atok {_Object_}: atok tokenizer instanciated with provided options. Also set as this.atok DO NOT DELETE
    • self {_Object_}: reference to this

    Predefined methods:

    • write(data)
    • end([data])
    • pause()
    • resume()
    • debug([logger (_Function_)])
    • track(flag (_Boolean_))

    Events automatically forwarded from tokenizer to parser:

    • drain
    • debug
  • createParser(data[, parserOptions, parserEvents, atokOptions]): same as createParserFromFile() but with supplied content instead of a file name

    • data {String | Array | Function}: the content to be used, can also be an array of strings or a function. If a function, its parameters are used as parser options unless parserOptions is set


Helpers are a set of standard Atok rules organized to match a specific type of data. If the data is encountered, the handler is fired with the results. If not, the rule is ignored. The behaviour of a single helper is the same as a single Atok rule:

  • go to the next rule if no match, unless continue(jump, jumpOnFail) was applied to the helper
  • go back to the first rule of the rule set upon match, unless continue(jump) was applied to the helper
  • next rule set can be set using next(ruleSetId)
  • rules can be jumped around by using continue(jump, jumpOnFail). A helper has exactly the size of a single rule, which greatly helps defining complex rules.
// Parse a whitespace separated list of floats
var myParser = [
	'atok.float(function (n) { self.emit("data", n) })'
,	'atok.continue(-1, -2)'
,	'atok.whitespace()'

var Parser = require('atok-parser').createParser(myParser)
var p = new Parser

p.on('data', function (num) {
	console.log(typeof num, num)
p.end('0.133  0.255')

Arguments are not required. If no handler is specified, the [data] event will be emitted with the corresponding data.

  • whitespace(handler): ignore consecutive spaces, tabs, line breaks.
    • handler(whitespace)
  • number(handler): process positive integers
    • handler(num)
  • float(handler): process float numbers. NB. the result can be an invalid float (NaN or Infinity).
    • handler(floatNumber)
  • word(handler): process a word containing letters, digits and underscores
    • handler(word)
  • string([start, end, esc,] handler): process a delimited string. If end is not supplied, it is set to start.
    • start {String}: starting pattern (default=")
    • end {String}: ending pattern (default=")
    • esc {String}: escape character (default=)
    • handler(string)
  • utf8([start, end,] handler): process a delimited string containing UTF-8 encoded characters. If end is not supplied, it is set to start.
    • start {String}: starting pattern (default=")
    • end {String}: ending pattern (default=")
    • handler(UTF-8String)
  • chunk(charSet, handler):
    • charSet {Object}: object defining the charsets to be used as matching characters e.g. { start: 'aA', end 'zZ' } matches all letters
    • handler(chunk)
  • stringList([start, end, separator,] handler): process a delimited list of strings
    • start {String}: starting pattern (default=()
    • end {String}: ending pattern (default=))
    • separator {String}: separator character (default=,)
    • handler(listOfStrings)
  • match(start, end, stringQuotes, handler): find a matching pattern (e.g. bracket matching), skipping string content if required
    • start {String}: starting pattern to look for
    • end {String}: ending pattern to look for
    • stringQuotes {Array}: array of string delimiters (default=['"', "'"]). Use an empty array to disable string content processing
    • handler(token)
  • noop(next): passthrough - does not do anything except applying given properties (useful to branch rules without having to use atok#saveRuleSet() and atok#loadRuleSet())
    • next {String}: next ruleset to load
  • wait(atokPattern[...atokPattern], handler): wait for the given pattern. Nothing happens until data is received that triggers the pattern. Must be preceded by continue() to properly work. Typical usage is when expecting a string the starting quote is received but not the end... so wait until then and resume the rules workflow.
  • nvp([nameCharSet, separator, endPattern] handler): parse a named value pair (default nameCharSet={ start: 'aA0_', end: 'zZ9_' }, separator==, endPattern={ firstOf: ' \t\n\r' }). Disable endPattern by setting it to '' or [].
    • handler(name, value)


A set of examples are located under the examples/ directory.