TokenList.js

An ECMAScript tokenizer

This project is a tokenizer under the LGPL 3.0 license (http://www.gnu.org/licenses/lgpl-3.0.txt).

It consists in a usefull tool to create a parser.

It allows you to define your own keywords.

It supports Space, Comment, Spread (ES6), Namespace, Label, Keyword, Number, String, RegExp, Quasis (ES6) & Symbol tokens.

Compatible with https://github.com/Lcfvs/Sandbox.js

Syntax :

new self.TokenList(filename, source);

Arguments :

[required] String filename : the file url
[required] String source : the file content

The TokenList properties :

Function addKeywords() : the function to add a list of keywords

The TokenList instance properties :

Function item( key ) : the function to retrieve a token

Compatibility :

ECMAScript 3rd Edition +