High level design overview
The parser is a slightly modified fork of the esprima harmony branch. It has two important functions:
read converts a string to a token tree. A token tree is similar to tokens produced by the standard esprima lexer but with the critical difference being that token trees match delimiters. So the standard esprima lexer would transform the string "{ 42 }" into three tokens:
[{
type: 7, // the punctuator type
value: "{"
},
{
type: 6, // the numeric literal type
value: 42
},
{
type: 7,
value: "}"
}]But read will transform the same string into a single {} token tree with inner tokens:
[{
type: 11, // the delimiter type
value: "{}",
inner: [{
type: 6,
value: 42
}]
}]parse converts a token tree to an AST that conforms to the Parser API
Note that parse only understands the JavaScript grammar. All macros must be expanded from the token tree before calling parse.
This is where expansion happens. The expander takes a token tree, finds and loads any macro definitions, and the expands any invocations of those macros. The important functions in the expander are expandToTermTree, expandTermTreeToFinal, and enforest.
To understand how things work in here you might want to read "Macros that Work Together" (which explains how hygiene works among other things) and "Honu: A Syntactically Extensible Language" (which explains how to do expansion for non-lispy languages).
Enforest was first described in the honu paper. The idea is to look at the first few tokens in the token tree and convert them into a pseudo-AST called a term tree. Whereas a token tree is just flat tokens and nested delimiters, a term tree are complex objects that look similar to an AST like ForStatement or FunctionExpression. It basically adds structure to the mostly flat token tree. This is also where macros get expanded.
This is similar to parse1 in the honu paper. It calls enforest repeatedly until the entire token tree has been converted into a term tree. Each time enforest is called the result is checked; if the result is a macro definition the definition is loaded into the environment so it can be used later.
This is similar to parse2 in the honu paper. It takes the term tree generated by expandToTermTree and does some final processing to complete the expansion. Mainly this means handling issues of hygiene for function definitions.
This is where the pattern matching functions live. They get called from macro definitions when a macro is invoked to bind syntax to pattern variables and substitute syntax into a template.
This is where the definitions for syntax objects live. Some hygiene related stuff is here too.
This ties everything together. It provides some convenience functions like compile that take a string and calls read, expand, parse, and generate to produce a string with all the macros expanded.