Using directives we tell XCHEME how to tokenize an input string in the lexer, how to parse a token in the parser and, in some cases, we want to tell how to ignore some characters like white-spaces, tabs, and so on... Something you must consider when writing your parser is the order of the directives, for example, no matter where a skip directive is placed in the code, skip directives always performs first... There are three main directives you should know about, and both of them are written almost in the same way, let's see how to write them and what's the expected behavior for each one.
The token directive is used to consume an input string and produce an output token (as the directive evaluation result). The output token will compose a token list that will be passed as input to the parser (in the parsing step). It performs in the lexer step, after all the skip directives and before all the node directives, for multiple token directives, the second one performs after the first one and so on.
Syntax:
token <IDENTITY> IDENTIFIER as EXPRESSION;
Example:
token <100> T_FOO as 'foo';
Expect a
'foo'string to produce aT_FOOtoken identified by100.
The node directive is used to consume an input token and produce an output node (as the directive evaluation result). The output node will be attached into the main AST during the parsing step. It performs after all skip and token directives, for multiple node directives, the second one performs after the first one and so on.
Syntax:
node <IDENTITY> IDENTIFIER as EXPRESSION;
Example:
node <200> BAR as T_FOO;
Expect a
T_FOOtoken to produce aBARnode identified by200.
The skip directive is used to consume an input string without producing any output result. All skip directives have no identities or identifiers and therefore cannot be referenced. It performs in the lexer step and before everything, for multiple skip directives, the second one performs after the first one and so on.
Syntax:
skip EXPRESSION;
Example:
skip '\r' | '\n';
Expect a CR or LF character and go ahead without producing any output.
In some cases we want to reuse a token or a node expression to avoid duplicating the source code, but if we declare it as in the examples above, it will produce the directive result (which can be a token or node output) and may become unexpected behavior... So, to achieve the expected behavior without producing any output, you must prefix the directive with the alias modifier, then no token or node output will be produced during the respective directive evaluation. An alias directive performs in its respective token or node step and respecting the reference order.
The alias token directive is used to consume an input string without producing any output token and can be referenced by another token or alias token directive.
alias token ALIAS as EXPRESSION;
token <100> TOKEN as ALIAS;
Identities are optional for alias token directives.
To make an alias token directive more flexible, is possible to set any number of template parameters in the alias statement and further pass the same number of arguments when referencing the respective alias in another directive.
alias <ID, EXPRESSION>
token <ID> ALIAS as opt EXPRESSION;
token <100> TOKEN as ALIAS <10, TOKEN>;
The arguments
10andTOKENare being passed to theALIASreference.
The alias node directive is used to consume an input token without producing any output node and can be referenced by another node or alias node directive.
alias node ALIAS as EXPRESSION;
node <200> TOKEN as ALIAS;
An identity is optional for alias node directives.
Such as an alias token, to make an alias node directive more flexible, any number of template parameters can be specified in the alias statement to be able to receive the same number of arguments when referencing it in a directive.
alias <ID, EXPRESSION>
node <ID> ALIAS as opt EXPRESSION;
node <200> NODE as ALIAS <20, NODE>;
Template arguments
20andNODEare being passed to theALIASreference.
Unlike the normal token or node directive, an alias directive must be referenced by a non aliased directive to take effect, and if that reference need some template parameter, the only acceptable parameter types are identities and references.