Token Type Definitions

tokens

Command	Description
tokens tokens GroupName	Begins a new group of token type definitions. Group names are arbitrary. Suggestion: tokens Symbols, tokens Keywords, and general tokens for miscellaneous.
NAME	NAME is both the name and the symbol.
NAME symbol	SYMBOL_PLUS +, SYMBOL_PLUS_PLUS ++. Symbols cannot contain spaces.
NAME(symbol)	EOL(end of line) - alternate declaration where symbols can contain spaces.
NAME symbol [attributes] NAME(symbol) [attributes]	Specify one or more token attributes. Attribute content is built-in and automatically saves the Scanner's internal scan buffer as a Token's content string when a Token is created. Other attributes are defined just by writing them. One common custom attribute is structural to mark end-of-structure token types such as } and endIf.

Example Token Type Definitions

Taken from the Simple-Scanner example.

--------------------------------------------------------------------------------
tokens
--------------------------------------------------------------------------------
EOL(end of line)
IDENTIFIER identifier [content]
NUMBER     number     [content]
STRING     string     [content]

--------------------------------------------------------------------------------
tokens Symbols
--------------------------------------------------------------------------------
SYMBOL_BANG        !
SYMBOL_CARET       ^
SYMBOL_CLOSE_PAREN )
SYMBOL_EQUALS      =
SYMBOL_MINUS       -
SYMBOL_OPEN_PAREN  (
SYMBOL_PLUS        +
SYMBOL_SLASH       /
SYMBOL_STAR        *

--------------------------------------------------------------------------------
tokens Keywords
--------------------------------------------------------------------------------
KEYWORD_PRINTLN println