-
Notifications
You must be signed in to change notification settings - Fork 0
stdlex
Anssi Halmeaho edited this page Dec 10, 2022
·
1 revision
Provides access to FunL interpreter lexer.
Tokenizes string given as argument to list of tokens (represented as maps).
Return value is list:
- bool: true if tokenizing was successful, false otherwise
- string: error description if tokenizing failed
- list: contains map for each token
Format:
call(stdlex.tokenize <source-text:string>) -> list(<ok> <err> <tokens-list>)
Each token is map which has following key-values:
Key | Value |
---|---|
'type' | type of token (string) |
'value' | value of token as string |
'line' | line of source code (int) |
'pos' | position in line of source code (int) |
Following token types are supported:
Token type | Meaning |
---|---|
'tokenSymbol' | symbol |
'tokenAs' | as keyword |
'tokenStartNS' | ns keyword |
'tokenEndNS' | endns keyword |
'tokenNumber' | number |
'tokenString' | string |
'tokenDot' | "." -character |
'tokenOpenBracket' | opening bracket |
'tokenClosingBracket' | closing bracket |
'tokenComma' | "," -character |
'tokenTrue' | true (bool) |
'tokenFalse' | false (bool) |
'tokenFuncBegin' | func keyword |
'tokenProcBegin' | proc keyword |
'tokenFuncEnd' | end keyword |
'tokenImport' | import keyword |
'tokenEqualsSign' | "=" -character |
'tokenExpander' | ":" -character |
'tokenLineComment' | line comment (started with #) |
'tokenMultiLineComment' | multiline comment (/*, */) |