Scanner Command Reference

A .froley file should have one scanner section which is comprised of any number of routines.

Routines

- routine_name
  ...  # commands

- another_routine
  ...

The topmost routine is the start routine. The Scanner framework repeatedly calls the start routine until a halt command is given. The general approach is to orient the Scanner logic to produce one Token per repetition.

Routines cannot accept arguments or return values (use global variables if needed).

Any tokens created or produced are added to a token queue, which then becomes the input for the [Parser](Parser Command Reference).

Variables

x = "creating a global variable called x"
if (consume(x)) ...

• Variables are created simply by assigning a value to a name.
• All variables are global variables (accessible to all Scanner routines).
• Variables are statically typed. The static type of a variable is implicitly determined by the type of the first value assigned it.
• Variables can be used to pass values into or out of routines.

Built-in Variables

Name	Description
buffer	The internal scan buffer. scan() and collect() commands automatically add characters to the buffer. You can print the buffer out for debugging with println "buffer:" buffer or replace the buffer content with an arbitrary string. For example, buffer = "" will effectively clear the buffer. The buffer can also be assigned to a variable and restored later etc.

Literals

"abc"  # a String
'a'    # a Character
123    # an Integer
true   # Logical true
false  # Logical false

Strings and Characters are mostly interchangeable. The only caveat is if the initial value of a variable is a Character, it will not be able to later store a String.

Patterns

A number of commands accept patterns that are implemented as a limited and custom form of regular expressions.

Pattern Element	Description
{...}	Curly braces are used to explicitly enclose a pattern sequence. They can be nested - {{"ab"}*'c'} matches any number of ab pairs followed by a single c.
"literal string" 'character'	Literal strings and characters are matched exactly.
[...]	Defines a set of single characters, any single one of which can be read to match that pattern element. {'a' [xyz]} matches ax, ay, or az.
[^...]	Matches any single character except those listed - "NOT these".
[A-Z]	Matches any single character between A and Z. Mix and match: [A-Za-z_] etc.
*	Allow zero or more occurrences of the previous element. [a-zA-Z_][a-zA-Z_0-9]*
+	Allow one or more occurrences of the previous element. [0-9]+
?	Allows the previous element to be optional (zero or one occurrence). {{"0x" [0-9A-Fa-f]+}? [0-9]*}

Statements

Statement	Description
++	Increment operator. Increment variables with ++varname or varname++.
--	Decrement operator. Decrement variables with --varname or varname--.
-> TOKEN_NAME -> "symbol"	-> is an alias for produce.
routine-name	Call the named routine. If the routine does not produce or restart then control will return to the following line of the calling routine.
call user-routine-name	Call an arbitrarily-named user routine AKA "native routine". The routine is not defined in the .froley file, but rather in the ScannerCore class that Froley generates. Define the method by overriding it in the customizable Scanner class which extends ScannerCore.
collect "string"	Add the specified literal string to the internal scan buffer.
collect 'character'	Add the specified literal character to the internal scan buffer.
create TOKEN_NAME create "symbol"	Creates a Token of the specified type, storing the scan buffer as the Token's .content:String if the token is defined with the [content] attribute, adds the Token to the output queue, and clears the scan buffer.
discardPosition	Discards the most recent savePosition.
halt	Halts the scanner and signals that tokenization is complete.
markPosition	Notes the current line and column to be used as the source position of the next Token that's created.
mode routine-name	Change the start routine and then continues execution. Write restart after a mode change to switch to that new start routine immediately.
mustConsume(character) mustConsume(string) mustConsume(pattern)	Throws an error if unable to consume (read and discard) the specified sequence.
noAction	Does nothing. Useful as a placeholder single command for if & while.
print arg1 [arg2...]	Prints one or more expressions without a newline.
println arg1 [arg2...]	Prints one or more expressions followed by a newline.
produce TOKEN_NAME produce "symbol"	creates a Token and then restarts the Scanner from the start routine.
restart restart routine-name	Restarts the Scanner from the start routine. A new start routine can optionally be specified.
restorePosition	Rewinds the input to the point of the most recent savePosition. Any tokens created since that point are discarded.
return	Returns from the current routine. If execution reaches the end of a scanner routine and there is not return, execution continues on into the next routine.
savePosition	Saves the current input position (line and column, etc.) to be either rewound to with restorePosition or else discarded with discardPosition.
syntaxError syntaxError "message"	Throws a CompileError with either a default "unexpected input" message or a custom message.

Control Structures

Control Structure	Description
if (condition) ... elseIf (condition) ... else ...	Single line if statement; separate multiple commands with ;.
if (condition)   ... elseIf (condition)  ... else   ... endIf	Multi-line line if statement.
match buffer   produceAny [section]   case "string": ...   case "string"     ...   others: ... endMatch	Match-Buffer.
match input   produceAny [section]   case "string": ...   case "string"     ...   others: ... endMatch	Match-Input.
while (condition) ...	Single line while loop; separate multiple commands with ;.
while (condition)   ... endWhile	Multi-line while loop.

Expressions

Expressions result in a value that can be used as a condition for if or while. They can also be used as standalone statements.

Operators

Operator	Description
( expression )	Standard precedence operator.
{ pattern-sequence }	A pattern sequence - see Patterns.
[ pattern-set ]	A pattern character set element - see Patterns.
+, -, *, /, ^	Add, subtract, multiply, divide, power.
==, !=, <=, >=, <, >	Comparison Operators
not	Logical not - if (not hasAnother) ....
and, or	Logical AND and OR. if (hasAnother and nextIs(';')) ....

Functions

Expression	Description
buffer	The current string contents of the scan buffer - see Variables.
Character(integer)	Create a Character using an integer character code - Character(10) is equivalent to \n.
consume('character') consume("string") consume({pattern})	Read and discard the specified sequence if possible; evaluates to true on success. No characters are removed if there is not a full match.
hasAnother	Evaluates to true if another input character is available.
Integer(character)	Converts a character into its Integer character code. Integer('A') is 65.
nextIs('character') nextIs("string") nextIs({pattern})	Evaluates to true if the next available input matches the specified sequence.
peek peek(lookahead-count)	Returns the next character or the Nth next character, where peek(0) == peek.
read	Reads the next input character.
scan('character') scan("string") scan({pattern})	Works like consume but additionally puts the consumed sequence into the scan buffer.

Match-Input

The match input control structure contains any number of case "symbol" statements. It finds the literal string with the longest match that can be read from the input, then executes the commands in that case. If no cases match then the others block is executed.

match input
  case "+":  produce SYMBOL_PLUS
  case "++": produce SYMBOL_PLUS_PLUS
  others: ...
endMatch

Along with case and others, a match may contain the command produceAny [token-group], which is shorthand for matching and producing each token type defined in the given token group.

For example, given the following tokens Symbols section:

tokens Symbols
SYMBOL_PLUS      +
SYMBOL_PLUS_PLUS ++

The following match-input is equivalent to the match shown above:

match input
  produceAny Symbols
  others: ...
endMatch

Match-Buffer

match buffer is very similar to match input with two differences:

1. The input comes from the characters already collected in buffer
2. Only complete matches are allowed, where the case string completely matches the content of buffer.

Also note that any string variable can be used in place of buffer.

Example

From the Simple-Scanner scanner example.

--------------------------------------------------------------------------------
scanner
--------------------------------------------------------------------------------
- main
  consume [ \r\t]*  # whitespace
  if (consume('\n')) produce EOL

  # single-line comment
  if consume("#" [^\n]*) restart

  if (not hasAnother) halt
  markPosition

  match
    produceAny Symbols
  endMatch

  scan_id_or_keyword
  scan_number
  scan_string

  syntaxError

- scan_id_or_keyword
  if (not scan([_a-zA-Z][_a-zA-Z0-9]*)) return
  match buffer
    produceAny Keywords
    others:    produce IDENTIFIER
  endMatch

- scan_number
  if (not hasAnother) return

  if (scan [0-9])
    scan_integer
    if (scan '.') scan_integer
    produce NUMBER
  elseIf (scan '.')
    scan_integer
    produce NUMBER
  else
    return
  endIf

- scan_integer
  while hasAnother
    if (not scan([0-9]) and not consume('_'*)) return
  endWhile
  return

- scan_string
  if (not consume('"')) return
  while (hasAnother and not nextIs('\n'))
    ch = read
    if (ch == '"') produce STRING
    elseIf (ch == '\\' and hasAnother) ch = read
    collect ch
  endWhile
  syntaxError "Unterminated string."