Implementation Notes

Implementation Notes (Lexer / Parser / VM)

This page is for contributors hacking on the language runtime.

Lexer

Whitespace & Comments

The lexer skips:

• spaces, tabs, CR, LF
• // line comments
• /* ... */ block comments

Tokens

Recognizes:

• integers (decimal with underscores, or hex 0xFF with underscores)

• identifiers / keywords: let, fn, if, else, while, for, in, return, break, continue, throw, try, catch, true, false, nil

• strings (double-quoted, allows escapes)

• operators/punctuation:

  • ()[]{} , ; . : ?
  • + - * / %
  • ! !=
  • = == += -= *= /=
  • < <=
  • > >=
  • &&
  • ||

Source Locations

Tokens carry line and col which update on newline.

Parser

AST Node Types

Statements:

• N_BLOCK
• N_LET, N_ASSIGN, N_SETINDEX
• N_IF, N_WHILE, N_RETURN
• N_FORIN (for-in loops)
• N_BREAK, N_CONTINUE
• N_THROW, N_TRY (exception handling)
• N_EXPR_STMT
• N_FNDEF

Expressions:

• N_BINOP, N_UNOP
• N_TERNARY (ternary operator ? :)
• N_CALL
• N_INDEX
• N_GETFIELD
• N_FUNCLIT
• N_LISTLIT (list literals [...])
• N_MAPLIT (map literals {...})
• N_IDENT
• N_LIT_INT, N_LIT_STR, N_LIT_BOOL, N_LIT_NIL

Assignment Grammar

Assignments are statements only:

• name = expr;
• name += expr; (also -=, *=, /=)
• target[index] = expr;

Parser uses a lookahead approach:

• if a statement starts with IDENT, it tries parsing an lvalue and checks for assignment operators.
• It rewinds lexer state and then parses properly.
• Compound assignments (+= etc.) are desugared to name = name + expr during parsing.

Semicolons

Semicolons are optional in many places; maybe_semi() consumes an optional ;.

VM / Runtime

Values

cs_value carries a cs_type tag and either:

• immediate (bool, int)
• pointer to refcounted heap objects (string, list, map, strbuf, function, native)

Environments

cs_env is a chained scope (linked list via parent), storing parallel arrays of:

• keys[] (malloc'd C strings)
• vals[] (cs_value)

Lookup walks outward; assignment updates nearest existing scope, else creates in current scope.

Functions

A function value stores:

• parameter list
• body AST pointer
• closure env (refcounted)

Strings

Parser stores raw token text (including quotes) for string literals. VM unescapes at evaluation time.

Field Access Behavior

• map.field → map_get(map, "field")
• If a.b.c is used and a is undefined, VM can fall back to a dotted global lookup of "a.b.c" for compatibility with "namespaced globals".

strbuf Methods

Handled as special cases in CALL when callee is GETFIELD:

• append, str, clear, len