Lexer and Tokens

Lexer and Tokens

Who this is for: parser engineers and performance readers. Consumers only need the token model (the table below) if they read tokens directly; the keyword perfect-hash mechanics live in Internals.

The lexer is a single-pass scalar tokenizer in scalar_lexer.zig. An older two-phase bitmap lexer once lived in lexer.zig; it has been retired, and lexer.zig now contains only the public tokenize* shims (which delegate to scalar_lexer.tokenizeScalarFull) plus the keyword perfect-hash and the numeric/identifier classification helpers. Token streams are byte-for-byte identical to what the retired bitmap lexer produced across the conformance corpus, except on inputs containing invalid UTF-8 (where identifier-span behavior is inherently position-dependent).

Entry points

// lexer.zig — thin shims over scalar_lexer
pub fn tokenize(alloc, source) !TokenizeResult                          // .js, default opts
pub fn tokenizeWithLanguage(alloc, source, lang: Language) !TokenizeResult
pub fn tokenizeWithOptions(alloc, source, lang, is_module: bool) !TokenizeResult
pub fn tokenizeWithAllOptions(alloc, source, lang, opts: TokenizeOptions) !TokenizeResult

TokenizeResult (defined in lexer_helpers.zig) carries the token list plus optional comment trivia:

pub const TokenizeResult = struct {
    tokens: TokenList,
    comment_starts: []const u32,
    comment_ends:   []const u32,
    comment_kinds:  []const u8,   // 0 = line / HTML (Annex B), 1 = block
    comment_count:  u32,
    pub fn deinit(self: *TokenizeResult, allocator) void { … }
};

pub const TokenizeOptions = struct {
    is_module:          bool = false,
    annex_b:            bool = true,                 // gate HTML comments
    comment_sink:       ?*CommentSink = null,        // opt-in comment recording
    publish_to:         ?*std.atomic.Value(usize) = null,  // streaming
    publish_batch_mask: usize = PUBLISH_BATCH - 1,   // PUBLISH_BATCH = 1024
};

scalar_lexer.tokenizeScalarFull produces the TokenizeResult; tokenizeScalarWithOptions is the core implementation.

Token representation

Tokens are stored column-wise (Ast.TokenList = std.MultiArrayList(...)):

Field	Type	Meaning
tag	token.Tag (u8)	token kind
start	u32	byte offset of the token start
len	u32	byte length (end = start + len)
has_newline_before	bool	a line terminator separates this token from the previous one (drives ASI)
has_unicode_escape	bool	the token text contains a \u escape (lets the parser skip a backslash scan in reserved-word checks)

token.Tag (enum(u8), asserted @sizeOf(Tag) == 1) enumerates literals, identifier, all reserved keywords (kw_*), the TS contextual keywords (kw_interface, kw_type, kw_namespace, kw_declare, kw_abstract, kw_implements, kw_readonly, kw_keyof, kw_infer, kw_is, kw_asserts, kw_override, kw_satisfies, kw_module, kw_unique), punctuation, operators, assignment operators, jsx_text, and the specials eof, invalid, escaped_keyword, at_sign, hashbang. Templates are four tags — template_head (`…${), template_middle (}…${), template_tail (}…\``), and template_no_sub(``…` ``).

Tag carries helpers: lexeme() (the canonical text for keywords/punctuators, null for value-bearing tokens), isAssignment(), isKeyword(), and isTsContextualKeyword().

The scan loop

tokenizeScalarWithOptions pre-sizes the token list to max(src.len / 2 + 16, 64) and keeps the hot state in locals (i, prev_kind, saw_nl, at_line_start). Because the token list is a MultiArrayList, the loop caches raw column pointers (p_tag, p_start, p_len, p_nl, p_esc) and refreshes them only when the buffer grows (geometric cap*2 + 16).

Dispatch is per-first-byte. The identifier case is peeled out as a direct hot-path branch before the general switch, because identifiers dominate real code and a leading branch predicts better than a jump-table entry on mixed input. Everything else (digits, quotes, /, `, punctuation, high bytes, whitespace, < in JSX) falls through the switch.

SIMD ASCII-identifier fast path

asciiIdentEnd(src, start, n) scans an identifier run 16 bytes at a time (@Vector(16, u8)):

1. load 16 bytes;
2. lowercase via c | 0x20, test [a-z] as the range [0x61, 0x7A], test [0-9] as [0x30, 0x39], test _/$ by equality;
3. OR the masks; the first non-identifier byte is @ctz(~mask);
4. scalar-finish the tail.

The returned end is the first byte that is not in [A-Za-z0-9_$] — which means any 0x80+ byte or a \ stops the fast path. A pure-ASCII identifier (the overwhelmingly common case) is then a single keyword-hash lookup; only identifiers containing \u escapes or Unicode continue bytes drop to the slow scanIdentRun path, which decodes UTF-8 and consults the ID_Continue tables in unicode_id.zig.

The same 16-byte vector idiom appears in lineTerminatorScan (comment/regex line scanning) and in span.computeLineStarts (lazy line index), each searching for \n, \r, or a 0xE2 lead (the UTF-8 prefix of U+2028 LS / U+2029 PS).

Keyword recognition

After an identifier run, lexer.keywordLookup(text, ts) classifies it with a hand-rolled perfect hash rather than a hash-map probe: a length gate, an O(1) comptime first-char bitset, then a narrow dispatch comparing each candidate as a single packed u64 word — at most ~2 comparisons per lookup. The ts flag selects whether the TS contextual keywords participate (type, keyof, satisfies, …); in plain JS they stay identifier. After ./?. the keyword test is skipped entirely — the next identifier is a property name. (Mechanics: Internals.)

Regex vs. division

/ is context-sensitive: /re/ is a regex literal where an expression may begin, and the division operator otherwise. The lexer decides from the previous significant token via Lex.regexAllowed(prev). The full allow-set (read the switch, not this prose): eof (regex at start of input); the brackets ( [ {; ; , :; =>; ? and ?.; = and every compound assignment; all the binary/unary/relational/equality/logical operators (`+ - * / % ** & | ^ ~ ! <

<< >> >>> == != === !== <= >= && || ??); the expression-leading keywords (return, typeof, void, delete, throw, new, in, instanceof, await, case); and template_head/template_middle(inside${…}). Otherwise /and/=are operators. (Noteyieldis *not* in the set.) This is why the lexer must trackprev_kind— tokenization is not context-free at the/` boundary.

Template literals

Templates nest through ${ … }, tracked up to 16 levels deep (brace_d: [16]u32); deeper nesting is not pushed onto the depth stack. The lexer keeps a tmpl_depth stack (capped at 16) and a per-level brace counter brace_d[]:

• ` calls templateChunkEnd, which scans to the next `, \ (escape), or ${; the result says whether the chunk continues into an interpolation. It emits template_head (more to come) or template_no_sub.
• inside an interpolation, { increments the current level's brace counter;
• a } that closes the interpolation (brace counter back to zero at the current template level) re-enters template scanning and emits template_middle or template_tail, popping the level.

This brace bookkeeping is what lets `${ {a:1} }` distinguish the object literal's braces from the interpolation's closing brace.

JSX lexing

When the language is .jsx/.tsx, the lexer tracks JSX structure so it can suppress regex and comment recognition inside tags and lex attribute strings and text correctly:

• jsx_tag_depth increments on a < that opens a tag (regex context and a following tag-name start or >) and decrements when the tag header closes;
• jsx_brace_nest tracks {…} expression containers inside a tag so a > inside an attribute expression does not close the tag;
• regex is specifically suppressed when the previous token was < or > in JSX mode (layered on top of the normal regexAllowed), so < / > at tag boundaries are never misread as the start of a regex;
• string scanning has a JSX mode (scanStringJsx): JSX attribute strings may span newlines, terminate at <, and treat \ as a literal byte (jsx_no_escape);
• raw text between tags becomes jsx_text tokens, which the parser later coalesces into jsx_text_nodes.

The parser owns the JSX grammar; the lexer only provides the context-correct token stream. See TypeScript and JSX.

Trivia: comments and line starts

• Comments are recorded only when a CommentSink is supplied (opts.comment_sink). The sink keeps three parallel arrays (start, end, kind); Annex B HTML comments (<!--, -->) are gated by is_module and annex_b.
• Line starts are not produced by the lexer. Location lookup is lazy: the diagnostic layer builds a line-start table on first use via span.LineIndex/computeLineStarts, so a clean file that reports no diagnostics never pays for it. computeLineStarts counts every terminator — \n, lone \r, \r\n (coalesced), U+2028, U+2029 — including those inside strings, templates, and comments, matching Location.fromLineStarts.

Numeric and identifier validation

lexer.validateNumericLiteral enforces the full numeric grammar after the span is taken: 0x/0o/0b prefixes with at least one digit, BigInt n suffix rules (no fraction/exponent, not on legacy-octal), numeric separator _ rules (no leading/trailing/double _, none adjacent to ./e/prefix), and decimal fraction/exponent shape. isIdentStartAtPos decodes multi-byte UTF-8 and consults unicode_id.isIdStart, explicitly excluding LS/PS, the BOM (U+FEFF), and Zs-category whitespace (isUnicodeWhitespace, which also treats U+0085 NEL as whitespace, matching TypeScript).

---

Next: AST and Memory Layout · Parser