Lexer and Tokens

Lexer and Tokens

The lexer is single-pass and scalar: one loop, one switch on the first byte of each token, producing a struct-of-arrays token list. SIMD is used only for the handful of inner scans where it pays — identifier runs, line-terminator hunts, comment and template bodies. scalar_lexer.zig does the work; lexer.zig is a thin shim over it (the two-phase bitmap lexer it once held has been retired).

Entry points

lexer.zig exposes the public tokenize* functions, and each delegates to the scalar lexer:

Lexer.tokenize(alloc, src)                    // .js, script mode    (lexer.zig:24)
Lexer.tokenizeWithLanguage(alloc, src, lang)  //                     (lexer.zig:27)
Lexer.tokenizeWithOptions(alloc, src, lang, is_module)  //           (lexer.zig:30)
Lexer.tokenizeWithAllOptions(...) ── delegates ─▶ scalar_lexer.tokenizeScalarFull
                                                  (lexer.zig:33 → :42)

The driver is tokenizeScalarWithOptions (scalar_lexer.zig:532). Two wrappers sit on top: tokenizeScalar (:493) for the parse-only path, and tokenizeScalarFull (:501), which additionally collects comment trivia.

Tokens

A token is five SoA columns (scalar_lexer.zig:898):

tag                 Tag (enum(u8))
start               u32 — byte offset into source
len                 u32 — byte length
has_newline_before  bool — drives ASI and the regex/division decision
has_unicode_escape  bool — set for names containing \uXXXX

Because the token list is a MultiArrayList, each column is contiguous — ASI, for instance, streams has_newline_before alone. Tokens are owned by the TokenizeResult, not the Ast (lexer_helpers.zig:29); the caller frees the lexer result separately. TokenizeResult also carries the optional comment trivia as parallel comment_starts / comment_ends / comment_kinds columns (lexer_helpers.zig:22). The lexer does not compute line starts — those are built lazily on the first diagnostic by span.LineIndex (span.zig:79).

The scan loop

After hashbang handling (#!, valid only at byte 0), the loop dispatches on the first byte (scalar_lexer.zig:579). Whitespace, newlines (\r\n coalesced), and ASCII identifier-start bytes are peeled out ahead of the jump table; the identifier branch is deliberately kept off the switch to avoid a branch-target misprediction on the hottest token class. Everything else — single-char operators, digits, strings, templates, /, <, -, ., \, and the high-byte / Unicode cases — goes through one switch (c) (scalar_lexer.zig:623), a single indirect branch. A shared tail writes the token through cached column pointers and maintains JSX depth.

SIMD scans

Every vector scan is 16 bytes wide (@Vector(16, u8)), each with a scalar tail:

asciiIdentEnd        identifier run → first non-[A-Za-z0-9_$] via @ctz   (scalar_lexer.zig:223)
lineTerminatorScan   \n / \r / 0xE2 (LS/PS lead byte)                     (scalar_lexer.zig:90)
blockCommentEnd      * / \n / \r / 0xE2                                   (lexer_helpers.zig:82)
templateChunkEnd     ` / \ / $                                           (lexer_helpers.zig:135)
span.computeLineStarts   \n / \r / 0xE2                                   (span.zig:85)

Whitespace skipping in the main loop is scalar, not vectorized.

Regex vs. division

A / after a value is division; after an operator or keyword it opens a regex literal. The lexer decides from the previous token's tag via regexAllowed (lexer_helpers.zig:248) — where "previous tag" is normalized first: a keyword used as a property name after . / ?. is recorded as an identifier, so x.in / y divides. The complete set is in source; the easy-to-miss members are eof, arrow (=>), question (?), question_dot (?.), and template_head / template_middle (for ${ /re/ }). await is in the set; yield is not — a / after yield lexes as division.

In JSX, the rule is suppressed after < or > so that </tag> lexes its slash rather than starting a regex:

if (regexAllowed(prev) and !(is_jsx and (prev == .less_than or prev == .greater_than)))
// scalar_lexer.zig:779

Template literals

Template nesting is tracked in a fixed brace_d: [16]u32 with a tmpl_depth counter (scalar_lexer.zig:547). A ` with interpolation pushes a level only while tmpl_depth < 16 (:662); inside a template, { bumps the current level's brace count and } either resumes chunk scanning at the interpolation boundary or closes a brace. templateChunkEnd SIMD-scans each chunk and reports whether it ended at ` (tail) or ${ (middle); an unterminated chunk becomes an .invalid token.

JSX lexing

In a JSX context the lexer hands tag structure to the parser as ordinary tokens and lexes element text through scanStringJsx (scalar_lexer.zig:36), which terminates JSX text and attribute strings at < and may span newlines. JSX text is therefore emitted as .string_literal tokens — the .jsx_text token tag that exists in token.zig is never produced by this lexer; the parser builds the jsx_text_node AST node from these tokens (see TypeScript and JSX). The emit tail tracks JSX tag depth and brace nesting so <, >, and { … } headers are delimited correctly (scalar_lexer.zig:909).

Keyword recognition

Keyword recognition is a staged comparison, not a hash lookup — no hash is computed (a per-length first-char bitmask is indexed, then bytes are compared). keywordLookup (lexer.zig:210) gates in stages so an identifier, the common case, exits in a couple of comparisons:

1. length < 2 or > 10                        → identifier
2. first byte outside a..z                   → identifier
3. KW_FC_MASK[len] bit for first char unset  → identifier   (lexer.zig:178)
4. switch(len) → switch(first byte), then compare the candidate's bytes
   packed little-endian into a u64 (loadU64 / pK)           (lexer.zig:77, :51)

KW_FC_MASK is a comptime first-char bitset per length, and the packed-u64 compare replaces a byte-by-byte memcmp with one integer equality. The hot path also skips the lookup entirely after . / ?. — a property name is never a keyword — gated by isPropertyAccess(prev) (scalar_lexer.zig:621). The token.keywords / token.ts_keywords string maps are used only on the rare escaped-identifier path (\u-containing names).

Comments

When a CommentSink is supplied (lexer_helpers.zig:41), the lexer records each comment as three parallel columns — starts, ends, kinds (kind 0 = line or Annex-B HTML, kind 1 = block) — and tokenizeScalarFull transfers them into the TokenizeResult. The parse-only path leaves the sink null and pays nothing for trivia. Unterminated block comments are emitted as .invalid tokens rather than recorded.