Architecture

Architecture

es-parser is a three-phase pipeline — lex, parse, resolve — with one twist that shapes everything else: the parser emits a linear stream of semantic events as it runs, and the resolver replays that stream instead of walking the AST.

        bytes              tokens                 AST + event stream
source ──────▶ Lexer ───────────▶ Parser ─────────────────────────▶ Resolver ──▶ SemanticResult
                                     │                                   ▲
                                     └──── scope / binding / reference ──┘
                                           / control-flow events

Two goals drive the design. The output must be ESTree-compatible, so the JavaScript tooling ecosystem can consume the tree. And the implementation is built for throughput — struct-of-arrays storage, a single-pass lexer with a SIMD ASCII fast path, up-front buffer sizing, and the event stream below — to keep allocation and pointer-chasing off the parse path. Most of the choices below are one of those two goals cashed out.

Modules

The supported entry points are Parser.parse / parseWithOptions and the semantic.SemanticAnalyzer facade; everything else sits beneath them. root.zig re-exports the modules in two tiers: a Stable tier (Lexer, Parser, ast, token, span, diagnostic, scope, symbol, reference, semantic) and an Advanced tier (code_path, layout, parent_builder, scope_events, event_resolver, scalar_lexer, debug) for tooling that reaches under the facade.

Module	Role
lexer.zig	Tokenizer entry points, the keyword recognizer, numeric/identifier classification.
scalar_lexer.zig	The single-pass scalar tokenizer that does the actual scan.
lexer_helpers.zig	TokenizeResult, TokenizeOptions, the regex-allowed table, template-chunk scanning.
token.zig	Token Tag, Language, keyword maps, character classification.
unicode_id.zig, unicode_props.zig	ID_Start / ID_Continue range tables.
parser.zig	The Parser, statement grammar, the driver loop, error recovery, event emission.
expressions.zig	The Pratt expression engine.
typescript.zig	TS type grammar, declarations, speculative parsing, the token-rewrite log.
jsx.zig	JSX elements, fragments, attributes, children.
ast.zig	Node, NodeIndex, the extra_data payload structs, the Ast container.
layout.zig	The comptime Tag → ESTree name table and its C-ABI exports.
span.zig	Span, Location, lazy line-index, SIMD line-start computation.
diagnostic.zig	Diagnostic, Severity, text and JSON formatters.
scope_events.zig	The Event packed struct, EventKind, the event stream.
event_resolver.zig	Replays events into the scope tree, symbol table, reference table, and CFG.
code_path.zig	The control-flow graph builder.
scope.zig, symbol.zig, reference.zig	The SoA semantic tables.
semantic.zig	The SemanticAnalyzer facade, SemanticResult, loop-exitability, the zeroing allocator.
parent_builder.zig	On-demand parent-index construction.

Why a flat event stream beats a scope walk

Scope resolution in the JavaScript world is almost always a second tree walk: acorn parses, then eslint-scope walks the result to bind references. The cost hides there. Many nodes in real source — every literal, operator, and bit of structural punctuation — bind no name and reference nothing, yet the walk visits all of them, paying a tag dispatch and a cache-cold pointer chase per node just to learn each one has nothing to say.

es-parser skips it. The parser already knows, at the instant it happens, when a scope opens, a name binds, or an identifier is used — so it stamps a fixed-size event right there and keeps parsing. Scope collection rides on a pass that was already running; its marginal cost is one array append at the handful of points that matter, not a traversal of the whole tree.

What reaches the resolver is then a flat array, not a tree:

// src/scope_events.zig:100
pub const Event = packed struct(u64) {
    kind: EventKind,  // discriminator
    aux:  u8,         // sub-kind for this event
    _pad: u16 = 0,    // low bit: scope strict-mode flag on scope_open
    node: u32,        // the NodeIndex this event describes
};

kind   35 variants (scope_events.zig:21): scope open/close, declare,
       reference, and the CFG
       markers branch_* / loop_* / try_* / switch_* / logical_* / cond_* /
       if_* / label_* / nop — enough to rebuild scopes AND the CFG
aux    refines the event: a ScopeKind, BindingKind, or ReferenceKind
node   the AST node; names are read lazily from the node, so the event
       itself stays one word — eight events to a 64-byte cache line

The resolver streams this array front to back — an iterative loop dispatching on each event's kind, over the scope-relevant events only, not a recursive descent over every AST node, and touching no literals — and builds the scope tree, symbol table, reference table, and CFG, finishing reference resolution in a short retry pass. The win is in the cost model: the conventional second O(n) tree walk is replaced by O(k) event emits during the parse and an O(k) replay (k the scope-relevant subset), plus the reference passes (an O(refs) counting sort and the unresolved-reference retry). Because the stream is produced during the parse, Parser.parse emits it by default, and the returned AST is ready for the analyzer with nothing in between.

Memory

The whole pipeline is built to keep data contiguous and allocation rare.

SoA everywhere    nodes, tokens, scopes, symbols, references are each a
                  MultiArrayList; a pass over one field streams a dense
                  column and drags nothing else through cache
Flat side tables  variable-arity children live in one extra_data: []u32;
                  a node holds a SubRange window into it, not a heap list
Pre-sized         nodes/extra_data sized to tokens.len * 3/4 (parser.zig:657)
                  so addNode and event pushes write through cached pointers on
                  the fast path; an overflow doubles the buffer and refreshes
                  those pointers
Unmanaged         collections take the allocator per call; they store none

See AST and Memory Layout and Performance.

Error recovery

Parsing never stops at the first error. On a parse error the parser records a Diagnostic, resynchronizes to the next statement boundary (synchronize, parser.zig:1842), and drops in an error-node placeholder so the tree stays structurally walkable for everything downstream. Diagnostics carry a four-tier Severity (diagnostic.zig:5) — error, warning, info, hint — and consumers filter by it. See Parser.

Threading

Every path es-parser executes is single-threaded. The token columns are the one buffer that never grows during a parse — the lexer finalizes them and the parser only borrows the slice — so the raw token pointers the hot paths cache stay valid for the whole parse (this is what keeps the in-place >>→> token rewrite safe). The node and event buffers are pre-sized to make growth rare and refresh their cached pointers on the cold path when it happens.