Architecture

Architecture

Who this is for: parser engineers and performance readers — the design and module map. Consumers can skip to API Reference.

es-parser is a four-phase pipeline — lex → parse → (events) → resolve — in which the third phase is not a tree walk but the replay of a linear event stream the parser produces as a by-product of parsing. The design is shaped by two goals: ESTree-compatible output for tooling, and throughput competitive with the fastest native JS parsers. Everything below follows from those two goals.

Module map

The public surface and internal modules (src/root.zig):

Module	Role
lexer.zig	Tokenizer entry points (tokenize*) + keyword perfect-hash + numeric/identifier classification. The actual scan lives in scalar_lexer.zig.
lexer_helpers.zig	TokenizeResult, TokenizeOptions, CommentSink, and shared scanning helpers (regex-allowed table, template chunk scan).
scalar_lexer.zig	The single-pass scalar tokenizer (tokenizeScalarFull, tokenizeScalarWithOptions).
token.zig	Tag (token kinds), Language, keyword maps, isIdentChar/isNumericChar.
unicode_id.zig, unicode_props.zig	ID_Start/ID_Continue range tables and lookups.
parser.zig	The Parser struct, statement grammar, the driver loop, error recovery, event emission, streaming hooks.
expressions.zig	The Pratt expression engine (pub fns called on Parser).
typescript.zig	TS type grammar, declarations, speculative parsing, the token-rewrite log.
jsx.zig	JSX elements/fragments/attributes/children.
ast.zig	Node, NodeIndex, the extra_data struct definitions, and the Ast container.
layout.zig	Comptime Node.Tag → ESTree name table + C-ABI exports (ez_tag_name).
span.zig	Span, Location, lazy LineIndex, computeLineStarts (SIMD).
diagnostic.zig	Diagnostic, Severity, text/JSON formatters.
scope_events.zig	The Event packed struct, EventKind, the growable EventStream.
event_resolver.zig	Consumes the event stream → scope tree, symbol table, reference table, CFG.
code_path.zig	The ESLint-style control-flow graph builder.
scope.zig, symbol.zig, reference.zig	The SoA semantic tables produced by the resolver.
semantic.zig	SemanticAnalyzer facade + SemanticResult + loop-exitability analysis + ZeroingAllocator.
parent_builder.zig	On-demand parent-index construction (buildParentsOnly) + parent_fixups replay.
debug.zig, meta_compat.zig	Debug dumps and Zig-version reflection shims.

root.zig splits these into a Stable API (Lexer, Parser, ast, token, span, diagnostic, scope, symbol, reference, semantic) and an Advanced tier (code_path, layout, parent_builder, scope_events, event_resolver, scalar_lexer, debug) for downstream tooling that reaches below the semantic facade.

Note on pub fn. Many Parser methods are pub only because the parser is split across files (expressions.zig, typescript.zig, jsx.zig call back into it); Zig requires pub to cross the file boundary. The supported consumer surface is just Parser.parse / parseWithOptions / parseWithLanguage*. Do not treat mid-parse helpers as API.

Why an event stream instead of a tree walk

A conventional pipeline parses to a tree and then walks it for scope/binding resolution. A large fraction of nodes in real code carry no scope, binding, or reference significance — literals, operators, punctuation-bearing structure — so a full walk spends tag dispatch and cache misses on nodes that contribute nothing to scoping. (The design notes in scope_events.zig cite ~30% such nodes on acorn.js as the motivating measurement.)

es-parser instead emits a linear event stream during the parse (scope_events.zig). Each event is a packed u64:

pub const Event = packed struct(u64) {
    kind: EventKind,  // ~35 variants — scope_open/close, declare, reference,
    aux:  u8,         //   terminator, branch_*/if_*/cond_*, loop_*, try_*,
    _pad: u16 = 0,    //   switch_*, logical_*, label_*, nop
    node: u32,        // NodeIndex the event refers to
};

kind is the discriminator — the full control-flow family (branch_*, loop_*, try_*, switch_*, logical_*, cond_*, if_*, label_*, nop) are EventKind members, which is what lets the resolver reconstruct the CFG from the stream. aux carries the sub-kind for a given kind — ScopeKind for scope_open, BindingKind for declare, ReferenceKind for reference, loop/branch flags otherwise; _pad is unused. Eight events per 64-byte cache line. The resolver (event_resolver.zig) consumes this stream in a single forward pass — no recursion, no per-node tag dispatch, no visiting of literals — and builds the scope tree, symbol table, reference table, and the CFG. The event payload is deliberately minimal: names are resolved lazily from a node's main_token, so the stream stays dense.

This is the central architectural decision. It is what makes the scope/symbol phase cheap, and it is why Parser.parse turns event emission on by default — the returned Ast is ready for the fast-path analyzer with no separate tree-walk pass.

Memory strategy

• SoA everywhere. Nodes, tokens, scopes, symbols, and references are all std.MultiArrayList (column-wise). A pass that needs only one field (e.g. every symbol's name, or every reference's symbol_id) streams a dense column without dragging unrelated fields through cache. See AST and Memory Layout.
• Flat side tables. Variable-arity children live in a single extra_data: []u32; a node stores a SubRange{ start, end } (or a typed ExtraIndex) into it rather than owning a heap list.
• Unmanaged collections. Per Zig 0.16+ convention the allocator is passed explicitly to each mutating call; the structures hold no allocator except where ergonomics demand it (ScopeTree, SymbolTable, ReferenceTable, LineIndex).
• Pre-sizing from token count. The parser sizes its nodes/extra_data buffers to tokens.len * 3/4 up front (~0.75 nodes/token, measured) so the hot addNode/event-push paths hit appendAssumeCapacity. See Performance and Concurrency.

Diagnostics and recovery

Parsing never aborts on the first error. The parser runs in panic mode: on a ParseError it records a Diagnostic, resynchronizes to the next statement boundary (synchronize, SIMD-assisted), and inserts an error_node placeholder so the resulting Ast stays structurally usable. Diagnostics carry a four-tier Severity (error, warning, info, hint); consumers filter by severity. See Parser and diagnostics.

Concurrency model

Every path this library ships is single-threaded. There is concurrency scaffolding — StreamingHooks for a lex‖parse(‖sem) pipeline (the parser-side pump is implemented; the lexer-side publish is not wired) and a ScopeCfgParallel scope/CFG split (the library's only Thread.spawn) — but none of it is driven by the default parse/analyze path, and analyze never calls the parallel resolver. The design constraint behind the scaffolding is still load-bearing for the single-threaded paths: shared buffers are pre-sized to safe upper bounds so they never realloc mid-parse (a realloc would invalidate raw column pointers — and would invalidate a consumer thread's pointers if streaming were wired). For the full status and what would need a driver, see Performance and Concurrency.