Conformance and Testing

Conformance and Testing

es-parser is validated against the major public parser corpora, a property-based fuzz suite, and a whole-corpus robustness sweep. The numbers below were reproduced locally with the pinned toolchain (Zig 0.17.0-dev.607); the README carries the same figures.

Conformance numbers

tc39/test262-parser-tests   5355 / 5355                       (conformance-parser-tests)
    pass/ 1983 + pass-explicit/ 1983 = 3966 must-parse
    fail/ 721  + early/ 668          = 1389
Babel parser fixtures       valid 1928/1928 · invalid 1548/1548   (conformance-babel)
    1740 of 5216 fixtures skipped
TypeScript tests/cases      19120 / 19136                     (conformance-typescript)
    17910 must-parse + 1210 must-reject, 71 skipped

A precision note on test262-parser-tests: the README groups fail + early as "must-reject 1,389", but a syntactic parser is supposed to parse the 668 early/ programs — their errors are early errors, not syntax errors — and the runner counts them as parsed. Either way the suite is 5355/5355. (The runner also skips 8 known-stale fail/ fixtures, so fail/ counts 721 of the 729 on disk.)

What the numbers don't cover:

• The residual 16 TypeScript cases need cross-file type analysis or transpile-level recovery, out of scope for a single-file syntactic parser.
• The Babel rows are over es-parser's supported-feature subset; the ~1,740 skipped fixtures are Flow, the pipeline operator, record/tuple, and other proposals it does not target.
• test262-parser-tests is the complete pass / fail / early set, no skips.

What CI gates vs. what's measured by hand

CI (.github/workflows/ci.yml) runs zig build test — units plus the file-based suites plus the bundled test262-parser-tests runner — and zig build conformance-semantic. The Babel, full-TypeScript, and full-test262 runners are not invoked in CI — a deliberate choice, not a submodule- availability limit (CI does check the submodules out). Treat those three as point-in-time measurements, not continuously-enforced gates.

Test suites

zig build test (build.zig:95) runs six things:

embedded unit tests in src/                          (build.zig:24)
tests/parser_test.zig                                (build.zig:35)
tests/lexer_test.zig                                 (build.zig:46)
tests/semantic_test.zig                              (build.zig:57)
tests/fuzz_test.zig — regression mode, replays the corpora   (build.zig:71)
conformance-parser-tests — bundled tc39 runner       (build.zig:91)

Fuzzing

tests/fuzz_test.zig uses Zig's built-in fuzzer. Run plainly it replays the seed corpora as a regression gate; under the coverage-directed engine it mutates:

zig build test --fuzz

It fuzzes the whole pipeline across language and option settings — js, module, ts, jsx, tsx, global_return, annex_b off, experimental decorators, dts, the lexer, and the semantic pipeline (with and without CFG, with parents) — plus a short-buffer variant for EOF edges. The contract under fuzz is robustness: no crash, no OOM, on arbitrary input. The engine is seeded from five curated corpora — js_corpus, ts_corpus, jsx_corpus, sem_corpus, lex_corpus — and lex_corpus includes deliberately truncated UTF-8 (\xE2, \xE2\x80, \xC3, \xF0\x9F) so malformed encodings are exercised from the first run.

Whole-corpus robustness sweep

zig build conformance-semantic runs the full parse + scope/symbol/reference/CFG pipeline over the ~19k-file TypeScript corpus. It is a robustness gate, not a correctness one: there is no expected-output comparison; it tallies structural counts and fails only if a file crashes or OOMs. The local run processed 19,233 files with 0 crashes. The diagnostic tally is nonzero because many files carry expected errors — transpile-only cases, constructs that require module mode — which the sweep does not treat as failures.

Running the external corpora

The three large corpora are git submodules. Initialize the one you want, then run its step (each runner has a built-in default fixture path, so no arguments):

zig build conformance-parser-tests          # bundled; also in `zig build test`

git submodule update --init tests/conformance/test262
zig build conformance-test262

git submodule update --init tests/conformance/babel
zig build conformance-babel

git submodule update --init tests/conformance/typescript
zig build conformance-typescript            # input set: tests/cases
zig build conformance-semantic              # robustness sweep over the same corpus

The runners live in tests/conformance/ (wired at build.zig:131), each built ReleaseFast over one shared ReleaseFast build of the parser.

CI cadence

ci.yml runs on push/PR to main across Linux and macOS against the pinned Zig. nightly.yml runs daily against Zig master as a forward-compatibility canary and opens a labeled issue on a break; the pinned-version CI is the source of truth.