Conformance and Testing

Conformance and Testing

Who this is for: consumers evaluating whether to trust es-parser, and contributors running the gates.

es-parser is validated against the major public parser test corpora plus a property-based fuzz suite and a whole-corpus robustness sweep.

Conformance numbers

From README.md:

Suite	Result
tc39/test262-parser-tests	must-parse 3,966 / 3,966 · must-reject 1,389 / 1,389
TypeScript compiler tests (tests/cases)	19,120 / 19,136
Babel parser fixtures — valid	1,928 / 1,928
Babel parser fixtures — invalid (correctly rejected)	1,548 / 1,548

Caveats (stated in the README)

• The residual TypeScript failures require cross-file type analysis or transpile-level error recovery — out of scope for a single-file syntactic parser.
• The Babel numbers are over the supported-feature subset: roughly 1,740 of the 5,216 parser fixtures are skipped — Flow syntax, the pipeline operator, and assorted Stage-x proposals (record/tuple, module blocks, do-expressions, …), none of which es-parser targets. Babel's TypeScript-specific fixtures are also skipped there (they run under the TypeScript suite instead).
• The test262-parser-tests rows are the complete pass/ + fail/ set (no skips); it is the one corpus run in CI on every push (below).

What CI actually gates vs. what's measured manually. CI runs only zig build test (which includes the test262-parser-tests runner) plus the conformance-semantic robustness sweep. The Babel, TypeScript, and full test262 numbers come from the standalone conformance-* runners, which are not run in CI (they need manually-initialized submodules). Treat those three as point-in-time measurements, not continuously-enforced gates.

Test suites

zig build test runs (see build.zig):

• Unit tests embedded in src/ (the test {} blocks across ast.zig, token.zig, span.zig, diagnostic.zig, scope.zig, symbol.zig, reference.zig, semantic.zig, layout.zig, lexer.zig, parser.zig, …).
• tests/lexer_test.zig, tests/parser_test.zig, tests/semantic_test.zig — focused suites against the library module.
• tests/fuzz_test.zig — in regression mode each corpus entry is fed once.
• conformance-parser-tests — the bundled tc39 test262-parser-tests runner (its submodule ships with the package).

Fuzzing

tests/fuzz_test.zig uses Zig's built-in fuzzing (std.testing.Smith). Run normally it replays the corpus as a regression gate; with the coverage-directed engine it mutates continuously:

zig build test --fuzz

The harness fuzzes the full pipeline across many language/option settings — parse js, parse js module, parse ts, parse jsx, parse tsx, parse global_return, parse annex_b disabled, parse ts experimental decorators, parse dts, lexer all languages, and the semantic pipeline (semantic, semantic ts, semantic no-cfg, semantic with parents, semantic annex_b disabled). It includes a short-buffer variant (fuzzBytesShort) to stress EOF-boundary conditions. The contract under fuzz is robustness: no crash, no OOM, on arbitrary input.

The mutation engine is seeded from five curated corpora of edge cases — js_corpus, ts_corpus, jsx_corpus, sem_corpus, and lex_corpus. The invalid-UTF-8 robustness claim is concretely seeded: lex_corpus includes deliberately truncated/incomplete byte sequences (\xE2, \xE2\x80, \xC3, \xF0\x9F) alongside valid multi-byte identifiers, so malformed encodings are exercised from the first run, not only after mutation.

Whole-corpus robustness sweep

zig build conformance-semantic runs the entire parse + scope/symbol/ reference/CFG + redeclare pipeline over the ~19k-file TypeScript corpus. It is a robustness gate, not a correctness gate — it has no expected-output comparison; it tallies structural counts (scopes/symbols/refs/diagnostics) and fails if any file crashes or OOMs the pipeline.

Running the external corpora

The three large corpora are git submodules (.gitmodules). Initialize the one you want, then run its step (each runner uses a built-in default fixture path — no arguments):

zig build conformance-parser-tests   # bundled; also part of `zig build test`

git submodule update --init tests/conformance/test262
zig build conformance-test262

git submodule update --init tests/conformance/babel
zig build conformance-babel

git submodule update --init tests/conformance/typescript
zig build conformance-typescript     # input set is tests/cases
zig build conformance-semantic       # robustness sweep over the same corpus

The runners live in tests/conformance/ (parser_tests_runner.zig, test262_runner.zig, babel_runner.zig, typescript_runner.zig, semantic_runner.zig); each is built ReleaseFast and imports one shared ReleaseFast build of the parser.

CI

.github/workflows/ci.yml runs on push/PR to main across ubuntu-latest and macos-latest. It checks out submodules, installs the pinned Zig (0.17.0-dev.607+456b2ec07), caches .zig-cache, then runs:

1. zig build test --summary all — units + conformance (test262-parser-tests);
2. zig build conformance-semantic — the robustness gate over the TypeScript corpus (a hard crash aborts the build; a caught failure exits non-zero).

.github/workflows/nightly.yml runs daily against Zig master on both OSes as a forward-compat canary and opens a labeled nightly-failure issue on a break. The pinned-version CI is the source of truth; nightly is informational.

---

Next: Building and Integration · Home