Testing

Testing

Everything is measured; nothing is promised. ctest --test-dir build --output-on-failure runs all of it.

Unit tests

One test program per subsystem under tests/ (test_layout, test_flex, test_floats, test_forms, test_style_resolver, test_css_parser, test_http, test_connections, test_cache, test_cookies, test_png, test_jpeg, test_gif, test_bmp, test_truetype, test_rasterizer, test_font_manager, and the rest), written with a small CHECK/CHECK_EQ harness. Network tests serve loopback HTTP from a thread and never touch the network. Codec tests build their inputs by hand — the JPEG test carries its own baseline and progressive encoders — and the PNG test re-decodes the encoder's output through an independent inflate.

The JavaScript engine has a test per module: test_js_lexer, test_js_regex, test_js_heap, test_js_strings, test_js_objects, test_js_parser, test_js_interpreter, test_js_runtime, test_js_array, test_js_string, test_js_json, test_js_date and test_js_number, some seven thousand checks, every one run with the heap in stress mode — a collection at every allocation — so that a value held across an allocation without a root fails at once. test262, the ECMAScript conformance suite, is pinned in tests/test262/ and fetched by tools/test262-fetch.sh the way the Web Platform Tests are; test262_runner scores it in ctest and in its own CI job, against a baseline that only grows (Measurements). test_bindings drives a page's realm the way the shell does — parsing with scripts, events from the host, timers on a virtual clock, the hooks a host answers — in 259 checks under the same heap stress; tests/shell/scripting.script drives the shell itself with a scripted page.

Conformance suites

Suite	Score	How it is held
html5lib tokenizer	7032 / 7032	A baseline that only ratchets upward
html5lib tree construction	1784 / 1784	The same
WPT urltestdata	893 / 893	The same
Unicode BidiCharacterTest	91,707 / 91,707	Every case must pass
Unicode BidiTest	770,241 / 770,241	The same
WPT CSS reference tests	see Measurements	tests/wpt/passing.txt names every passing test; a listed test that fails is a regression and fails CI

Reference tests

tests/pages/*.html render to PNGs compared byte for byte against tests/goldens/*.png, on Linux, Windows and macOS, with gcc and with clang. They use the built-in face only, so a golden is one file for every platform. Re-bless with build/tests/reftest tests/pages tests/goldens --update after reading the new picture.

Every page it renders is also walked for the faults a picture cannot show: an element placed more than once where only an inline box may be (two boxes drawing the same content at the same coordinates differ in no pixel), a coordinate that is not a number, a size below zero. Only faults that are wrong on any page belong there — one that fires on correct layout is noise, and noise is how an instrument comes to be ignored — and each has a unit test that builds it by hand and sees it named, since a check that never fires is not a check.

Shell scripts

tests/shell/*.script drive the headless shell (see Scripting the shell); their chrome screenshots are goldens too, compared the same way. live.script needs the network and is run by hand.

The Unicode bidi conformance files

The two files the Unicode Consortium publishes for the bidirectional algorithm — one naming characters, one naming classes — come to 15 MB, so they are fetched rather than committed, at the version pinned in tests/ucd/VERSION. With no checkout the bidi test runs its hand-written cases alone and says so; CI fetches them and runs everything.

bash tools/ucd-fetch.sh                      # the data files at tests/ucd/VERSION
build/tests/test_bidi ucd/BidiCharacterTest.txt ucd/BidiTest.txt

The same checkout is what tools/gen-unicode.cpp reads to regenerate the committed tables. The control there is that every header already committed comes back byte-identical: if one changes, the tool and the data version no longer agree, and nothing new the tool emits can be trusted until they do.

The WPT CSS reference tests

bash tools/wpt-fetch.sh                      # a shallow, sparse checkout at tests/wpt/REVISION
build/tests/wpt_reftest wpt tests/wpt/directories.txt tests/wpt/passing.txt --revision tests/wpt/REVISION

The runner reads the suite's own manifest rules (reference names, -manual/-visual/-crash/-print flags), serves the pages as if from http://web-platform.test:8000/ so shared references and the Ahem font resolve with no network, follows <link rel="match"> and <link rel="mismatch"> chains the way wptrunner does, honors the fuzzy-meta allowances, and renders everything at 800×600 with the built-in face plus the page's own fonts. The whole suite — over 13,000 tests, 20,000 renders — takes about 25 seconds.

Flag	Does
--only <substring>	Run the matching tests only (the baseline is not enforced)
--dump <dir>	Write the test and reference PNGs of every failure — the instrument
--print N	Print up to N failures
--update	Re-bless the baseline
--json <file>, --html <file>	Write the per-directory table (docs/wpt.json, docs/wpt.html)
SASHFOLD_WPT_TRACE=1	Print each render as it starts, to find a hang

A pixel comparison between two renders of the same engine passes when both are equally wrong, so passes are spot-checked with --dump too, and every test that flips is explained before the baseline moves.

Fuzzing

libFuzzer harnesses under fuzz/ for the HTML tokenizer and tree builder, the CSS tokenizer and parser, the URL parser, inflate, the HTTP response parser, and the PNG, GIF, JPEG, BMP/ICO and TrueType readers, with seed corpora. The fuzz build is instrumented end to end — coverage hooks and both sanitizers on every object, undefined behaviour fatal — so a harness sees the code it drives. CI runs them as a smoke pass on every push; the sanitizer lane runs the whole test suite under AddressSanitizer and UndefinedBehaviorSanitizer.

The two checks

tools/pledge-check.sh (the binary's imports against the per-OS allowlist) and tools/egress-check.sh (network primitives behind the one choke point) run in CI and in bash tools/buildall.sh. See The Pledge.

Instruments

• --dump-layout on --render prints the fragment tree; --report writes a JSON account of a load; --bench times the phases. See Running Sashfold.
• --dump-dom prints the parsed tree; --font-info, --font-list and --font-sampler show what the font code sees.