Testing

Testing

Everything is measured; nothing is promised. ctest --test-dir build --output-on-failure runs all of it.

Unit tests

One test program per subsystem under tests/ (test_layout, test_flex, test_floats, test_forms, test_style_resolver, test_css_parser, test_http, test_connections, test_cache, test_cookies, test_png, test_jpeg, test_gif, test_bmp, test_truetype, test_rasterizer, test_font_manager, and the rest), written with a small CHECK/CHECK_EQ harness. Network tests serve loopback HTTP from a thread and never touch the network. Codec tests build their inputs by hand — the JPEG test carries its own baseline and progressive encoders — and the PNG test re-decodes the encoder's output through an independent inflate.

The JavaScript engine has a test per module: test_js_lexer, test_js_regex, test_js_heap, test_js_strings, test_js_objects, test_js_parser, test_js_interpreter, test_js_runtime, test_js_array, test_js_string, test_js_json, test_js_date and test_js_number, some seven thousand checks, every one run with the heap in stress mode — a collection at every allocation — so that a value held across an allocation without a root fails at once. test262, the ECMAScript conformance suite, is pinned in tests/test262/ and fetched by tools/test262-fetch.sh the way the Web Platform Tests are; test262_runner scores it in ctest and in its own CI job, against a baseline that only grows (Measurements). test_bindings drives a page's realm the way the shell does — parsing with scripts, events from the host, timers on a virtual clock, the hooks a host answers — in 259 checks under the same heap stress; tests/shell/scripting.script drives the shell itself with a scripted page.

Conformance suites

Suite	Score	How it is held
html5lib tokenizer	7032 / 7032	A baseline that only ratchets upward
html5lib tree construction	1784 / 1784	The same
WPT urltestdata	893 / 893	The same
Unicode BidiCharacterTest	91,707 / 91,707	Every case must pass
Unicode BidiTest	770,241 / 770,241	The same
Unicode LineBreakTest	16,672 / 16,672	The same
WPT CSS reference tests	see Measurements	tests/wpt/passing.txt names every passing test; a listed test that fails is a regression and fails CI
WPT testharness tests	see Measurements	tests/wpt/harness-passing.txt names every passing subtest; the same rule

Reference tests

tests/pages/*.html render to PNGs compared byte for byte against tests/goldens/*.png, on Linux, Windows and macOS, with gcc and with clang. They use the built-in face only, so a golden is one file for every platform. Re-bless with build/tests/reftest tests/pages tests/goldens --update after reading the new picture.

Every page it renders is also walked for the faults a picture cannot show: an element placed more than once where only an inline box may be (two boxes drawing the same content at the same coordinates differ in no pixel), a coordinate that is not a number, a size below zero. Only faults that are wrong on any page belong there — one that fires on correct layout is noise, and noise is how an instrument comes to be ignored — and each has a unit test that builds it by hand and sees it named, since a check that never fires is not a check.

Shell scripts

tests/shell/*.script drive the headless shell (see Scripting the shell); their chrome screenshots are goldens too, compared the same way. live.script needs the network and is run by hand.

Planted-fault controls. A script that passes says the chrome is right only if it would have failed were the chrome wrong. Beside a script, tests/shell/<name>.controls lists faults to plant in the source — a perl substitution each, which must land exactly once — and for each the script lines that must then fail, those and no others. tools/shell-controls.sh plants them one at a time, builds, runs the script and compares: a control matches, or is reported as what went wrong — the plant landing nowhere, the planted tree not building, the script passing, other lines failing. The source is put back after every control, on any exit, and at the start of a run that finds a copy left by one that was killed; --self-test proves the runner tells a match, a mismatch and a plant that landed nowhere apart. A control is written before it is run: the lines it must fail are predicted, not read off the output.

The Unicode conformance files

The files the Unicode Consortium publishes for the bidirectional algorithm — one naming characters, one naming classes — and for line breaking come to 25 MB with the data files, so they are fetched rather than committed, at the version pinned in tests/ucd/VERSION. With no checkout the bidi and line-break tests run their hand-written cases alone and say so; CI fetches them and runs everything.

bash tools/ucd-fetch.sh                      # the data files at tests/ucd/VERSION
build/tests/test_bidi ucd/BidiCharacterTest.txt ucd/BidiTest.txt
build/tests/test_line_break ucd/LineBreakTest.txt

The same checkout is what tools/gen-unicode.cpp reads to regenerate the committed tables. The control there is that every header already committed comes back byte-identical: if one changes, the tool and the data version no longer agree, and nothing new the tool emits can be trusted until they do.

The WPT CSS reference tests

bash tools/wpt-fetch.sh                      # a shallow, sparse checkout at tests/wpt/REVISION
build/tests/wpt_reftest wpt tests/wpt/directories.txt tests/wpt/passing.txt --revision tests/wpt/REVISION

The runner reads the suite's own manifest rules (reference names, -manual/-visual/-crash/-print flags), serves the pages as if from http://web-platform.test:8000/ so shared references and the Ahem font resolve with no network, follows <link rel="match"> and <link rel="mismatch"> chains the way wptrunner does, honors the fuzzy-meta allowances, and renders everything at 800×600 with the built-in face plus the page's own fonts. The whole suite — over 13,000 tests, 20,000 renders — takes about 25 seconds.

Flag	Does
--only <substring>	Run the matching tests only (the baseline is not enforced)
--dump <dir>	Write the test and reference PNGs of every failure — the instrument
--print N	Print up to N failures
--update	Re-bless the baseline
--json <file>, --html <file>	Write the per-directory table (docs/wpt.json, docs/wpt.html)
SASHFOLD_WPT_TRACE=1	Print each render as it starts, to find a hang

A pixel comparison between two renders of the same engine passes when both are equally wrong, so passes are spot-checked with --dump too, and every test that flips is explained before the baseline moves.

The WPT testharness tests

build/tests/wpt_testharness wpt tests/wpt/harness-directories.txt tests/wpt/harness-passing.txt --revision tests/wpt/REVISION

The suite's scripted tests — each file a page (or a .any.js / .window.js script the runner wraps in one, the way the suite's server does, honouring its // META: lines) that loads /resources/testharness.js and asserts, subtest by subtest, what the DOM, the window and the event loop do. The runner serves the checkout as http://web-platform.test:8000/ — and every other host of the suite's two sites, on any port, from the same tree, so a cross-origin test finds its other origin there — applies the server's {{host}}, {{domains[…]}}, {{ports[…]}}, {{location[…]}} and {{GET[…]}} substitutions to .sub. files, sends a .headers file's headers with its resource, runs the page's scripts, and pumps the event loop on a virtual clock (the tasks, then the clock moved to the next timer) until the harness reports or nothing is left to run, with a wall-clock deadline per test. The verdicts are the harness's own: the runner serves its own testharnessreport.js, whose completion callback hands every subtest's name, status and message back through the console. The unit scored is the subtest, as the suite's dashboards count them; a file whose harness never reports contributes none, and the files column of the table says how many did. Tests run side by side (--jobs, half the machine's threads by default); the layout questions a test asks take turns, since there is one font manager. No server-side handler runs, so a test that needs one ends as the harness times it out.

Flag	Does
--only <substring>	Run the matching tests only (the baseline is not enforced)
--print N	Print up to N unreported files and failing subtests
--messages N	The N failure messages that recur most across the suite — the missing piece worth the most subtests is at the top
--files N	The N files failing the most subtests, each with its most common message — where one missing piece stops a whole file
--jobs N	Run N tests at a time
--hang <seconds>	A watchdog: a test running past this is named and the run aborts — a hang in the engine's own code, which no script deadline can stop
--update	Re-bless the baseline. It refuses when a listed subtest no longer passes, naming each; --accept-losses lets them go, by name of the flag — a bless never drops a pass unread
--json <file>, --html <file>	Write the per-directory table (docs/wpt-harness.json, docs/wpt-harness.html)
SASHFOLD_WPT_TRACE=1	Print each test as it starts and ends, and its console, to find a hang
SASHFOLD_WPT_TRACE=2	That, and the event loop: every script and fetch served, timer set and fired, task run, frame opened, closed or navigated, navigation asked, message delivered — why a test waits

The ten slowest tests are printed after the table: a test at the deadline is a hang to look at. The reference-test runner's --update refuses losses the same way.

Fuzzing

libFuzzer harnesses under fuzz/ for the HTML tokenizer and tree builder, the CSS tokenizer and parser, the URL parser, inflate, brotli, the HTTP response parser, and the PNG, GIF, JPEG, BMP/ICO and TrueType readers, with seed corpora. The fuzz build is instrumented end to end — coverage hooks and both sanitizers on every object, undefined behaviour fatal — so a harness sees the code it drives. CI runs them as a smoke pass on every push; the sanitizer lane runs the whole test suite under AddressSanitizer and UndefinedBehaviorSanitizer.

The two checks

tools/pledge-check.sh (the binary's imports against the per-OS allowlist) and tools/egress-check.sh (network primitives behind the one choke point) run in CI and in bash tools/buildall.sh. See The Pledge.

Instruments

• --dump-layout on --render prints the fragment tree; --report writes a JSON account of a load; --bench times the phases. See Running Sashfold.
• --dump-dom prints the parsed tree; --font-info, --font-list and --font-sampler show what the font code sees.
• In a shell script, print-pixel <x> <y> says what the frame holds there and find-pixel <#rrggbb> [<x> <y> <w> <h>] where the frame is exactly that color: how the one pixel of a glyph its color fully covers is found before an assertion is written, and how a color is shown to be nowhere.
• png_diff <before> <after> [--within x y w h]... says where two pictures differ and whether only where they were meant to; png_sheet lays pictures out on one labelled sheet, and tools/theme-sheet.sh <out.png> <theme>... feeds it the chrome under each theme given — ours, or a Firefox or Chrome theme — for judging a look by eye, or a before and an after of a change to it.