Validation

Validation

What the test suite actually proves, tier by tier, and which machine proves it.

The four tiers

Cheapest and most diagnostic first:

1. Layout goldens (tests/n8PDF.Tests/Golden/*.json) — every positioned run's coordinates, font and size, compared against a committed trace. A failure names the run that moved and by how much. These prove nothing changed, not that anything is correct. Re-bless after an intended change with N8PDF_BLESS=1 dotnet test — and review the diff, don't just regenerate: a golden updated to match new output proves nothing.
2. Unit and structural tests — font metrics against published values, unit conversions, the style cascade including toggle-property cancellation, and PDF structure read back out of the generated file.
3. Reference comparison — against PDFs exported from Word into Fixtures/Reference/, named after the fixtures. This is the only tier that can say we match Word. Every fixture is required to have a reference: a missing one fails rather than skips, because a skipped comparison is indistinguishable from a passing one. Both PDFs go through one content-stream parser and are compared line by line in points; Fidelity_report writes the full table to artifacts/test-output/fidelity-report.txt.
4. Structural validation — qpdf --check over every converted fixture, verifying the hand-rolled cross-reference table, stream lengths and object graph. A tolerant viewer renders a structurally broken PDF perfectly well; this catches what eyeballing cannot.

Alongside the tiers, a deterministic mutation fuzzer (FuzzTests, #71) runs in the normal suite on every push: thousands of mutated images and hundreds of mutated documents through the two untrusted entry points, seeded from valid files and the crafted hostile corpus the security hardening built, each on a time-bounded thread. Its oracle is the hostile-input contract (Security): no raw runtime crash, no hang, malformed pieces costing their placement rather than the conversion. It is deterministic — fixed seed, generated corpus — so it needs none of Word's faces and passes identically everywhere; a by-hand deep run just raises the iteration counts.

The three second opinions

Each is an independent implementation sharing nothing with this one:

Checker	What it validates	How
qpdf	PDF structure	qpdf --check on every converted fixture
fontTools	Subset fonts	Reads the embedded subsets back
FriBidi	The bidi algorithm	Thousands of random mixed-direction lines compared level-for-level and position-for-position

They are optional locally and required in CI. If installed, run with them required, because CI will:

N8PDF_REQUIRE_QPDF=1 N8PDF_REQUIRE_FONTTOOLS=1 N8PDF_REQUIRE_FRIBIDI=1 \
  dotnet test n8PDF.sln --configuration Release

Beyond those three: HarfBuzz validates shaping (glyphs, advances, offsets to the design unit, across 100+ words in 60+ scripts — with Apple-table faces stripped of AAT first, since HarfBuzz prefers tables this project deliberately reads differently); sips/tiffutil round-trip the image codecs (a file this writes must be readable by macOS's own decoders, and vice versa — which caught real faults round-tripping alone never would); tools/rasterize.swift draws pages with macOS's PDF engine for the things text-position comparison cannot see (drawings, CMYK orientation); and the PDF/A validator an archive would run checks the PdfA claim.

What runs where

The comparison suite is set partly in faces Word brings with it — Calibri, Cambria, and its Japanese and Chinese faces. A hosted runner has not got them:

	ci.yml, every push	full.yml, by hand	A Mac with Word
Runner	macos-15, hosted	self-hosted, labelled word — none registered	local
Documents compared against Word	69 of 143	all 143	all 143 (dotnet test)
Also	qpdf, fontTools, FriBidi, dotnet pack	the three checkers	whatever is installed

There is deliberately no self-hosted runner registered — that would mean a public repository's workflows running on a personal machine — so the 46 Word-face fixtures are only fully compared on a machine that has Word. Which fixtures those are is measured rather than declared (a fixture is on the list when laying it out with and without those faces gives different answers), and OfficeFontTests keeps the list honest at both ends: it regenerates the list where the faces are present, and prints how much was skipped where they are absent. N8PDF_REQUIRE_OFFICE_FONTS=1 turns absence into failure, which the full run sets.

Bear this in mind before trusting a green hosted CI on anything touching font selection or metrics. The hosted runner's own macOS faces are fingerprinted on every run (a runner image shipping a different version of a face moves glyphs by fractions of a point) — the first thing to compare when a golden fails only in CI.

Viewers are not validation

Preview (Apple Quartz) and Chrome (PDFium) agreeing is decent evidence a file is well formed; they are not a substitute for tier 4. Glyph positions come from the content stream, so all conforming viewers agree on geometry; what differs is rasterisation, and the one feature genuinely sensitive to that is synthetic bold (text render mode 2).

Reproducibility

With CreationDate null, the same document converts to identical bytes — which is what makes golden comparison possible, and why nothing in the output path reads the clock or randomness (Promises).