v0.5.0 — a scanned form is a fax, and a fax was a blank page

A scanned form is a fax, and a fax was a blank page

67 of the 1 633 real forms in the corpus — the eleven issuing bodies, not the vendor test suites — carry a CCITT-encoded image, 263 images between them, and 6 of their pages have nothing else on them at all. Those pages drew entirely blank, and a blank page is the failure a reader notices before any other.

Group 3 one-dimensional, Group 3 mixed and Group 4 are all decoded, with /K, /Columns, /Rows, /BlackIs1, /EncodedByteAlign and /EndOfBlock read.

Why it belongs in the reader

The other filters this package stops at carry an image with its own idea of how many components it has and how deep they are. A fax does not: it produces bilevel samples, one bit a pixel, rows padded to a byte, and the stream dictionary says what they mean. That is a byte stream, so it is a filter — and decoding it here means every caller gets it rather than each writing its own.

ImageFilter no longer returns true for /CCITTFaxDecode or /CCF. That is a change to a documented contract: a caller that switched on the filter name and handled the fax itself will now receive samples with an empty filter name instead.

The reference, read before writing

ITU-T T.4 and T.6, by way of the tables and the changing-element algorithm in golang.org/x/image/ccitt — read, not imported, because this package has no dependencies outside the standard library and gains none. The 218 code table entries were extracted mechanically from that package's gen.go rather than retyped, because a table of 218 variable-length codes transcribed by hand is a table with a mistake in it.

Measured against that reference, image by image

Every CCITT image in the corpus, decoded by both and compared byte for byte, with both sides working the height out for themselves where /Rows is absent:

	images	byte-identical	differ	reference refuses
1 633 real forms	263	260	0	3
arXiv sample	47	47	0	0

The three the reference gives up on — "invalid code", "invalid offset" — are pages 2, 3 and 4 of fr-cerfa/cerfa_10455.pdf, where it stops and we decode all 3 504 rows, exactly the /Height the image dictionary names. Being more forgiving is deliberate: a damaged scan comes back as far as it got, because refusing it turns a form into a blank page.

Two things the measurement found that reading the specification did not

Zero bits decode as perfectly good two-dimensional modes — pass mode is 0001, vertical-left-3 is 0000010 — so a decoder that does not stop at the padding goes on inventing rows that look like the last real one. fr-cerfa/cerfa_10701 gave 1 636 rows for a 208-row image that way, every surplus row with plausible ink on it.

The fuzz target found the second: on a truncated fax the early return skipped the padding that makes the answer as long as /Rows promised, so two bytes of input gave one row where the caller had asked for four, with no error to say so. 6.7 million fuzz executions since, none failing.

What it changes when a page is drawn

Through render, on the first page of all 1 633 real forms, 69 pages change: one goes from blank to drawn (fr-cerfa/cerfa_11818, 0 inked pixels to 28 989), fourteen gain ink, and fifty-one lose ink.

That last number is the interesting one. Those pages were not blank before — they were noise. 273 of the corpus's image masks carry an encoded filter, and render handed a mask to its stencil path before asking whether the bytes were still compressed, so a fax was drawn as a stencil of its own compressed bytes. Decoding it replaces the noise with the picture, which has less ink in it because most of a form is white. Fixed on the other side in go-pdfkit/render#20; the 9 JBIG2 masks that remain are now not drawn rather than drawn wrong.

The same file drew a different picture on different runs

An inline image dictionary may carry both spellings of a key and disagree with itself — /W 20 beside /Width 10 — and expanding it in one pass over the map let Go's deliberately randomised iteration order pick the winner. Eight renders of safedocs' Inline_Image_Abbreviations fixture gave five different answers; now they give one. Measured scope: of 3 668 corpus files rendered three times over, exactly one disagrees with itself, and it is the fixture built for this case.

100% statement coverage, go vet and -race clean, nine cross-compile targets, nothing outside the standard library.