v0.11.0

CheckLinearized: asking whether a file is what its front says

Linearized(b) has been here since v0.10.0 and answers the cheap question a reader asks before streaming: are the parameters at the front well formed? Of Table F.1's claims it checked exactly one against the file — /L against the byte count — because that is all a reader can afford, having fetched 1024 bytes and resolved nothing.

That makes it a weaker test than it looks, and a writer developed against it would pass without linearizing anything: /O, /E, /N and /T were read for type and never consulted, so a file with a correct /L and four invented integers was accepted. That is precisely what a first linearizer emits.

CheckLinearized(b) is the other question, for the other caller: not "may a reader trust the front of this file" but "is this file actually what the front says". It opens the document, so it costs what opening costs, and it is the judgement a writer needs on bytes it has just produced. Every parameter must point at what it claims to:

	checked against
/N	the document's real page count
/O	the real first page, and that a reader will treat it as page one
/H	a real stream object, its extent matching exactly
/T	the main cross-reference section — either of the two meanings Table F.1 gives it
/E	the end of the last object the first-page cross-reference section lists

/E, and why a second producer was needed

/E is the parameter a writer is most likely to get wrong, so leaving it ungated was the hole that let a writer be its own judge. It is now checked against the first-page cross-reference section — the one Annex F puts at the front and startxref points at, whose entries are the first page's objects.

The subtlety is that F.3.1 permits the primary hint stream on either side of the first page. qpdf writes it before (/H 550, /E 902); Ghostscript -dFastWebView writes it after, at /E exactly (/H 829, /E 829). A rule that simply took the highest-offset entry would therefore refuse every Ghostscript file. Both qpdf fixtures place it before and cannot tell the two rules apart, which is why a Ghostscript-produced file is committed beside them.

Values a 32-bit build cannot address are refused, not truncated

int(Integer) discards the high half where int is 32 bits, so a /H length of 2³²+127 read as 127 — the committed fixture's own hint length, a plausible value every later check then accepted. The same bytes were judged sound on one machine and refused on another, with nothing to say which. Every integer the dictionary offers now goes through one conversion that refuses what it cannot represent, /P included. The new error is ErrLinearizationMalformed.

Known limits, stated rather than implied

• /E is checked against the first-page cross-reference section, which is not the same as checking the section is right. A writer that omits objects page one genuinely needs produces an internally consistent file that passes. Ghostscript does this on a document with shared fonts, outlines and links: /H = /E = 1941 is accepted here, while qpdf --check computes 5500..5501 and reports page one as 9 objects where the hint table claims 4. Deciding what page one needs means walking its resources — the partition the hint tables describe — and that judgement is still outstanding.
• The hint tables' contents are not decoded or compared with the file; only /H's extent is checked.
• /P is range-checked but nothing cross-checks its value.

One deliberate disagreement with qpdf

qpdf computes /E as a range (901..902 on the committed fixture), accepting the end with or without the line ending after endobj. This requires the end including it, which is what both measured producers write. A file carrying the lower value would pass qpdf --check and be refused here; none has been observed, and the alternative is a window that cannot tell a correct /E from one a byte short — the same argument that made the /H length an exact comparison.

Across the rest, there are 0 disagreements with qpdf over 42 generated files: 1, 2, 3, 5, 25 and 100 pages × plain, object streams, uncompressed streams, 256-bit encryption and --newline-before-endstream.

All limits are tracked in #40.

v0.10.0

What's Changed

• ci: install qemu-user, and resolve the emulator this matrix names by @tannevaled in #36
• reader: read and check a file's linearization (ISO 32000-1 Annex F) by @tannevaled in #37

Full Changelog: v0.9.0...v0.10.0

v0.8.0 — an Operation says which bytes of the stream it is

Operation gains Start and End: the byte range of the operation in the
stream it was scanned from — Start at its first operand (or at the operator
when it has none), End just past the operator.

⛔ They are here so a content stream can be edited in place

A tool that wants to put something before every BT has two honest ways to
find one:

• re-serialise the whole stream, which risks changing what an inline
  image's bytes mean;
• know where the operator was — and this scanner is the only thing that
  does.

BT appears inside strings, inside names, and inside an inline image's data,
and a byte search cannot tell those from an operator. The test pins the
discriminator:

(BT in a string) Tj  BT /F1 12 Tf ET
 ^ byte search: 1                      ← inside somebody's text
                     ^ the scan: 20     ← the operator

A tool splicing at index 1 corrupts the string.

Two details that are easy to get wrong

• End is the lexer's position, not the operator's start plus its length.
  An inline image's operator token is BI and its operation runs to the EI
  three keywords later, so measuring from the operator would put the end in
  the middle of the image's data — where a splice corrupts the image instead
  of the stream around it. Asserted;
• Start is the first operand's position, so splicing before an operation
  does not land between its operands and its operator. 1 0 0 RG starts at
  the 1, not at the RG.

The ranges of a stream are asserted ordered and disjoint, which is what
lets a caller splice at several of them in one pass, and a stream with rubbish
in it keeps honest ranges around what the scan stepped over.

Why in reader rather than in the caller

Three packages already scan content streams through this one scanner —
extract, render and latex — so the primitive is shared already; what was
missing was the one fact only the scan has. The first consumer is
go-pdfkit/ops, for a verb that recolours text without touching anything else.

Checks

100 % statement coverage, go vet and gofmt clean. Two mutations caught: the
range reduced to the operator alone, and the first-operand mark overwritten by
each later operand.

🤖 Generated with Claude Code

v0.6.0 — salvage, kept out of the field callers paint from

Three defects were put to this release. One of them did not exist, and that is reported as plainly as the two that did.

A broken filter chain kept what it had, and now says it is salvage

Half of this was already here and lying: flateDecode returned a damaged stream's inflated prefix with a nil error, so 37 streams — 33 in 18 real forms, 4 in fixtures — were reported as clean decodes of a fragment. The other half was missing: 263 streams in 212 of the 1 633 real forms (13.0%) failed and yielded nothing at all.

The recovered bytes are now kept structurally apart from the clean ones. Decoded.Data holds only what a filter decoded; Decoded.Undecoded holds what the chain could not get past, set instead of Data, never beside it. Writing that guarantee as a test found two live holes: cryptFilter was returning ciphertext as Data, and v0.5.0's own CCITT change meant a fax with a bad /Columns failed inside the chain with its encoded bytes landing in Data — a fax handed to a stencil path as if it were samples, which is exactly the shape of the defect render v0.10.0 had just fixed. The guarantee is a type, not a flag.

/Length against the cross-reference table — there was nothing to fix

The cross-check is already in parse.go. Measured over 344 367 streams in 4 283 files: 47 streams need the scan fallback, 0 are ambiguous, 0 truncate, 0 overrun the next object header. No change was made, and none was needed.

A masked error, worse than described

Two faults. The masking bites 30 of 30 arXiv open failures: every one reported no startxref in the last 4096 bytes where the rebuild had actually found the file holds no indirect objects (28) or no document catalogue found (2). And an ordering fault — repair() read object streams before the decryption key existed, so an encrypted file with broken tables was rebuilt from unreadable bytes. That one bites 0 files in either corpus; it ships anyway, with the negative result stated rather than dressed up.

Three the search turned up on its own

/Crypt is the biggest real gap: 209 of the 263 failures, in 209 files — 12.8% of the real forms. Implementing only the filter would be measurably the worst state (209 of 209 decode "cleanly", 0 of 209 give readable XMP); both halves together give 209 of 209 readable.

A second map-order non-determinism, in indexObjectStreams — distinct from the Expanded fix in v0.4.2, and still giving 5 answers in 5 runs with that fix in. The tie-break now matches what the header scan already does: later in the file wins.

/StmF and /StrF both /Identity means nothing in the file is encrypted and no password is needed. 3 of 495 encrypted files; all three now open with 30, 1 and 30 pages — poppler's exact numbers.

Measured

118 843 arXiv files	pages 138 333 → 138 333; content bytes 30 423 755 330 → 30 423 755 328; 0 panics
1 633 real forms	every per-file content hash identical; strict refusals 263 → 87
1 184 files cut to 92%	pages returning nothing 591 → 0; content 126 466 671 → 137 214 633 (+8.50%), of which 137 214 607 bytes are clean decodes and 26 bytes are flagged salvage; determinism 6-runs-5-answers → 3-runs-1-answer
cost	flat on healthy files (563 MB allocated either side); +12% allocation on the damaged path, wall clock inside noise

The two arXiv files differing by one byte differ by a redundant newline for an empty stream in a /Contents array; operation count and operator-sequence hash are identical on both.

100% statement coverage, go vet and -race clean, nine cross-compile targets — each built one per line, with bogus GOOS/GOARCH as controls. Two fuzz targets added where there were none. Nothing outside the standard library.

v0.5.0 — a scanned form is a fax, and a fax was a blank page

A scanned form is a fax, and a fax was a blank page

67 of the 1 633 real forms in the corpus — the eleven issuing bodies, not the vendor test suites — carry a CCITT-encoded image, 263 images between them, and 6 of their pages have nothing else on them at all. Those pages drew entirely blank, and a blank page is the failure a reader notices before any other.

Group 3 one-dimensional, Group 3 mixed and Group 4 are all decoded, with /K, /Columns, /Rows, /BlackIs1, /EncodedByteAlign and /EndOfBlock read.

Why it belongs in the reader

The other filters this package stops at carry an image with its own idea of how many components it has and how deep they are. A fax does not: it produces bilevel samples, one bit a pixel, rows padded to a byte, and the stream dictionary says what they mean. That is a byte stream, so it is a filter — and decoding it here means every caller gets it rather than each writing its own.

ImageFilter no longer returns true for /CCITTFaxDecode or /CCF. That is a change to a documented contract: a caller that switched on the filter name and handled the fax itself will now receive samples with an empty filter name instead.

The reference, read before writing

ITU-T T.4 and T.6, by way of the tables and the changing-element algorithm in golang.org/x/image/ccitt — read, not imported, because this package has no dependencies outside the standard library and gains none. The 218 code table entries were extracted mechanically from that package's gen.go rather than retyped, because a table of 218 variable-length codes transcribed by hand is a table with a mistake in it.

Measured against that reference, image by image

Every CCITT image in the corpus, decoded by both and compared byte for byte, with both sides working the height out for themselves where /Rows is absent:

	images	byte-identical	differ	reference refuses
1 633 real forms	263	260	0	3
arXiv sample	47	47	0	0

The three the reference gives up on — "invalid code", "invalid offset" — are pages 2, 3 and 4 of fr-cerfa/cerfa_10455.pdf, where it stops and we decode all 3 504 rows, exactly the /Height the image dictionary names. Being more forgiving is deliberate: a damaged scan comes back as far as it got, because refusing it turns a form into a blank page.

Two things the measurement found that reading the specification did not

Zero bits decode as perfectly good two-dimensional modes — pass mode is 0001, vertical-left-3 is 0000010 — so a decoder that does not stop at the padding goes on inventing rows that look like the last real one. fr-cerfa/cerfa_10701 gave 1 636 rows for a 208-row image that way, every surplus row with plausible ink on it.

The fuzz target found the second: on a truncated fax the early return skipped the padding that makes the answer as long as /Rows promised, so two bytes of input gave one row where the caller had asked for four, with no error to say so. 6.7 million fuzz executions since, none failing.

What it changes when a page is drawn

Through render, on the first page of all 1 633 real forms, 69 pages change: one goes from blank to drawn (fr-cerfa/cerfa_11818, 0 inked pixels to 28 989), fourteen gain ink, and fifty-one lose ink.

That last number is the interesting one. Those pages were not blank before — they were noise. 273 of the corpus's image masks carry an encoded filter, and render handed a mask to its stencil path before asking whether the bytes were still compressed, so a fax was drawn as a stencil of its own compressed bytes. Decoding it replaces the noise with the picture, which has less ink in it because most of a form is white. Fixed on the other side in go-pdfkit/render#20; the 9 JBIG2 masks that remain are now not drawn rather than drawn wrong.

The same file drew a different picture on different runs

An inline image dictionary may carry both spellings of a key and disagree with itself — /W 20 beside /Width 10 — and expanding it in one pass over the map let Go's deliberately randomised iteration order pick the winner. Eight renders of safedocs' Inline_Image_Abbreviations fixture gave five different answers; now they give one. Measured scope: of 3 668 corpus files rendered three times over, exactly one disagrees with itself, and it is the fixture built for this case.

100% statement coverage, go vet and -race clean, nine cross-compile targets, nothing outside the standard library.

v0.4.2 — the same file drew a different picture on different runs

The same file drew a different picture on different runs

An inline image dictionary may say the same thing twice and disagree with itself: /W 20 beside /Width 10, /CS /RGB beside /ColorSpace /3chanRGB, /D beside /Decode with the components reversed. Expanded() walked the dictionary in one pass, so when both spellings were present the winner was whichever Go's map iteration happened to yield last — and that order is deliberately randomised per range.

Eight renders of safedocs' Inline_Image_Abbreviations fixture, same binary, same file:

before	79b9edef 79b9edef b2594aef 6bf89747 6bf89747 544c8e6b 79b9edef 43030bf3
after	87f31291 × 8

Five different answers out of eight, against one.

The scope is measured, not guessed

Rendering the first page of 3 668 corpus files three times over — the 1 633 real forms, 1 400 arXiv papers and all 635 vendor fixtures — turns up exactly one file that disagrees with itself, and it is that fixture, which safedocs built for this case: its seven images each pair an abbreviation against the written-out name, with comments marking the line to remove "to see the effect".

So the corpus measurements already taken are not contaminated. One file is, and it is the one designed to be.

Which spelling wins

The specification permits both in an inline image and says nothing about precedence, and the public implementations disagree: pdf.js asks for the abbreviation and falls back to the written-out name; MuPDF asks the other way round.

The abbreviation wins here, for two reasons: it is the spelling Table 93 gives for an inline image, the written-out name being the tolerated alias; and safedocs marks the abbreviation as the line whose removal changes what a viewer shows, which is a statement about the reference viewer.

Two passes settle it — the written-out names first, then the abbreviations over them. No order within either pass can matter, because the names in each are distinct, so the answer no longer depends on the run whichever way it is read.

The test expands the same dictionary two hundred times and requires every answer to match the first. Against the parent commit it fails on the first comparison, reporting Height = 40 where the first gave 10.

100% statement coverage, go vet and -race clean, nine cross-compile targets, no dependencies outside the standard library.

v0.4.1 — a 219-byte denial of service

Repairing a damaged file means listing its objects in order, and that was done by counting from nought to the highest object number and asking the cross-reference table about each one.

A file may name any object number it likes. mozilla/pdf.js's bug1980958.pdf is 219 bytes and declares object 2147483647 beside its three real ones. Counting up to that is two thousand million map lookups, done twice, for three objects.

before	219 bytes in, Open took 25.39s
after	219 bytes in, Open took 0s

No allocation either way — it was a loop, not a memory bomb, which is why nothing caught it. Any file a reader opens could do this.

The numbers that exist are collected, sorted and walked, which is what the order was wanted for in the first place.

Found by battle-testing go-pdfkit/forms against 1 046 files — 69 public forms plus the whole of mozilla/pdf.js's test corpus, which is adversarial by construction. 1 029 opened, 14 580 fields read and filled, no panics, and this one file took 12.87 of the run's 13.1 seconds.

100% statement coverage, go vet clean.

v0.4.0 — say how a file is protected

Document.Protection() reports what a file said about itself on the way in: the method in words (RC4-40, RC4-128, AES-128, AES-256, or none for a file that declares a security handler and protects nothing with it), the handler's revision, the permissions it grants, and whether the password it opened with was the owner's — in which case those permissions do not apply.

Permissions gains Allows and String, so what a file grants reads as print, copy rather than as a bit pattern.

100% statement coverage.