stet 0.9.0

Breaking

• stet_pdf_reader::xref::XrefTable::entries yields (u32, &XrefEntry)
  pairs, lowest object number first, where it returned
  &[Option<XrefEntry>]. The slice had a slot for every object number up
  to the highest the file gives, so its size was set by a number read from
  the file; see the fix below. Object numbers without an entry are no
  longer visited. get, len and is_empty are unchanged: len is
  still one more than the highest object number, not the count of
  entries, as is Resolver::xref_len. Code indexing the slice by object
  number should call get.
• Seven public structs are #[non_exhaustive], as earlier releases
  said they would be: stet_core's GraphicsState, FileEntry,
  FilterState, VmMarks, NameCache and CachedType3Glyph, and
  stet_pdf_reader's PdfGraphicsState. Each gained a field in a patch
  release (0.8.2, 0.8.3, 0.8.4 and 0.8.7), which broke a struct literal
  outside its crate; now none can be written there, so a field added
  later breaks nothing. Read their fields as before, or match them with
  ... CachedType3Glyph::new makes one.

Added

• ParsePhase::ObjectMemoryLimit and PdfError::ObjectMemoryLimit:
  objects not read for the memory they would take. See "Fixed" below.

• Resolver::object_numbers: each object number the cross-reference
  table gives a place for, lowest first. Counting up to
  Resolver::xref_len is not a way to visit the objects: one entry can
  name object 4,294,967,294.

• PdfDocument::set_host_files: read no file but the document. Off,
  the PDF reader reads nothing from the machine it runs on, for an
  application reading files it did not write. A font the document does not
  carry is drawn with stet's own fonts instead of one installed on the
  machine, the machine's fonts are never listed and
  ~/.cache/stet/system_fonts.json is not written. A composite font
  keeps its codes and widths and its text is read, though only the
  characters stet's Latin fonts have are drawn. No CMap file is read from
  STET_CMAP_DIR, ~/.local/share/stet/CMap or the poppler and
  Ghostscript folders, so a predefined CMap the document does not carry is
  reported as not found. CMYK is drawn with stet's own profile, which
  stet-pdf-reader now embeds, in place of the system's from the
  ColorSync folders, /usr/share/color/icc or the Windows spool color
  folder. On by default, and nothing changes then. See
  docs/PDF-READER-API.md, "Reading no file from the machine".

• What the PDF reader repaired or left out is reported by its kind.
  Four new ParsePhase variants let an application tell these apart in
  parse_warnings without reading messages. ImageLeftOut { width, height } is an image over the set_max_image_pixels ceiling, for the
  page it is on, with its size as the file stores it rather than the
  reduced size a JPEG 2000 image is measured at. MissingFont is a font
  that is not embedded and is not one of the standard 14, drawn with a
  substitute, named in the message; this was not reported at all. MissingCMap is a predefined
  CMap that is neither embedded nor found; the text in its font is still
  drawn and read as before, and the message still names the CMap and says
  how to install CMaps. CrossReference, at Severity::Info, is a
  cross-reference table rebuilt by scanning the file, which was silent.
  A page lists at most 64 warnings of each of the new kinds. An image
  left out is also the new error PdfError::ImageOverLimit. See
  docs/PDF-READER-API.md, "Images left out, fonts and CMaps missing".

• A page list found by scanning the file is a PageTree warning.
  When the page tree cannot be followed, the pages are found by scanning
  for page objects, which may miss some or include ones an update
  replaced; that was silent. Opening lists at most 64 page tree warnings,
  then one saying the rest are not listed.

• An image that cannot be read is reported by its kind, and the rest
  of its page is drawn. An image XObject whose data cannot be decoded
  (a JPEG cut short) or whose dictionary cannot be used was a Content
  warning naming the Do operator, with nothing to tell it from content
  the page lost. It is now ParsePhase::ImageUnreadable, for the page it
  is on, and its message says what failed; the image is left out as
  before. An inline image that cannot be read is still a Content
  warning. See docs/PDF-READER-API.md, "Images left out, fonts and
  CMaps missing".

• Past 1,024 saved graphics states, a page's q is skipped, and the
  page draws on. Nothing bounded how many graphics states a page could
  save: a 51 KB file whose Flate content stream decodes to 50 MB of q
  took 17.7 s and 16.9 GB to read. The states stet saves around a form,
  pattern, glyph procedure or soft mask count toward MAX_SAVED_STATES
  (1,024), and are always saved, so the page's state is restored after
  them. Past the limit, a page's own q is skipped. Skipped q are
  counted, so the Q that matches one restores nothing rather than an
  earlier state. The page is drawn on, and gets one warning of the new
  kind ParsePhase::SavedStateLimit, which stet inspect lists as
  saved-state limit and stet prints when it renders the page. A page
  that leaves more than 1,024 q unclosed is still drawn. ISO 32000-1
  gives 28 as a typical limit. See
  docs/PDF-READER-API.md, "A ceiling on page content".

• PdfDocument::set_max_content_bytes: a ceiling on a page's
  content. It counts the decoded bytes of content one page interprets,
  a form each time it is drawn, and decodes no stream but a glyph
  procedure further than what is left of it, a raw-deflate retry of a
  stream with a damaged zlib header included. Off by default, since a
  heavy page of vector art reaches a hundred megabytes. A page over it
  is not drawn, wherever the limit is reached on it: render_page
  returns the new PdfError::ContentLimit, carrying a ContentLimit
  that names the limit, and adds no parse warning for it, so a caller
  tells it apart from a damaged page by kind. The other pages read as
  before. See docs/PDF-READER-API.md, "A ceiling on page content".

• PdfDocument::set_leave_out_text_without_cmap: leave out the text
  of a font whose CMap is missing. On, nothing is drawn or extracted
  in a composite font whose predefined CMap is neither embedded nor
  found, where its codes would otherwise be read two bytes at a time, a
  guess when the CMap gives some codes one byte. The text position is
  still advanced as if every code were two bytes, so the text that
  follows is placed right when the font's codes are all two bytes, and
  text in a text rendering mode that clips still makes ET set a clip,
  one that lets nothing through. A UCS-2 CMap keeps its text, and so
  does a CMap the document embeds whose usecmap base is missing. The
  MissingCMap message then says the text is left out, and the
  MissingFont message for such a font names it and says its text is
  left out, not that a substitute is drawn. Off by default, and nothing
  changes then. See docs/PDF-READER-API.md, "Leaving out text whose
  CMap is missing".

Changed

• A page the page tree names that cannot be read is a blank page in
  its place. Opening such a file failed with that object's error; an
  object that is not a dictionary was passed over in silence. A blank
  page now stands in for it, with the MediaBox it inherits or US Letter,
  and a PageTree warning names the page as LocationHint::Page. A
  branch of the tree that cannot be read is as many blank pages as its
  parent's /Count leaves for it, with one warning naming them all, or
  one blank page when that cannot be told or when the count is more
  pages than one for every 4 bytes of the file. Every page counts against
  that bound, the ones that can be read too: past it, a kid that cannot
  be read has no blank page, and its warning says so, and the pages the
  tree names are left out from there to the end, with one warning, so
  that no page kept changes its number. A page dictionary written in the
  /Kids array itself, which is no object of the file, counts the same
  way; a million of them took a 44 KB file to 715 MB. A kid 0 0 R is
  no page at all, however many times it is listed and even where the
  file gives object 0 an entry in use, as object 0 can never be a page.
  When
  page tree warnings are cut short, the last one no longer says more
  pages may be left out. page_count() counts these pages, so for these
  files it and the numbers of the pages after them change. See
  docs/PDF-READER-API.md, "Warnings from opening the document".

• A document opened with from_bytes or from_owned looks for the
  system's CMYK profile when it first needs its colors, not as it is
  opened. The profile is the same and is still found once per process;
  opening costs that much less and the first page that much more.

• An image over set_max_image_pixels is an ImageLeftOut warning,
  where it was a Content warning. A caller that looks for it under
  ParsePhase::Content no longer finds it there.

• A predefined CMap that is not found is a MissingCMap warning, where
  it was a Content warning. The message is the same. A caller that
  looks for it under ParsePhase::Content no longer finds it there.

• An image refused for its size is PdfError::ImageOverLimit, where it
  was PdfError::Other. The message is the same.

• An image XObject that cannot be read is an ImageUnreadable warning,
  where it was a Content warning. That includes a JPEG 2000 image
  refused because its codestream claims an absurd size. An inline image
  that cannot be read is still a Content warning. A caller that looks for
  these under ParsePhase::Content no longer finds them there.

• A font that is not embedded and cannot be stood in for is reported
  only as MissingFont. It also gave a Content warning naming the
  font resource and the error; that warning is no longer given.

• Past 1,024 graphics states saved at once, a page's q saves
  nothing. Every q saved the state and its Q restored it, however
  many were open. Now a page's q past MAX_SAVED_STATES is skipped,
  and the Q that matches it restores nothing, so a page that changes
  the state inside those levels and relies on Q to undo it is drawn
  differently. Such a page has a SavedStateLimit warning. A page
  within the limit draws as before.

Fixed

• A cross-reference entry with a huge object number costs what any
  other entry does. The PDF reader kept the cross-reference table as one
  slot per object number up to the highest, so one entry numbered
  4,294,967,295 asked for about 69 GB and took the process, and on a
  machine with swap much else, down with it. Only the entries the file
  gives are kept now.
• The objects read from a file take at most 256 bytes of memory for
  each byte of it, and never less than 16 MiB. A few kilobytes of
  compressed object stream decode to megabytes of object text, and each
  two bytes of that can be a 40-byte object: a 44 KB file of a million
  page dictionaries took 441 MB to open, and one of a million integers
  2.4 KB took 81 MB. Every object read is now charged what it holds, and
  an object stream its decoded text, against an allowance set by the
  file's length; an object that would pass it is not read, as if it were
  missing, with one ParsePhase::ObjectMemoryLimit warning naming it.
  Such files now take at most a few tens of megabytes. No real file comes
  near the allowance: with every object read, none of 717 test files
  holds more than 5.5 bytes for each of its bytes. Parsed arrays and
  dictionaries also no longer keep room for entries they do not hold,
  which made a one-entry dictionary 324 bytes where it needs about 100,
  and a Flate stream's decoded buffer no longer grows past the most it
  may hold, which made decoding up to three times as costly as that at
  its peak. stet prints the new warning, and stet inspect lists it as
  object memory limit.
• The operands waiting for an operator take at most 16 MiB. A
  content stream's operands were held until an operator took them,
  however many there were, so 98 KB of Flate decoding to fifty million
  numbers with no operator took 1.7 GB to render, and the same numbers as
  one array 4 GB. Past 16 MiB the operands waiting are dropped, and an
  array or dictionary too large on its own is read past, with one
  Content warning for the page; the content after them is drawn as
  before. An inline image's dictionary has as much again, its entries past
  that left out, and is no longer searched one entry at a time for each
  key it adds: 200,000 keys took 19 seconds to read. The most any operator
  takes on any page of 717 test files is 14.7 KB, and rendering is
  otherwise unchanged.
• An object numbered 100,000 or more is found when the file is
  scanned for objects. When the cross-reference table is missing or
  damaged, or leaves objects out, the reader scans the file for them, and
  passed over any numbered 100,000 or more; a page among them became a
  blank page. They are found now.
• An object in an object stream is the one the stream lists under its
  number. A cross-reference entry gives an object's index in its object
  stream, and the object at that index was taken whatever number the
  stream gave it, so an entry could stand in for another object; a small
  file made that way could take minutes to open. The index is now only
  where the reader looks first, as in MuPDF. An index past 65,535 is read
  too, where before it named another object.
• An object stream's /N and /First reserve nothing. A huge /N
  reserved memory for that many objects and could abort the process, and
  a negative /N or /First panicked. The stream's header is read for
  as many objects as it holds, a negative value makes the stream
  unreadable, and an offset that would overflow is refused.
• A number in a PDF too large for what it counts is not read as a
  smaller one. A reference whose object number or generation does not
  fit is a reference to the null object, as ISO 32000-1 §7.3.10 says; a
  cross-reference subsection or /Index range that starts below 0 or runs
  past the last object number gives those entries no object; and so does
  an object number in an object stream's header that does not fit. Each
  was cut to its low bits before, and could be read as another object.
• A predefined CMap is looked for only under a name a CMap can have.
  The PDF reader takes the CMap name a document gives as a file name in
  the folders CMaps are kept in. It is now used only if it is made of
  letters, digits, - and _, as CMap names are, and anything else is
  treated as not found. Before, a crafted name could make the reader open
  a file outside those folders.
• A content stream cut short by the end of the file says so. A
  truncated file's last content stream, of a page or of a form, was read
  as far as the file went with nothing to say that the rest was missing.
  It is now a Content warning for the page.
• An embedded font program or CMap that cannot be read is a Content
  warning. A substitute font, or two-byte codes, were used in their
  place without a word.
• A Type 1 or CFF glyph stops once it has run 1 MiB of charstring.
  Subroutine calls were limited only in how deep they nest, so a small
  font whose subroutines call one another many times over could keep a
  glyph running for hours, in PDF and PostScript input alike. A glyph now
  runs at most stet_fonts::charstring::MAX_CHARSTRING_BYTES, counting a
  subroutine each time it is called; past that it is left undrawn, as a
  glyph nested too deep is. That is sixteen times the largest charstring
  Type 2 allows, and no glyph in our test files comes near it. Found by
  the weekly fuzzing run.
• Writing a PostScript font into a PDF follows its subroutines within
  the same limits. Finding a glyph's width followed subroutine calls
  with no depth limit, so a subroutine that called itself exhausted the
  stack, and the scan for the subroutines a subset keeps had only the
  depth limit. When that scan gives up, every subroutine is kept.
• Unbalanced q and Q in a form, pattern, soft mask or Type 3 glyph
  no longer change the page's graphics state. An extra Q restored a
  state the page had saved, so what the page drew next lost its
  transformation, colour or clip, and the page's own Q came up short.
  A q left open in a pattern or soft mask also kept the page from
  restoring what it had saved. q and Q balance within each content
  stream (ISO 32000-1 §8.4.2): a Q with no q of its own stream to
  match now restores nothing, and a q left open is closed when its
  stream ends.

Downloads

stet-<version>-x86_64-unknown-linux-musl.tar.gz is the one most people
want: statically linked, no glibc requirement, no runtime libraries, no GUI.
It is the right choice for servers, CI and containers.

The -gnu Linux build adds the desktop viewer and needs glibc 2.35 or newer
(Debian 12, Ubuntu 22.04 and later) plus X11 or Wayland and OpenGL to open a
window. Rendering to files works without them.

macOS: the binaries are unsigned, so opening a downloaded archive from
Finder trips Gatekeeper ("cannot be opened because the developer cannot be
verified"). Installing with curl avoids it entirely, because the quarantine
attribute is set by the downloading application and curl does not set one:

curl -fL <asset-url> | tar xz

If you already downloaded it in a browser, clear the flag with
xattr -dr com.apple.quarantine stet.

Windows: SmartScreen will warn that the publisher is unknown; choose
More info -> Run anyway. Code signing certificates cost money and stet does
not have one.

Rendering is identical across these artifacts with one caveat: the musl
build's libm differs from glibc's in the last bits, which can shift
antialiasing coverage by a pixel or two on heavily curved PostScript
artwork (measured: 265 of 5.4M pixels on one sample, invisible in practice;
all PDF samples tested were byte-identical). Do not generate rendering
baselines with one build and compare them against another.

Verify a download against SHA256SUMS.