stet 0.8.2

Added

• DisplayElement::TextRun, a display-list element for text
  extraction, with TextRunParams, ShownGlyph and UnicodeSource in
  stet_graphics::device (re-exported by the stet facade). A run is a
  stretch of shown text in one font along one baseline, however many show
  operations displayed it, in Unicode, with each glyph's device-space
  origin and advance, its character code, and where its text came from;
  glyph_to_device, ascent and descent give each glyph's box, rotated
  or skewed like the text, start and end span the run, vertical marks
  vertical writing, and word_breaks marks where words were set apart by
  distance rather than by a space character, as TeX sets them. The rule
  that cuts runs and finds word gaps is TextRunParams::step_to, in the
  new stet_graphics::text module, which both producers share. It paints
  nothing, and every renderer and the PDF writer skip it. Both producers
  take a TextExtraction level, Off by default —
  InterpreterBuilder::text_extraction (or Context::text_extraction) for
  PostScript and PdfDocument::set_text_extraction for PDF. With it off,
  display lists are unchanged; Glyphs records runs with every glyph, and
  Runs the same runs with glyphs left empty, in about half the memory,
  for callers that want a page's text and lines but not each glyph's
  place. Both record runs (below). Renderers that match on
  DisplayElement already have the wildcard arm the enum's
  #[non_exhaustive] requires.
• Text extraction from PDF. With PdfDocument::set_text_extraction,
  the text-showing operators (Tj, TJ, ', ") record TextRuns
  beside the glyphs they draw — in forms, annotation appearances, layers and
  transparency groups, nested like the content, so a viewer extracts only
  visible layers' text with the LayerSet it renders with. Text inside an
  /ActualText marked-content span is the span's text — the author's
  statement of what ligatures, symbols or drawn characters say, or, when
  empty, that a glyph such as a line-end hyphen says nothing: the span's
  first glyph carries it all and the rest none, the outermost span wins,
  and forms drawn inside the span are covered by it. Outside a span, a
  glyph's text comes from the font's /ToUnicode CMap, else its glyph
  name through the Adobe Glyph List, else — for CJK fonts on Adobe's Japan1, CNS1, GB1 and
  Korea1 collections, or with a Uni… encoding CMap — its CID or code;
  failing all of those it is left empty. One rule is a heuristic, taken
  from Poppler so TeX documents made with dvips extract: a glyph name
  outside the AGL that spells a number (a80, g65) is read as that
  character code in Latin-1. Invisible text
  (render modes 3 and 7, as OCR layers use) is recorded and flagged. Text
  that is not the document's is not recorded: text drawn inside a Type 3
  glyph procedure (the glyph is the text), in a tiling pattern cell, or in
  a soft-mask group.
• Text extraction from PostScript. With
  InterpreterBuilder::text_extraction, every show operator — show,
  ashow, widthshow, awidthshow, kshow, xshow, yshow, xyshow
  and glyphshow — records TextRuns for Type 1, CFF, Type 42
  and Type 3 fonts, and for composite fonts: CID-keyed (CFF and
  TrueType, horizontal and vertical) and FMapType, a new run starting
  wherever the descendant font changes. Text comes from each glyph's name
  through the Adobe Glyph List (with the same dvips numeric-name rule as
  PDF); for a CIDFont, from the character code when the CMap is a Uni…
  one, else from the CID through Adobe's Japan1, CNS1, GB1 or Korea1
  table. PostScript has no ToUnicode, so a font whose glyph names mean
  nothing gives empty text. Glyph space is the font's own, with ascent and
  descent from its FontBBox (a TrueType font's hhea table; for a
  Type 3 font with an empty FontBBox, as dvips writes, its glyphs'
  setcachedevice boxes). Text drawn inside a
  Type 3 BuildChar / BuildGlyph or a pattern cell is not recorded; a
  show inside a kshow procedure carries on the kshow's run, and one
  inside a cshow procedure records the whole character
  code cshow selected (its procedure sees only the last byte). Forms
  record once and are placed wherever execform draws them. A
  glyphshow glyph, shown by name, records code 0.
• stet text <file>, a CLI subcommand that prints the text a PDF,
  PostScript or EPS file shows, a line at a time in the order the file
  draws it, each page ending with a form feed. TeX's placed words come out
  separated, invisible OCR text is included, and layers hidden by default
  are left out. --json prints JSON with each line's position in points
  from the page's top-left corner, and --word-boxes adds each word's;
  -o / --output writes every selected page to one file instead of
  stdout, and --pages and --password work as for rendering.
• Assembling extracted text into words and lines, in
  stet_graphics::text and re-exported by the stet facade (with
  LayerSet). text_runs collects a page's TextRuns in content order,
  skipping layers a LayerSet hides; text_lines joins them into lines —
  superscripts and changes of font stay in their line and word — and
  splits the lines into words at shown spaces and at gaps, which is how
  TeX output ("PaperTitle", drawn as two placed words) reads as "Paper
  Title". It gives the same text at both extraction levels, with line
  boxes at both and word boxes when glyphs were recorded. It is
  deliberately simple: content order is kept, and columns, tables and
  reading order are not detected.
• PdfDocument::set_render_annotations(bool), to render pages without
  their annotations' appearances — for an application that draws
  annotations itself as editable objects, where the baked-in appearances
  would show twice. On by default; page_annotations still reads them.
• DisplayList::remove, to take an element out of a display list.
• Debug for DisplayList and DisplayElement (and the param structs
  that lacked it: PatternFillParams, GroupParams, SoftMaskParams), so
  a display list can be printed or compared as text.
• Unicode tables for text extraction in stet-fonts, the groundwork for
  extracting text from both PostScript and PDF input:
  • stet_fonts::to_unicode::ToUnicodeMap parses a PDF /ToUnicode CMap
    without losing text: surrogate pairs decode to one supplementary-plane
    character, multi-character destinations stay whole (<00660069> is
    fi, not U+FB01), and codes up to four bytes stay distinct. The PDF
    reader's rendering-side parser, which keeps one BMP code point per code
    for glyph selection, is unchanged.
  • stet_fonts::agl::glyph_name_to_text resolves a glyph name to text by
    the Adobe Glyph List specification's full algorithm over the complete
    AGL (4,281 names, against the 542 plus letters the renderer's table
    carries) — a.sc → a, f_f_i → ffi, uni00660069 → fi,
    u1D400 → 𝐀, afii10017 → А — and returns nothing for names that
    carry no text, such as g123. zapf_dingbats_glyph_name_to_text does
    the same for the ZapfDingbats font (a20 → ✔). The lists are Adobe's
    files, embedded unmodified with their BSD-3-Clause notice
    (crates/stet-fonts/LICENSE-ADOBE-AGL). glyph_name_to_unicode, which
    glyph selection uses, is unchanged.
  • stet_fonts::cid_unicode, the CID ↔ Unicode tables for Adobe-Japan1,
    CNS1, GB1 and Korea1, moved here from stet-pdf-reader so the
    PostScript interpreter can use them too, with a new cid_to_text that
    gives the text Adobe assigns each CID — every CID in the collection,
    including supplementary-plane characters and variation sequences.

Changed

• stet_pdf_reader::content::graphics_state::PdfGraphicsState has two new
  fields, fill_color_source and stroke_color_source, which the reader
  uses to re-convert colours when the rendering intent changes before
  painting (see Fixed). Code that builds a PdfGraphicsState with a struct
  literal no longer compiles; use PdfGraphicsState::new(ctm) and set the
  fields you need. Reading the struct is unaffected. This is interpreter
  state that is public only because its module is; a later release will mark
  it #[non_exhaustive] so outside code cannot construct it directly.
• stet_core::glyph_cache::CachedType3Glyph has a new field, bbox,
  the glyph's setcachedevice box, which text extraction needs for Type 3
  fonts with an empty FontBBox (see "Text extraction from PostScript"
  under Added). Code that builds a CachedType3Glyph with a struct literal no longer compiles; add
  bbox: None. It is the interpreter's glyph-cache entry, public only
  because its module is, and will be marked #[non_exhaustive] in a later
  release like PdfGraphicsState.

Deprecated

• stet_pdf_reader::content::cid_unicode::{cid_to_unicode, unicode_to_cid}. The tables moved to stet_fonts::cid_unicode; the old
  functions forward there and give the same results.
• stet_graphics::icc::intent_from_pdf_byte. It decoded the PDF
  reader's former private intent numbering, which display lists no longer
  carry. Use stet_graphics::icc::intent_from_byte, which decodes the
  documented encoding, and the constants in the new
  stet_graphics::rendering_intent module.

Removed

• stet-wasm: the set_page_callback() / clear_page_callback() JS
  exports. These registered a callback for streaming rendered bands out of
  WASM memory. The only sink that ever invoked it, MemorySink, stopped
  being constructed when the browser viewer moved to on-demand viewport
  rasterization — the change that fixed a 4.6 GB OOM on a 139-page document
  at 300 DPI. Registering a callback has been a silent no-op ever since, so
  the exports and the dead sink are removed rather than left looking
  functional. Nothing in the bundled frontend called them.

Fixed

• An error inside a cshow procedure no longer changes the next glyph
  shown. With a CID-keyed font, cshow hands its procedure the
  character's code and sets aside the CID it selected for a show inside
  the procedure to draw. A procedure that failed — caught by stopped —
  left that CID set aside, so the next show anywhere drew it in place of
  its own first glyph.

• PostScript vertical CJK text (WMode 1) is drawn where PLRM puts it.
  A vertical glyph's outline is drawn from its origin 0, the current point
  less its position vector v (half its width across, 880 units up by
  default); stet drew it from the current point, so every vertical glyph
  sat half a character right and most of a character high. That held for
  show, the xshow family and charpath. Separately, a TrueType
  CIDFont's default vertical advance and position vector — defined in
  1000-unit text space — were applied in the font's own units, so a font
  with 2048 units per em advanced less than half an em per glyph, in
  show, charpath and stringwidth alike. The PDF reader was already
  right.

• A PDF form XObject whose content fails to parse part-way no longer
  corrupts the rest of the page. An error such as an unterminated string
  inside an array made the reader return from the form before restoring
  what it had saved, so everything after it on the page drew under the
  form's transformation and resources — misplaced, or in a fallback font.
  A form with a transparency group also left its group in place of the
  page's display list and its nesting level counted, so after twenty such
  forms no further form, pattern or Type 3 glyph was drawn. The form now
  keeps what it drew before the error and the page carries on as it was.

• Text in a PostScript CMYK transparency group no longer changes how the
  group blends. A non-isolated /CS /DeviceCMYK group with Difference,
  Exclusion or a non-separable blend mode composites in CMYK when all it
  paints is CMYK. The check treated the text element that show records
  as non-CMYK content, so one line of text anywhere in the group switched
  the whole group to sRGB blending and changed the colour of everything in
  it. PDF input was unaffected.

• Forms drawn with execform keep their transparency groups, soft masks
  and layers. execform caches a form's output and replays it at each
  use, and the replay discarded groups, soft masks and layers as if only
  PDF input could produce them; stet's PostScript transparency operators
  (begintransparencygroup, beginsoftmask, beginoptionalcontent) make
  them too, so a form built with them drew nothing. Their bounding boxes are
  fixed in device space when they are made, which a replay from form space
  cannot correct, so such a form is now executed afresh at each use instead
  of being replayed.

• PDF output keeps every glyph of a Type 3 font that wraps another
  font. When a Type 3 glyph procedure draws its glyph with show (effect
  fonts that outline or shadow a real font do), the glyph cache replayed
  each later use of a glyph without moving the text it had recorded. PDF
  output draws that text rather than the glyph outlines, so every repeat of
  a character landed on its first occurrence: AAAA came out as one A.
  Raster output was unaffected.

• The stet facade no longer leaves released objects on its stacks at the
  end of a job. A program that ended with composite objects on the operand
  stack, dictionaries it created still on the dictionary stack, or an
  unfinished loop — all common in real PostScript — had them discarded only
  after the restore that released them. Release builds recovered, but in a
  debug build the dangling-reference audit panicked, so a library user's
  test suite failed on such a program. render, render_to_display_list,
  render_to_pdf and exec now discard the job's stacks first, as the CLI
  always has. The WASM viewer had the same order and is fixed too.

• The licences of third-party material stet ships are now shipped with
  it. The stet, stet-pdf-reader and stet-wasm crates embed the
  URW++ base 35 fonts, which are under the GNU AGPL v3 with a font
  exception, not stet's Apache-2.0 OR MIT; each crate now carries that
  licence as LICENSE-URW-FONTS. The prebuilt release archives, which
  shipped only stet's own licence files, now include
  THIRD-PARTY-NOTICES.txt: the font licence, Adobe's terms for the CMap
  and glyph-list data in stet-fonts, and the licence of every Rust crate
  linked into that binary, generated per target by cargo about
  (scripts/gen-third-party-notices.sh). CI fails a change that adds a
  dependency under a licence not accepted in about.toml.

• CJK text drawn with a substitute font picks the right glyphs. When a
  PDF uses a CJK CID font it does not embed, the reader maps each CID to
  Unicode to find a glyph in the substitute font. The table it used was
  built by inverting Adobe's Unicode → CID CMaps, which kept an arbitrary
  one of the code points that share a CID and dropped every CID that no
  Unicode encoding reaches, or that lies outside the Basic Multilingual
  Plane. Visible effects:

  • Some common ideographs came out as the Kangxi radical sharing their CID
    (Japan1 CID 3284, 日, as ⽇ U+2F47), which substitute fonts often lack.
  • Proportional, half-width and other variant glyphs, and characters such
    as 𠮷, drew nothing — about 6,600 CIDs in Adobe-Japan1 alone.

  The tables are now generated reproducibly
  (scripts/gen_cid_unicode_tables.py) from Adobe's CID → Unicode CMaps
  for text and its Unicode → CID CMaps for glyph selection, with Adobe's
  BSD-3-Clause notice alongside in crates/stet-fonts/LICENSE-ADOBE-CMAP.
  The PDF reader's width lookup for UCS2-encoded fonts, which maps Unicode
  back to a CID, also found no CID for such characters and now does.

• Rendering intents are no longer scrambled between the PDF reader and
  everything downstream of it. The display list's rendering_intent byte
  is documented as 0=RelativeColorimetric, 1=Absolute, 2=Perceptual,
  3=Saturation, but the PDF reader wrote its own numbering (0=Perceptual,
  1=RelativeColorimetric, 2=Saturation, 3=Absolute) and the rasterizer
  decoded with the reader's. Visible effects:

  • --device pdf on PDF input rewrote every explicit intent as a different
    one: /RelativeColorimetric became /AbsoluteColorimetric,
    /Saturation became /Perceptual, and so on.
  • PostScript images in ICC-based colour spaces were converted with the
    wrong intent; the PostScript default, RelativeColorimetric, was applied
    as Perceptual.
  • Third-party renderers that followed docs/DISPLAY-LIST.md misread every
    display list built from PDF input.

  All producers and consumers now share stet_graphics::rendering_intent.
  stet_pdf_reader::content::color_space::components_to_device_color_icc_with_intent
  and PdfGraphicsState::rendering_intent use the same encoding.

• The rendering intent in effect when a shape is painted now applies, as
  the PDF specification requires, rather than the one in effect when its
  colour was set. A content stream that sets a colour and then selects an
  intent (60 0 0 sc /Perceptual ri … f) was converted with the earlier
  intent. Shadings painted with sh ignored the intent entirely, and shading
  patterns used the intent current when the pattern was selected. Only
  documents with an output intent are affected; the GWG 22.1 output-intent
  test is built this way.

• PDF content that selects no rendering intent now uses
  RelativeColorimetric, the initial value the PDF specification gives,
  instead of Perceptual; so does an unrecognised intent name. This changes
  rendered colour only for RGB, gray and Lab ICC-based colour in documents
  with an output intent, where stet builds a separate conversion chain per
  intent; other colour is converted as before. PDF-to-PDF rewrites no
  longer add a rendering intent the source never selected.

• PostScript images now honour setrenderingintent. Sampled images and
  rasterized shadings carried a fixed intent whatever the graphics state
  said.

Downloads

stet-<version>-x86_64-unknown-linux-musl.tar.gz is the one most people
want: statically linked, no glibc requirement, no runtime libraries, no GUI.
It is the right choice for servers, CI and containers.

The -gnu Linux build adds the desktop viewer and needs glibc 2.35 or newer
(Debian 12, Ubuntu 22.04 and later) plus X11 or Wayland and OpenGL to open a
window. Rendering to files works without them.

macOS: the binaries are unsigned, so opening a downloaded archive from
Finder trips Gatekeeper ("cannot be opened because the developer cannot be
verified"). Installing with curl avoids it entirely, because the quarantine
attribute is set by the downloading application and curl does not set one:

curl -L <asset-url> | tar xz

If you already downloaded it in a browser, clear the flag with
xattr -dr com.apple.quarantine stet.

Windows: SmartScreen will warn that the publisher is unknown; choose
More info -> Run anyway. Code signing certificates cost money and stet does
not have one.

Rendering is identical across these artifacts with one caveat: the musl
build's libm differs from glibc's in the last bits, which can shift
antialiasing coverage by a pixel or two on heavily curved PostScript
artwork (measured: 265 of 5.4M pixels on one sample, invisible in practice;
all PDF samples tested were byte-identical). Do not generate rendering
baselines with one build and compare them against another.

Verify a download against SHA256SUMS.