Skip to content

v0.8.1 — text-extraction quality fixes

Choose a tag to compare

@Detective-XH Detective-XH released this 18 Jun 01:47
· 92 commits to master since this release
v0.8.1
3c38d09

v0.8.1 — 2026-06-18

Added

  • Page.Tables() now recovers rect-bordered tables — those with full column rules and an
    outer frame but no horizontal rules between data rows, which previously collapsed every column
    into a single cell and dropped the open row-label column. The detector infers row boundaries
    from the data text and recovers the open label column, so a frame-ruled, row-unruled grid
    extracts with its rows and label column intact. False positives on framed non-table regions
    (callout boxes, two-column prose, a ruled table's wrapped last row) are guarded against, and
    existing ruled-table output is unchanged. This remains Experimental — it is validated on
    table geometry, not yet a quality contract for rect-bordered tables generally: a region whose
    geometry is a single tall multi-line cell (one wrapped row rather than many rows) is
    indistinguishable from a data grid and will be split into rows.

Changed

  • Page.Tables() gains blocking accuracy regression gates on the public surface and a
    precise, honest documented scope; it remains Experimental. Reconstruction accuracy is
    now locked by corpus regression tests so it cannot silently drift: EPA eGRID 476/476 cells,
    IRS SOI 160/160 (including the recovered half-open right column, 32/32), and a 17×2 NIST
    HB44 grid at 33/34 content. The documented scope is made precise: ruled (lattice)
    tables
    — interior cells closed by visible rules between rows and columns, plus
    structurally-recovered half-open edge columns. Borderless and partially-ruled or banded
    tables (ruled only at group boundaries, rows separated by shading) are out of scope —
    best-effort, not a contract: they may return no table or an incomplete grid. A superscript
    renders at a distinct vertical position and font size, so it extracts as a spaced token
    (cm² → cm 2) — a Y-offset effect specific to superscripts, not a general spacing
    artifact, and it never changes which cell a value lands in. Graduation to a Stable API is
    gated on a forthcoming extraction-quality program. See API-STABILITY.md and EXAMPLES.md.

Fixed

  • GetPlainText, Page.Words(), and Page.Tables() no longer garble a composite (Type0) font
    that declares /Encoding /Identity-H and /ToUnicode as the name /Identity-H (rather than
    a CMap stream) — a pattern produced by some tools (e.g. certain LibreOffice/FreeType output) whose
    two-byte codes are in fact UCS-2 big-endian Unicode. Such a font previously decoded each two-byte
    code as two single bytes, emitting a U+FFFD before every character (roughly half the output), so a
    whole document came out unreadable; it now decodes directly to text. The recovery is narrowly
    gated — it applies only when the font is Type0, both /Encoding and /ToUnicode name
    Identity-H/Identity-V, and the descendant CIDSystemInfo ordering is explicitly Identity
    — so a genuine CJK CIDFont (Adobe-Japan1/GB1/CNS1/Korea1), whose Identity-H codes are glyph
    identifiers rather than Unicode, is unaffected and continues to report a missing-ToUnicode warning.
    There is no API change and every existing corpus golden is unchanged.

  • Page.Tables() no longer emits phantom empty columns from decorative double-wall borders.
    A frame drawn with two close-set parallel rules — common on report cover and navigation pages —
    leaves a thin gap that the lattice detector turned into an entirely empty column in the grid (a
    cover frame could reconstruct with eight such phantom columns). A thin, entirely empty gutter
    column is now dropped. The drop is width-gated: a column is removed only when it is both entirely
    empty AND narrow — below both a fraction of the table's median column width and an absolute
    ceiling — so a legitimately empty, normal-width data column is always preserved. There is no API
    change, and every ruled data table is unchanged.

  • Page.Tables() no longer emits tabular dot-leader filler — a run of four or more leader
    dots (.) that visually connects a row label to its value — into cell text. Such a run is page
    typography, not data: a Year label that previously extracted as 1973 ...................... now
    reads 1973, and a spurious extra column built from an all-leader overflow run is no longer
    fabricated. The trim is conservative — it fires only on a token of four or more consecutive .
    and nothing else, so decimals (4.0), abbreviations (U.S.A.), and three-dot ellipses are
    untouched, and a cell whose entire content is a dot run is preserved rather than emptied. The
    contamination was observed across multiple publishers' statistical and regulatory tables. There
    is no API change and every corpus golden is unchanged.

  • Page.Lines() and Page.Blocks() now keep columns in the correct reading order on dense
    multi-column pages. Previously a column whose text began slightly left of the detected
    inter-column gutter was mis-assigned to the previous column, so Lines() merged two columns into
    one line and Blocks() (which reads each column top-to-bottom) interleaved them — scrambling the
    reading order on layouts such as multi-column government, regulatory, and journal pages. A line or
    word is now assigned to the column a gutter opens only when the majority of it lies to the right of
    the gutter, so a genuine column start is placed correctly while a short or indented line in the
    left column stays put. There is no API change; single-column and CJK pages are unaffected. Reading
    order around a full-width line that spans the gutters (a masthead or heading) remains best-effort.

  • GetPlainText, Page.Words(), and Page.Tables() no longer emit U+FFFD replacement
    characters for a simple (Type1/TrueType) font whose embedded ToUnicode map marks a glyph as
    U+FFFD — a quirk of some PDF producers (e.g. a decimal point or other punctuation the subsetter
    could not map back to Unicode). Such a code now falls back to the font's own /Encoding,
    recovering the real character, so values that previously read as 4<U+FFFD>0 now extract as
    4.0. A code genuinely absent from the ToUnicode still reads as U+FFFD, and composite (Type0)
    fonts are unaffected; the fallback is used only where the font's encoding gives a confident
    mapping (a known base encoding or explicit /Differences), never to guess an unknown one.

  • Page.Words(), Page.Lines(), and Page.Tables() now correctly segment text from PDFs
    using composite (Type0 / CIDFont) fonts. Previously the glyph advance was always 0 for
    these fonts — GoPDF read only the simple-font /Widths array, which does not exist in
    CIDFont dicts — causing the word segmenter to split every character into its own word and
    break table cell matching. Glyph advance is now read from the descendant CIDFont's /DW
    entry (falling back to 1000 per PDF spec §9.7.4.3 when absent). Per-CID /W lookup is
    deferred; the uniform /DW approximation is sufficient for segmentation and is locked by a
    corpus regression gate.

  • GetPlainText no longer emits U+FFFD replacement characters for text drawn with composite
    (Type0 / Identity-H, two-byte CMap) fonts in two cases:

    • Text shown after a q/Q graphics-state save/restore — the active text encoder is now
      saved on q and restored on Q, so text that relies on the outer font restored by Q
      (with no fresh Tf) decodes through the correct encoder instead of the inner font's.
    • A T* line-move under such a font — the line break is now emitted as a newline instead of
      being decoded through the two-byte CMap and turning into a replacement character.

Verify this release

Signed with an ED25519-SK FIDO2 hardware key:

256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)

The trusted signing keys live on the GitHub account, so verification needs nothing copied from here:

curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
  | python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.1
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY

Module integrity is independently guaranteed by the Go checksum database:

go mod download github.com/Detective-XH/gopdf@v0.8.1 && go mod verify