Repository navigation
v0.8.1 — text-extraction quality fixes
v0.8.1 — 2026-06-18
Added
Page.Tables()now recovers rect-bordered tables — those with full column rules and an
outer frame but no horizontal rules between data rows, which previously collapsed every column
into a single cell and dropped the open row-label column. The detector infers row boundaries
from the data text and recovers the open label column, so a frame-ruled, row-unruled grid
extracts with its rows and label column intact. False positives on framed non-table regions
(callout boxes, two-column prose, a ruled table's wrapped last row) are guarded against, and
existing ruled-table output is unchanged. This remains Experimental — it is validated on
table geometry, not yet a quality contract for rect-bordered tables generally: a region whose
geometry is a single tall multi-line cell (one wrapped row rather than many rows) is
indistinguishable from a data grid and will be split into rows.
Changed
Page.Tables()gains blocking accuracy regression gates on the public surface and a
precise, honest documented scope; it remains Experimental. Reconstruction accuracy is
now locked by corpus regression tests so it cannot silently drift: EPA eGRID 476/476 cells,
IRS SOI 160/160 (including the recovered half-open right column, 32/32), and a 17×2 NIST
HB44 grid at 33/34 content. The documented scope is made precise: ruled (lattice)
tables — interior cells closed by visible rules between rows and columns, plus
structurally-recovered half-open edge columns. Borderless and partially-ruled or banded
tables (ruled only at group boundaries, rows separated by shading) are out of scope —
best-effort, not a contract: they may return no table or an incomplete grid. A superscript
renders at a distinct vertical position and font size, so it extracts as a spaced token
(cm²→cm 2) — a Y-offset effect specific to superscripts, not a general spacing
artifact, and it never changes which cell a value lands in. Graduation to a Stable API is
gated on a forthcoming extraction-quality program. SeeAPI-STABILITY.mdandEXAMPLES.md.
Fixed
-
GetPlainText,Page.Words(), andPage.Tables()no longer garble a composite (Type0) font
that declares/Encoding /Identity-Hand/ToUnicodeas the name/Identity-H(rather than
a CMap stream) — a pattern produced by some tools (e.g. certain LibreOffice/FreeType output) whose
two-byte codes are in fact UCS-2 big-endian Unicode. Such a font previously decoded each two-byte
code as two single bytes, emitting a U+FFFD before every character (roughly half the output), so a
whole document came out unreadable; it now decodes directly to text. The recovery is narrowly
gated — it applies only when the font is Type0, both/Encodingand/ToUnicodename
Identity-H/Identity-V, and the descendant CIDSystemInfo ordering is explicitlyIdentity
— so a genuine CJK CIDFont (Adobe-Japan1/GB1/CNS1/Korea1), whose Identity-H codes are glyph
identifiers rather than Unicode, is unaffected and continues to report a missing-ToUnicode warning.
There is no API change and every existing corpus golden is unchanged. -
Page.Tables()no longer emits phantom empty columns from decorative double-wall borders.
A frame drawn with two close-set parallel rules — common on report cover and navigation pages —
leaves a thin gap that the lattice detector turned into an entirely empty column in the grid (a
cover frame could reconstruct with eight such phantom columns). A thin, entirely empty gutter
column is now dropped. The drop is width-gated: a column is removed only when it is both entirely
empty AND narrow — below both a fraction of the table's median column width and an absolute
ceiling — so a legitimately empty, normal-width data column is always preserved. There is no API
change, and every ruled data table is unchanged. -
Page.Tables()no longer emits tabular dot-leader filler — a run of four or more leader
dots (.) that visually connects a row label to its value — into cell text. Such a run is page
typography, not data: a Year label that previously extracted as1973 ......................now
reads1973, and a spurious extra column built from an all-leader overflow run is no longer
fabricated. The trim is conservative — it fires only on a token of four or more consecutive.
and nothing else, so decimals (4.0), abbreviations (U.S.A.), and three-dot ellipses are
untouched, and a cell whose entire content is a dot run is preserved rather than emptied. The
contamination was observed across multiple publishers' statistical and regulatory tables. There
is no API change and every corpus golden is unchanged. -
Page.Lines()andPage.Blocks()now keep columns in the correct reading order on dense
multi-column pages. Previously a column whose text began slightly left of the detected
inter-column gutter was mis-assigned to the previous column, soLines()merged two columns into
one line andBlocks()(which reads each column top-to-bottom) interleaved them — scrambling the
reading order on layouts such as multi-column government, regulatory, and journal pages. A line or
word is now assigned to the column a gutter opens only when the majority of it lies to the right of
the gutter, so a genuine column start is placed correctly while a short or indented line in the
left column stays put. There is no API change; single-column and CJK pages are unaffected. Reading
order around a full-width line that spans the gutters (a masthead or heading) remains best-effort. -
GetPlainText,Page.Words(), andPage.Tables()no longer emit U+FFFD replacement
characters for a simple (Type1/TrueType) font whose embeddedToUnicodemap marks a glyph as
U+FFFD — a quirk of some PDF producers (e.g. a decimal point or other punctuation the subsetter
could not map back to Unicode). Such a code now falls back to the font's own/Encoding,
recovering the real character, so values that previously read as4<U+FFFD>0now extract as
4.0. A code genuinely absent from theToUnicodestill reads as U+FFFD, and composite (Type0)
fonts are unaffected; the fallback is used only where the font's encoding gives a confident
mapping (a known base encoding or explicit/Differences), never to guess an unknown one. -
Page.Words(),Page.Lines(), andPage.Tables()now correctly segment text from PDFs
using composite (Type0 / CIDFont) fonts. Previously the glyph advance was always 0 for
these fonts — GoPDF read only the simple-font/Widthsarray, which does not exist in
CIDFont dicts — causing the word segmenter to split every character into its own word and
break table cell matching. Glyph advance is now read from the descendant CIDFont's/DW
entry (falling back to 1000 per PDF spec §9.7.4.3 when absent). Per-CID/Wlookup is
deferred; the uniform/DWapproximation is sufficient for segmentation and is locked by a
corpus regression gate. -
GetPlainTextno longer emits U+FFFD replacement characters for text drawn with composite
(Type0 / Identity-H, two-byte CMap) fonts in two cases:- Text shown after a
q/Qgraphics-state save/restore — the active text encoder is now
saved onqand restored onQ, so text that relies on the outer font restored byQ
(with no freshTf) decodes through the correct encoder instead of the inner font's. - A
T*line-move under such a font — the line break is now emitted as a newline instead of
being decoded through the two-byte CMap and turning into a replacement character.
- Text shown after a
Verify this release
Signed with an ED25519-SK FIDO2 hardware key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
The trusted signing keys live on the GitHub account, so verification needs nothing copied from here:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.1
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkYModule integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.8.1 && go mod verify