Repository navigation
Releases
v0.8.7 — canonical Kruti Dev 010 legacy Devanagari recovery
Added
- GoPDF now recovers real, searchable Unicode Devanagari from a canonical-coded simple Kruti Dev 010 font — a legacy non-Unicode font common in South Asian (Hindi) statistical PDFs whose bytes otherwise decode to Latin gibberish (for example
fodkinstead of विका, or,u-;w-,y-,e-instead of एन.यू.एल.एम.). Where the previous release only flagged such garble,Page.Words(),Page.Lines(),Page.Blocks(),Page.Tables(), andReader.GetPlainText()now emit the intended script for these fonts. The remap uses a built-in byte→Unicode transducer ported from the SIL wsresources mapping data (MIT — see NOTICE); no font files or external dependencies are bundled, and zero-width joiners are omitted so the recovered text stays searchable. Thelegacy_font_textwarning no longer fires on text that is cleanly recovered. - The remap is deliberately conservative: it fires only for a strict-legacy-named, simple, canonical-coded font whose current decode is non-Indic. Composite (Type0) Kruti, subsetted Walkman-Chanakya, and DevLys fonts are not remapped — they still decode to gibberish and still warn. Extraction output for every non-matched font is byte-identical to before. One edge case: on a page that overlaps a text column with a table in the same vertical band, a recovered table label's leading glyphs can bucket into an adjacent row; the numeric data and surrounding prose are unaffected.
Verify this release
This tag is signed with an ED25519-SK FIDO2 hardware key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
All of the maintainer's trusted keys are published on the GitHub account, so the recipe below works for any release regardless of which key signed it:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.7
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkYModule integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.8.7 && go mod verifyv0.8.6 — honest text/table quality
v0.8.6 — 2026-06-29
Added
Page.Tables()andReader.Warnings()now flag text drawn through a legacy non-Unicode font — a
pattern common in South Asian statistical PDFs where script labels (such as Devanagari/Hindi) are
rendered with fonts like Kruti Dev, DevLys, or Walkman-Chanakya, whose glyphs map to Latin
characters rather than the intended script.Page.Tables()emits a per-table
TableWarningLegacyFont(legacy_font_text) code onTable.Warningswhen a table's alphabetic
cell text is predominantly garbled in this way;Reader.Warnings()emits the same
legacy_font_textcode at the document level so consumers ofWords(),Lines(), and
GetPlainText()also receive the signal. The warning is strictly additive —Tables().Cellsis
byte-identical — and tells a caller that the garbled text cannot be recovered on a pure-text path.Page.Words(),Page.Lines(), andPage.Blocks()now omit glyphs from a recurring diagonal
watermark printed across most pages of a document — for example an agency stamp repeated on every
page. GoPDF detects a recurring watermark by sampling up to 16 evenly-spaced pages; when the same
angled-text signature recurs on at least half of them, its glyphs are excluded from word assembly,
so a value like11144no longer arrives fused with watermark characters. Documents without a
recurring page-spanning watermark are byte-unchanged;Page.Content(),Page.Texts(), and
DebugJSONalways return every glyph unfiltered.Page.Tables()is unaffected and byte-identical
— it already unconditionally excludes diagonal text from cell content.Page.Tables()now reports per-table quality: eachTablecarries aConfidence
(TableConfidence—"high"or"low") and aWarningsslice ([]TableWarning, each with a
TableWarningCode), so an LLM/RAG consumer can tell a trustworthy grid from a defective one
without re-reading the page.Confidenceis detection-relative:"high"means no quality
detector flagged the table — not that it is verified correct — so a table may move from"high"
to"low"in a future release as more detectors are added. The first detector flags a
phantom_table: a grid whose columns are mostly empty (≥ 60% entirely blank), the signature of a
bar chart or infographic misread as a ruled table (a genuinely very sparse table can also trip it,
and the warning text says so). The flag never alters the extracted cells.Page.TableRegions()returns the page-space bounding box of each detected table, 1:1 with
Page.Tables()by index, so a consumer can locate a low-confidence table on the page (for example
to crop and re-read it). Boxes are in the same Y-up display space asPage.Words()and
Page.Content(). These additions do not change the reconstructed grids —Page.Tables().Cellsis
unchanged.
Fixed
Page.Tables()now recovers the data rows of "comb-body" tables — a layout, common in statistical
yearbooks, where the header is ruled into columns but the data body is ruled only with vertical lines
extending downward (no horizontal row rules and no closing bottom rule). Previously every data value
fell outside the closed cells and was dropped from the grid, even thoughPage.Lines()held the data
and the page rendered correctly.Page.Tables()now synthesizes the data cells at the header's column
positions across the vertically-ruled data rows, recovering the full grid. False-positive guards ensure
that non-table layouts — tables of contents, formula and prose blocks, bar charts, and infographic maps
— are left untouched;Page.Words(),Page.Lines(), andPage.Blocks()are unaffected.Page.Tables()now recovers the data rows of "headers-only" ruled tables — a layout, common in
statistical yearbooks (e.g. quarterly tables), where the header is ruled into columns but the data body
is ruled only down the leftmost row-label column. Previously every data value fell outside the closed
cells and was dropped, so the grid came out with populated headers and entirely empty data columns even
thoughPage.Lines()held the data and the page renders correctly.Page.Tables()now synthesizes the
missing data cells at the header's column positions across the ruled data rows, but only for a column
whose values recur across two or more rows — so genuinely empty cells stay empty and a lone stray word
inside the table rectangle (e.g. a footnote) is not promoted to data. Recovery is confined to this
structural pattern; normally- and partially-ruled tables are left untouched, andPage.Words(),
Page.Lines(), andPage.Blocks()are unaffected. On tables whose columns are themselves only
whitespace-separated, the recovered values may land in a single fused column — the data is no longer
lost, but column splitting there remains a separate limitation.Page.Tables()now reconstructs landscape tables that are embedded on a portrait page via a 90°
counter-clockwise text matrix — a layout common in statistical yearbooks. Previously the rotation made
each value extract character-reversed (Birganj→jnagriB,1130.1→1.0311) and the whole grid came
out transposed, so the table was unusable even though the page renders correctly.Page.Tables()now
detects a predominantly-90°-rotated page and remaps it to reading orientation before reconstruction, so
cells come out de-reversed and in the correct row/column order. The remap is internal toPage.Tables();
Page.Words(),Page.Lines(), andPage.Blocks()continue to report such text in its raw rotated page
geometry. On very dense or sparsely-ruled rotated tables the data is recovered in the correct cells but
the grid may remain imperfect (a long number can be split, or a row flattened), so treat such a grid as
advisory.Page.Tables()no longer pulls diagonal watermark or rotated chart-label text into table cells. When a
document carries a watermark printed diagonally across the page, or a chart with slanted axis/arc labels,
the individual letters are scattered over the page at an angle; where one fell within a data row it was
absorbed into that table cell — turning a value like8,9intoG8,98,9, merging adjacent rows, or
adding a spurious empty column.Page.Tables()now excludes text whose baseline is more than ~10° off an
axis (horizontal or vertical) from cell content, so diagonal decoration can no longer corrupt the grid.
Axis-aligned text is always kept, including 90°/270° landscape and vertical tables and headers.
Page.Words(),Page.Lines(), andPage.Blocks()are unaffected — they still return every glyph,
including diagonal text — so this narrows only the structured table view. Tables on heavily watermarked
statistical pages, previously garbled, now extract cleanly.
Verify this release
This tag is signed with the maintainer's ED25519-SK FIDO2 hardware key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
All of the maintainer's trusted keys are published on the GitHub account, so the recipe below verifies any release regardless of which key signed it:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.6
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY...Module integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.8.6 && go mod verifyv0.8.5 — table extraction column fixes
Fixed
-
Page.Tables()now recovers the trailing groups of space-separated thousands numbers that overflow a
column's rule. In some statistical tables the figures are typeset on a slightly wider spacing than the
ruled columns, so the last group of a number — the315in66 315— crossed the column's right rule
and was dropped, truncating the rightmost numeric column on every row. Such a trailing digit group is
now re-attached to the number it belongs to, recovering the full value. The recovery is deliberately
narrow: only an all-digit group that straddles a numeric cell's right rule within the normal
intra-number spacing is pulled back, so labels, units, footnote markers, and adjacent columns are never
drawn across a boundary. The change is additive —Page.Words()andPage.Lines()are unaffected, and
every regression-locked corpus fixture is byte-for-byte unchanged. -
Page.Tables()no longer injects a phantom column into a banded table whose header background is
drawn as two side-by-side filled rectangles. The seam between those two header rectangles was being
mistaken for a column rule; where it fell inside a data column it split the value there — for example
a space-separated thousands figure5 417 724was broken into5 417and724, and the table
reported one column too many. The header band is now reconciled so the spurious column is dropped and
the value stays whole, with all header and data text preserved. This corrects single-row seam headers;
multi-row seam headers (two stacked header bands) are handled by a later fix below, while a genuine
grouped header carrying real per-column sub-labels is always left intact. Ruled tables and every
regression-locked corpus fixture are byte-for-byte unchanged. -
Page.Tables()no longer emits phantom empty columns where a drawn vertical rule cuts across a
wide spanning cell. When a real rule splits a cell that actually encloses several columns, the
lattice produced an all-empty "column" that the gutter filter could not drop — the spanning cell
made it look too wide to be a gutter. Such a column is now recognised structurally (an empty
column whose drawn span contains another column's position is a mis-split artifact) and removed.
Only all-empty columns are ever dropped, so no extracted text is affected. For example a Serbian
statistical-yearbook crop-production table reported 8 columns where the page has 6. Tables whose
empty columns are genuine, and every regression-locked corpus fixture, are unchanged. -
Page.Tables()now also removes phantom seam columns from banded tables whose header is drawn as
TWO stacked bands (e.g. a title row over a year-label row), extending the single-row seam fix above.
Each header row is checked independently and only a row carrying a decorative fill-rectangle seam
(a boundary with no corresponding data column) is collapsed; a genuine grouped/spanning header whose
sub-column edges align with real data columns is left intact, so no header or data text is lost. For
example an exchange-rate table and several monthly-digest tables that reported one or two extra empty
columns now match the printed column count. Regression-locked corpus fixtures are byte-for-byte
unchanged.
Verify this release
Signed with an ED25519-SK FIDO2 hardware key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
All of the maintainer's trusted keys are published on the GitHub account, so this recipe works regardless of which key signed the tag — nothing needs copying from these notes:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.5
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h2...Module integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.8.5 && go mod verifyv0.8.4
Fixed
Page.Words()andPage.Tables()now read proportional glyph widths for composite (Type0) fonts that declare a uniform default width but also carry a per-character width array — common for Latin/Cyrillic (and other alphabetic) text wrapped in a Type0/Identity-H subset font, especially in non-US statistical PDFs. Previously every glyph was given the uniform default advance, so a long row label drifted into the adjacent right-aligned number and the two were welded character-by-character (for example a labelinstituceand a value483extracted asinstitu4c8e3). Across a fresh multilingual corpus this character-interleaving dropped sharply and net extracted numeric content rose; ruled tables, CJK tables, and the character set ofPage.Words()are byte-for-byte unchanged. Composite fonts with a non-Identity encoding, and the degenerate zero-default-width case, are unaffected.- Known limitation: because this adjusts glyph positions on every page, a minority of documents may extract worse — specifically a banded table whose rows can be re-bucketed by the more-accurate positions; the font widths are correct, the sensitivity is in banded-table reconstruction, which remains best-effort.
Docs: the Page.Tables() scope in README/EXAMPLES is brought current (fully-ruled + rect-bordered locked classes; stroke-free per-cell-grid reconstruction + proportional Type0/Identity-H text as best-effort/improving; banded/borderless out of documented scope).
v0.8.3 — per-cell-grid table reconstruction & composite-font glyph placement
v0.8.3 — 2026-06-21
Added
-
Page.Tables()now merges the interleaved blank columns that appear when a stroke-free
per-cell-grid table's rules sub-divide each logical column into a wide (header-bearing) cell nested
with a narrow (data) cell. Such a table reconstructs with roughly twice its real column count — the
header label landing in one grid column and its data in the adjacent one (a header read with no data,
and data read with no header). GoPDF now merges each such nested pair back into a single column,
realigning the header with its data. The merge is conservative and loss-free: it fires only when the
two columns share a right wall (the nested sub-cell signature, not an ordinary column boundary), never
hold content in the same row (so no cell is ever overwritten), and one column is a sparse header while
the other is a dense data column (so two genuine data columns are never merged, and a small table is
not collapsed). It was validated against the nested-sub-cell signature by unit tests; every ruled,
rect-bordered, and previously-handled table is byte-for-byte unchanged, and reconstruction remains
best-effort per thePage.Tables()documentation. -
Page.Tables()now also recovers the rows of dense per-cell-grid statistical tables in which many
stacked records had collapsed into a single tall cell (every value for a column landing in one row).
Common in stroke-free government statistics — for example Japanese MHLW wage tables and US Census trade
releases — GoPDF now splits that cell into its separate records. The split is conservative: it fires only
where the would-be rows carry numeric values across two or more columns (so a wrapped multi-line label, a
column header, or a footnote line is left intact), and only for a genuinely stacked block of several rows.
It was validated against a blind-authored held-out wage-table fixture, where the recovered rows match the
printed table. Known limitations: a multi-row block of numeric column headers, or a non-tabular text block
(a cover page, a table of contents, or running prose) that reaches this reconstruction path, can be
over-split. Every ruled, rect-bordered, and previously-handled table is byte-for-byte unchanged, and
reconstruction remains best-effort per thePage.Tables()documentation. -
Page.Tables()now recovers the columns of dense per-cell-grid statistical tables whose numeric
columns previously collapsed into a single cell per row. In many stroke-free statistical tables — common
in non-US government statistics, for example German Destatis and Taiwan DGBAS releases — the data band
closed as one wide cell, so every figure in a row landed in the same column; GoPDF now splits that cell
at the table's own vertical column rules, restoring the separate columns (measured against a blind-authored
German insolvency regression fixture, the great majority of data cells now land in the right column). The split is conservative:
it fires only at column boundaries the table itself establishes elsewhere (so a rule from a different
table on the page cannot mis-split a cell) and only when the cell's own content actually spans those
columns (so a spanning label or merged-header title is left intact). Every ruled, rect-bordered, and
previously-handled table is byte-for-byte unchanged, and reconstruction remains best-effort per the
Page.Tables()documentation. -
Page.Tables()now reconstructs fill-banded "staircase" tables — statistical tables whose rows
are separated by alternating background shading (filled rectangles) rather than ruled lines, drawn as
a staircase of nested fills that share a common bottom edge (for example U.S. EIA Annual Energy Review
energy-price tables). Such a table previously collapsed to roughly two-thirds of its rows — the lattice
closed one cell per shading step and fused the rows that share a step — and GoPDF now splits those bands
back into individual rows and columns, recovering the full grid. Detection is deliberately conservative:
it requires the nested-fill staircase signature, so bar/column charts, shaded callout boxes, and shaded
prose are not promoted to tables, and every other table (ruled, rect-bordered, or borderless) is
byte-for-byte unchanged. Banded reconstruction remains best-effort (per thePage.Tables()
documentation) — this covers the common-bottom staircase style; other banded layouts may still yield an
incomplete grid. -
Page.Tables()now also reconstructs the opposite banded style — dense per-cell-grid tables drawn
with one filled rectangle per cell and no ruled lines (for example U.S. BEA Survey of Current
Business GDP tables). Such a table's grid already closes on its own, but the page's title and footnote
banners were swept in as extra phantom rows; GoPDF now restricts the result to the table's multi-column
data body, dropping those banner rows and recovering the intended grid. Trimming is conservative: a row
is removed only when it is a single full-width banner lying outside the data body — a real multi-column
row is never dropped, even when blank — and a genuine table with no phantom banners is left byte-for-byte
unchanged. As above, banded reconstruction is best-effort; cell values in these tables can still be
limited by a separate word-spacing issue when adjacent numeric columns are laid out with no real gap
between them.
Fixed
Page.Words(),Page.Tables(), andPage.Content()now place glyphs correctly for composite
(Type0) CIDFonts whose default glyph width (/DW) is zero — a quirk of some
Microsoft-Office-exported PDFs (for example a bold Cambria heading font). Such a font advances
every glyph by zero, so a whole label stacked at one position and the geometry-sorted extractors
read it scrambled —Personal consumption expenditurescame out as
esxpendituronsumption eersonal cP. GoPDF now reads the font's per-CID width array (/W) for these
zero-/DWfonts, so the glyphs are spaced and the label reads in order. Fonts with a normal
(non-zero)/DWare unchanged, andGetPlainText— which reads in stream order and never used
glyph widths — was already correct and is byte-for-byte unaffected.
Verify this release
This tag is signed with an ED25519-SK FIDO2 hardware key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.3
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkYModule integrity:
go mod download github.com/Detective-XH/gopdf@v0.8.3 && go mod verifyv0.8.2 — table extraction quality + robustness
v0.8.2 — 2026-06-19
Fixed
-
Page.Tables()now recovers row labels whose dot leader is rendered with its glyphs interleaved
into the label text. Some publishers (for example USDA statistical tables) lay out a row label and
its trailing dot leader as overlapping glyphs, so the label arrived fused with filler dots —
Alabamaextracting asAlaba...m...a..... GoPDF now detects this fused-leader signature (a run
of three or more dots flanked by a letter on both sides) and strips the filler dots, recovering
the label. Legitimate data text is left untouched: decimal points, abbreviations such asU.S., a
trailing ellipsis (continued...), and a dot-separated range (1...3) are all preserved. The
change affects onlyPage.Tables()cell text; plain text extraction (Words/Lines) is
unaffected. -
Page.Tables()no longer splits a cell value across adjacent columns when the PDF renders the
token with an embedded zero-advance space — for example a comma-grouped number laid out as
2,1+20(2,120), or a character-spaced word laid out asaddr+ess(address). Such tokens
previously fragmented at the space, with the tail bleeding into the neighbouring cell; GoPDF now
re-joins same-row pieces that abut with no real gap, so the value lands whole in one cell. Two
genuinely distinct values that meet at a ruled cell border are kept apart — the re-join never
crosses a vertical rule — and tables with normal column spacing are byte-for-byte unchanged. -
A font whose
/ToUnicodeCMap is malformed — for example, a stray PostScriptdefkeyword
inside a dictionary literal — no longer empties the entire document. Previously the malformed
CMap raised a parse error that propagated up through font loading and left every page's text
empty; GoPDF now recovers, discards only the broken CMap, and falls back to the same handling as
a font with no/ToUnicode(so Adobe CID and Identity fonts still decode correctly). A new
malformed_tounicodewarning is raised — alongside the existingmissing_tounicode— so the
degraded font stays visible rather than failing silently. A genuine internal fault is still
re-raised rather than being masked as malformed input, and valid PDFs are unaffected. -
Type0 CJK fonts that use
/Encoding /Identity-Hor/Identity-Vwith an Adobe-Japan1 CID
ordering (/Registry (Adobe) /Ordering (Japan1)) and no/ToUnicodeare no longer garbled.
Their 2-byte show-string codes are Adobe-Japan1 CIDs, not Unicode; GoPDF now decodes them through
Adobe's published CID→Unicode table instead of mangling each CID into two replacement characters.
On real Japanese government/academic PDFs this took the U+FFFD replacement-character rate from
~30–46% to 0, recovering coherent Japanese that matches an independent extractor character for
character. Font selection stays byte-identical for every other font: the path requires Type0 +
Identity-H/V + the genuine Adobe registry and ordering + no usable/ToUnicode, so private CID
collections,Identity-ordering fonts, and non-CJK fonts are unaffected. Vertical-writing fonts
still raise the vertical-writing-mode warning (reading order for vertical text is unchanged). The
Adobe-Japan1 mapping data is derived from Adobe'scid2code.txt(BSD-3-Clause; seeNOTICE).
Changed
DocumentSummary'sDecodeRatios.FallbackRationow also counts glyphs decoded through the Adobe
CID→Unicode table described above (it already counted predefined-CMap charset approximations).
Both decode paths share thefallback_encodingwarning, so the ratio and the warning stay
consistent. Documents without such fonts are unaffected.
Verify this release
Signed with an ED25519-SK FIDO2 hardware key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
All of the maintainer's trusted keys are published on the GitHub account, so this recipe verifies the tag regardless of which key signed it — nothing needs to be copied from these notes:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.2
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkYModule integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.8.2 && go mod verifyv0.8.1 — text-extraction quality fixes
v0.8.1 — 2026-06-18
Added
Page.Tables()now recovers rect-bordered tables — those with full column rules and an
outer frame but no horizontal rules between data rows, which previously collapsed every column
into a single cell and dropped the open row-label column. The detector infers row boundaries
from the data text and recovers the open label column, so a frame-ruled, row-unruled grid
extracts with its rows and label column intact. False positives on framed non-table regions
(callout boxes, two-column prose, a ruled table's wrapped last row) are guarded against, and
existing ruled-table output is unchanged. This remains Experimental — it is validated on
table geometry, not yet a quality contract for rect-bordered tables generally: a region whose
geometry is a single tall multi-line cell (one wrapped row rather than many rows) is
indistinguishable from a data grid and will be split into rows.
Changed
Page.Tables()gains blocking accuracy regression gates on the public surface and a
precise, honest documented scope; it remains Experimental. Reconstruction accuracy is
now locked by corpus regression tests so it cannot silently drift: EPA eGRID 476/476 cells,
IRS SOI 160/160 (including the recovered half-open right column, 32/32), and a 17×2 NIST
HB44 grid at 33/34 content. The documented scope is made precise: ruled (lattice)
tables — interior cells closed by visible rules between rows and columns, plus
structurally-recovered half-open edge columns. Borderless and partially-ruled or banded
tables (ruled only at group boundaries, rows separated by shading) are out of scope —
best-effort, not a contract: they may return no table or an incomplete grid. A superscript
renders at a distinct vertical position and font size, so it extracts as a spaced token
(cm²→cm 2) — a Y-offset effect specific to superscripts, not a general spacing
artifact, and it never changes which cell a value lands in. Graduation to a Stable API is
gated on a forthcoming extraction-quality program. SeeAPI-STABILITY.mdandEXAMPLES.md.
Fixed
-
GetPlainText,Page.Words(), andPage.Tables()no longer garble a composite (Type0) font
that declares/Encoding /Identity-Hand/ToUnicodeas the name/Identity-H(rather than
a CMap stream) — a pattern produced by some tools (e.g. certain LibreOffice/FreeType output) whose
two-byte codes are in fact UCS-2 big-endian Unicode. Such a font previously decoded each two-byte
code as two single bytes, emitting a U+FFFD before every character (roughly half the output), so a
whole document came out unreadable; it now decodes directly to text. The recovery is narrowly
gated — it applies only when the font is Type0, both/Encodingand/ToUnicodename
Identity-H/Identity-V, and the descendant CIDSystemInfo ordering is explicitlyIdentity
— so a genuine CJK CIDFont (Adobe-Japan1/GB1/CNS1/Korea1), whose Identity-H codes are glyph
identifiers rather than Unicode, is unaffected and continues to report a missing-ToUnicode warning.
There is no API change and every existing corpus golden is unchanged. -
Page.Tables()no longer emits phantom empty columns from decorative double-wall borders.
A frame drawn with two close-set parallel rules — common on report cover and navigation pages —
leaves a thin gap that the lattice detector turned into an entirely empty column in the grid (a
cover frame could reconstruct with eight such phantom columns). A thin, entirely empty gutter
column is now dropped. The drop is width-gated: a column is removed only when it is both entirely
empty AND narrow — below both a fraction of the table's median column width and an absolute
ceiling — so a legitimately empty, normal-width data column is always preserved. There is no API
change, and every ruled data table is unchanged. -
Page.Tables()no longer emits tabular dot-leader filler — a run of four or more leader
dots (.) that visually connects a row label to its value — into cell text. Such a run is page
typography, not data: a Year label that previously extracted as1973 ......................now
reads1973, and a spurious extra column built from an all-leader overflow run is no longer
fabricated. The trim is conservative — it fires only on a token of four or more consecutive.
and nothing else, so decimals (4.0), abbreviations (U.S.A.), and three-dot ellipses are
untouched, and a cell whose entire content is a dot run is preserved rather than emptied. The
contamination was observed across multiple publishers' statistical and regulatory tables. There
is no API change and every corpus golden is unchanged. -
Page.Lines()andPage.Blocks()now keep columns in the correct reading order on dense
multi-column pages. Previously a column whose text began slightly left of the detected
inter-column gutter was mis-assigned to the previous column, soLines()merged two columns into
one line andBlocks()(which reads each column top-to-bottom) interleaved them — scrambling the
reading order on layouts such as multi-column government, regulatory, and journal pages. A line or
word is now assigned to the column a gutter opens only when the majority of it lies to the right of
the gutter, so a genuine column start is placed correctly while a short or indented line in the
left column stays put. There is no API change; single-column and CJK pages are unaffected. Reading
order around a full-width line that spans the gutters (a masthead or heading) remains best-effort. -
GetPlainText,Page.Words(), andPage.Tables()no longer emit U+FFFD replacement
characters for a simple (Type1/TrueType) font whose embeddedToUnicodemap marks a glyph as
U+FFFD — a quirk of some PDF producers (e.g. a decimal point or other punctuation the subsetter
could not map back to Unicode). Such a code now falls back to the font's own/Encoding,
recovering the real character, so values that previously read as4<U+FFFD>0now extract as
4.0. A code genuinely absent from theToUnicodestill reads as U+FFFD, and composite (Type0)
fonts are unaffected; the fallback is used only where the font's encoding gives a confident
mapping (a known base encoding or explicit/Differences), never to guess an unknown one. -
Page.Words(),Page.Lines(), andPage.Tables()now correctly segment text from PDFs
using composite (Type0 / CIDFont) fonts. Previously the glyph advance was always 0 for
these fonts — GoPDF read only the simple-font/Widthsarray, which does not exist in
CIDFont dicts — causing the word segmenter to split every character into its own word and
break table cell matching. Glyph advance is now read from the descendant CIDFont's/DW
entry (falling back to 1000 per PDF spec §9.7.4.3 when absent). Per-CID/Wlookup is
deferred; the uniform/DWapproximation is sufficient for segmentation and is locked by a
corpus regression gate. -
GetPlainTextno longer emits U+FFFD replacement characters for text drawn with composite
(Type0 / Identity-H, two-byte CMap) fonts in two cases:- Text shown after a
q/Qgraphics-state save/restore — the active text encoder is now
saved onqand restored onQ, so text that relies on the outer font restored byQ
(with no freshTf) decodes through the correct encoder instead of the inner font's. - A
T*line-move under such a font — the line break is now emitted as a newline instead of
being decoded through the two-byte CMap and turning into a replacement character.
- Text shown after a
Verify this release
Signed with an ED25519-SK FIDO2 hardware key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
The trusted signing keys live on the GitHub account, so verification needs nothing copied from here:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.1
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkYModule integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.8.1 && go mod verifyv0.8.0 — Page.Tables() ruled-table extraction
Added
-
Page.Content()now reports stroked ruling lines — table cell borders, underlines,
and other vector rules drawn with the content stream's stroke operators — as
Content.Stroke, a slice ofStroke{From, To Point}segments, alongside the existing
text and rectangles. Endpoints are in the page's upright display space (honoring page
/Rotateandcmtransforms), the same coordinate space asContent.Rect. Bézier
curves break the run rather than forming a chord; fill-only and clip paths are excluded;
rectangles drawn with thereoperator stay inContent.Rect(even when stroked), so a
table-grid consumer should read bothStrokeandRect— some documents rule tables
with thin filled rectangles rather than stroked lines. This is the raw vector signal for
detecting table structure without rendering. Marked experimental: the type and field
are additive-stable, but which segments are emitted may be refined as table support
matures. -
Page.Tables()extracts ruled tables from a page, returning each as aTablewhose
Cells[row][col]holds the text of that cell. It reconstructs the grid from the page's ruling
lines — both stroked lines and thin filled rectangles — and assigns each word to its cell by
position, so dense numeric values that fragment under text-only heuristics land whole in the
right cell. Half-open edge columns common in statistical tables — the row-label column and the
last data column, which have no outer vertical rule — are recovered when the table's own
horizontal row rules extend into them (and only then, so margin text such as page numbers or a
sidebar label is never mistaken for a column). Only ruled tables are detected: borderless and
partially-ruled tables, and open columns whose rules stop at the inner border, return nothing
rather than guessing. Marked experimental — theTabletype and the detection geometry may
gain fields or be refined in a future minor release. SeeEXAMPLES.mdfor usage.
Performance
- Text extraction allocates less and runs faster on large pages. The content-stream
interpreter previously allocated a fresh operand slice for every operator; it now
reuses a per-pass scratch buffer, cutting garbage on the structured-extraction paths
(Page.Content,Reader.DebugJSON,ExtractionSummary, and the CJKWords/Texts
paths). Output is byte-identical. Measured same-machine A/B (Apple M4 Pro):
DebugJSONon a 22-page CJK PDF uses −13.9 % memory with −8.0 % allocations
and runs −5.1 % faster; CJK word extraction uses −10.4 % memory.
GetPlainTextis unaffected.
Fixed
-
Text extraction now recovers simple fonts (Type1/TrueType/Type3/MMType1) whose embedded
ToUnicodeCMap declares a 2-bytecodespacerangeover single-byte character codes — a
common pattern in Adobe-generated PDFs (notably subset TrueType fonts namedCIDFont+F*).
Previously the decoder chunked the content bytes by the ToUnicode codespace width, so it read
the 1-byte codes two at a time, matched none of the 1-bytebfcharentries, and emitted
U+FFFD (the replacement character) for the entire run; an affected page extracted as almost
pure replacement characters even though its text layer was fully valid (external tools such as
pdftotext/pdfminerrecovered the text). A simple font's character codes are single-byte by
definition (PDF 32000-1:2008 §9.6), so its ToUnicode is now decoded one byte per code,
independent of the declared codespace width. Composite (Type0) fonts are unchanged — their CMap
codespace width is authoritative. The change is byte-identical for simple fonts whose ToUnicode
already declares a 1-byte codespace, and for every font without a ToUnicode stream. -
Page.Content()andPage.Words()no longer emit a U+FFFD replacement character for the
word/line separators aTJarray synthesizes. Showing text with aTJarray appends a
newline after the array (and a space across a wide kerning gap) as a layout marker; the
interpreter routed that interpreter-chosen ASCII separator through the content font's encoder,
and a simple font whoseToUnicodeCMap has no entry for the0x0A/0x20byte decoded it to
U+FFFD — so a table laid out with oneTJarray per cell leaked one replacement character per
array into the structured-extraction stream (a real IRS tax-table page carried 190 of them in
Content().TextandWords()). The separator is already its own Unicode, so an unmappable
decode now falls back to the literal separator, matching the byte-identical passthrough every
other encoder already gave it.GetPlainTextwas never affected (it synthesizes no such
separator); cell-grid values were unaffected (the zero-advance markers fall at cell edges).
Verify this release
This tag is signed with a hardware-backed ED25519-SK key:
256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)
All of the maintainer's trusted signing keys are published on the GitHub account, so the recipe below verifies the tag regardless of which key signed it — nothing needs to be copied from these notes:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.0
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY...Module integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.8.0 && go mod verifyv0.7.9 — rotated and vertical text geometry
Added
-
Text.HandText.Rotationcomplete the per-glyph geometry on theTextstruct
(returned byPage.Texts(),Page.Content(), andReader.GetStyledTexts).
Text.His the nominal font-box height in points — the magnitude of the text
rendering matrix's up-vector, so it equals the font size for ordinary horizontal
text and stays positive and rotation-invariant for rotated runs, where
FontSize(taken from the matrix x-scale) collapses toward zero: on a 90°-rotated
run whoseFontSizereads 0,Hstill reports the true 12 pt.Text.Rotationis
the baseline's angle in degrees, counter-clockwise-positive from horizontal
(0for upright text) — the rendering matrix's baseline angle, distinct from and
opposite-signed to the page/Rotateattribute (clockwise), which is not applied
here. Both fields are additive (Textstays Stable-tier); no existing field value,
golden, orDebugJSONoutput changes.Word.H/Line.Hare unchanged (still the
first-glyph font size); unifying them with the rotation-awareText.His left to a
later vertical-layout change. Carrying the two newfloat64fields widensText
from 64 to 80 bytes, so the structured-extraction paths
(Texts/GetStyledTexts/Content/DebugJSON) allocate +8–13 % more bytes and
run +2–3.5 % slower, with the allocation count unchanged;GetPlainTextis a
separate path and is unaffected (measured same-machine A/B, Apple M4 Pro). A
fast-path that skips the per-glyph trigonometry on horizontal text was measured and
gave no benefit, confirming the cost is the wider struct rather than the
computation — so the simpler unconditional form ships. See EXAMPLES.md. -
Vertical writing-mode text (a
-Vpredefined CMap) now advances down the page
instead of horizontally. Previously such text decoded correctly but every glyph
advanced along the x-axis (or, for composite fonts, did not advance at all), so a
multi-glyph vertical run overprinted at a single point. The per-glyph advance and the
TJkerning adjustment now translate along the vertical axis using the PDF default
one-em displacement, and text following aTJarray in the same text block is no
longer pushed an extra line down. Thevertical_writing_modewarning message now
states that the advance is applied with a default-metrics approximation rather than
ignored. Per-glyph vertical metrics (/W2), the glyph position vector, and vertical
word/line grouping are not yet modelled; horizontal text,GetPlainText, and all
existing field values and goldens are unchanged. -
Page
/Rotate(the clockwise display-rotation attribute) is now honored, so
extracted text and image coordinates land in the page's upright display space instead
of the raw content space that previously ignored the rotation. A page authored sideways
with/Rotateset to display upright now extracts with a horizontal baseline — the font
size comes back, therotated_textwarning stops, and reading-order grouping works. A
newPage.Rotate() intaccessor returns the applied clockwise rotation
(0/90/180/270);/Rotateis inheritable and normalized (a non-multiple of 90,
an absent, or a non-integer value reads as 0).Text.Rotationnow reflects the combined
text-matrix and page rotation in display space. The change is additive and
golden-neutral: an unrotated page (/Rotate 0) is bit-for-bit unchanged,
GetPlainTextis unaffected, and every corpus,DebugJSON, and signal golden is
unchanged; the per-page cost is within measurement noise. See EXAMPLES.md.
Changed
-
Content().Rect(therepath-construction rectangles fromPage.Content()) now
honors the CTM: each rectangle's four corners are mapped through the current
transformation matrix and returned as their axis-aligned display-space bounding box, so
it agrees with the/Rotate-honored text and image geometry on a rotated or
cm-transformed page — the last mixed-frame surface after the page-/Rotatework. An
unrotated page with nocmtransform is bit-for-bit unchanged; the returned bbox is now
always normalized (Min <= Max), which additionally corrects un-normalized output for a
negative-dimensionre.DebugJSONdoes not emit content rectangles, so its goldens are
unaffected. -
Word.HandLine.H(and theBlockandDebugJSONboxes derived from them) are
now the rotation-aware nominal font height — the same basis asText.H— finishing
the unification noted whenText.Hshipped. Previously they came from the text
matrix's x-scale (FontSize), which collapses toward0on a 90°-rotated run, doubles
under horizontal text scaling, and goes negative under a horizontal flip; they now
report the text up-vector magnitude, which is rotation-invariant and always
non-negative, so a word, line, or block reports the same nominal height as the glyphs
in it. For ordinary horizontal text this value equals the font size exactly, so all
existing extraction output — every corpus golden,DebugJSON, andPage.Words()/
Page.Lines()result — is byte-for-byte unchanged; only rotated, flipped, or scaled
runs (none of which were previously locked by a test) take the new value. One
consequence on the ExperimentalPage.Blocks()API: because the corrected line height
also feeds block grouping, the way non-horizontal lines group into blocks can
change — in particular a 90°-rotated column, which used to collapse into a single block
because its old height read0, can now split into separate blocks.
Performance
- The text-extraction paths that group glyphs into lines —
Page.Words,
Page.Lines,Page.Blocks,Reader.DebugJSON, andPage.ExtractionSummary—
now allocate substantially less memory. The internal y-band grouping step used to
copy the page's glyph slice twice per page (a defensive copy plus per-band
value copies); it now sorts a lightweight index permutation and references the
original glyphs in place, leaving the page's content untouched. Output is
byte-for-byte identical. On a 22-page Traditional-Chinese document,
Reader.DebugJSONdrops −21.6 % bytes allocated (and ~1 % fewer allocations),
and the word/line path drops −29–31 % bytes. Time per operation is flat to
slightly lower — the workload is dominated by garbage-collection pressure, so
fewer bytes chiefly relieve GC. This more than offsets the +8–13 % bytes the new
Text.H/Text.Rotationfields added (above).
Verify this release
This is a signed git tag (ED25519-SK / FIDO2 hardware key). This release was
signed with:
256 SHA256:BlsTid0viIuF2db9zc/gY1JerD7mK47KH19brGGFKWM (ED25519-SK)
The maintainer's trusted signing keys are published on the GitHub account, so
nothing needs to be copied from these notes:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.7.9
# expect: Good "git" signature for * with ED25519-SK key SHA256:BlsTid0...Module integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.7.9 && go mod verifyv0.7.8 — document-level ToUnicode CMap cache
Performance
-
CJK and other ToUnicode-based text extraction is substantially faster and
lighter: a font's ToUnicode character map is now decoded once per document
instead of re-parsed on every page of every extraction pass. Repeated
extraction calls and multi-page documents reuse the decoded map, which is
bounded in memory and safe for concurrent use. Output is byte-identical — the
cache only changes when a map is parsed, never what it decodes to. On a
22-page Traditional-Chinese document,Reader.GetPlainTextdrops −28 % time,
−35 % bytes, −18 % allocations; cold open-and-extract −26 % / −31 % /
−17 %; andReader.DebugJSONa further −12 % / −10 % / −11 %. -
Reader.DebugJSON/Page.DebugJSONgot faster and lighter again: each page's
words are no longer assembled twice. Building on the v0.7.7 single-interpret change,
the structured-export path now reuses the words it already grouped into lines to
drive its routing summary, instead of re-grouping the page from scratch. On a
22-page CJK documentReader.DebugJSONdrops from 67.1 to 59.7 ms/op (−11 %),
74.8 to 59.3 MB/op (−21 %), and allocations from 921k to 863k per call
(−6 %). The output is byte-identical.
Verify this release
This is a signed git tag (ED25519-SK / FIDO2 hardware key). The maintainer's
trusted signing keys are published on the GitHub account, so nothing needs to be
copied from these notes:
curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
| python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.7.8
# expect: Good "git" signature for * with an ED25519-SK keyModule integrity is independently guaranteed by the Go checksum database:
go mod download github.com/Detective-XH/gopdf@v0.7.8 && go mod verify