Skip to content

v0.8.0 — Page.Tables() ruled-table extraction

Choose a tag to compare

@Detective-XH Detective-XH released this 14 Jun 17:07
· 126 commits to master since this release
v0.8.0
1d38caa

Added

  • Page.Content() now reports stroked ruling lines — table cell borders, underlines,
    and other vector rules drawn with the content stream's stroke operators — as
    Content.Stroke, a slice of Stroke{From, To Point} segments, alongside the existing
    text and rectangles. Endpoints are in the page's upright display space (honoring page
    /Rotate and cm transforms), the same coordinate space as Content.Rect. Bézier
    curves break the run rather than forming a chord; fill-only and clip paths are excluded;
    rectangles drawn with the re operator stay in Content.Rect (even when stroked), so a
    table-grid consumer should read both Stroke and Rect — some documents rule tables
    with thin filled rectangles rather than stroked lines. This is the raw vector signal for
    detecting table structure without rendering. Marked experimental: the type and field
    are additive-stable, but which segments are emitted may be refined as table support
    matures.

  • Page.Tables() extracts ruled tables from a page, returning each as a Table whose
    Cells[row][col] holds the text of that cell. It reconstructs the grid from the page's ruling
    lines — both stroked lines and thin filled rectangles — and assigns each word to its cell by
    position, so dense numeric values that fragment under text-only heuristics land whole in the
    right cell. Half-open edge columns common in statistical tables — the row-label column and the
    last data column, which have no outer vertical rule — are recovered when the table's own
    horizontal row rules extend into them (and only then, so margin text such as page numbers or a
    sidebar label is never mistaken for a column). Only ruled tables are detected: borderless and
    partially-ruled tables, and open columns whose rules stop at the inner border, return nothing
    rather than guessing. Marked experimental — the Table type and the detection geometry may
    gain fields or be refined in a future minor release. See EXAMPLES.md for usage.

Performance

  • Text extraction allocates less and runs faster on large pages. The content-stream
    interpreter previously allocated a fresh operand slice for every operator; it now
    reuses a per-pass scratch buffer, cutting garbage on the structured-extraction paths
    (Page.Content, Reader.DebugJSON, ExtractionSummary, and the CJK Words/Texts
    paths). Output is byte-identical. Measured same-machine A/B (Apple M4 Pro):
    DebugJSON on a 22-page CJK PDF uses −13.9 % memory with −8.0 % allocations
    and runs −5.1 % faster; CJK word extraction uses −10.4 % memory.
    GetPlainText is unaffected.

Fixed

  • Text extraction now recovers simple fonts (Type1/TrueType/Type3/MMType1) whose embedded
    ToUnicode CMap declares a 2-byte codespacerange over single-byte character codes
    — a
    common pattern in Adobe-generated PDFs (notably subset TrueType fonts named CIDFont+F*).
    Previously the decoder chunked the content bytes by the ToUnicode codespace width, so it read
    the 1-byte codes two at a time, matched none of the 1-byte bfchar entries, and emitted
    U+FFFD (the replacement character) for the entire run; an affected page extracted as almost
    pure replacement characters even though its text layer was fully valid (external tools such as
    pdftotext/pdfminer recovered the text). A simple font's character codes are single-byte by
    definition (PDF 32000-1:2008 §9.6), so its ToUnicode is now decoded one byte per code,
    independent of the declared codespace width. Composite (Type0) fonts are unchanged — their CMap
    codespace width is authoritative. The change is byte-identical for simple fonts whose ToUnicode
    already declares a 1-byte codespace, and for every font without a ToUnicode stream.

  • Page.Content() and Page.Words() no longer emit a U+FFFD replacement character for the
    word/line separators a TJ array synthesizes
    . Showing text with a TJ array appends a
    newline after the array (and a space across a wide kerning gap) as a layout marker; the
    interpreter routed that interpreter-chosen ASCII separator through the content font's encoder,
    and a simple font whose ToUnicode CMap has no entry for the 0x0A/0x20 byte decoded it to
    U+FFFD — so a table laid out with one TJ array per cell leaked one replacement character per
    array into the structured-extraction stream (a real IRS tax-table page carried 190 of them in
    Content().Text and Words()). The separator is already its own Unicode, so an unmappable
    decode now falls back to the literal separator, matching the byte-identical passthrough every
    other encoder already gave it. GetPlainText was never affected (it synthesizes no such
    separator); cell-grid values were unaffected (the zero-advance markers fall at cell edges).

Verify this release

This tag is signed with a hardware-backed ED25519-SK key:

256 SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY (ED25519-SK)

All of the maintainer's trusted signing keys are published on the GitHub account, so the recipe below verifies the tag regardless of which key signed it — nothing needs to be copied from these notes:

curl -s https://api.github.com/users/Detective-XH/ssh_signing_keys \
  | python3 -c "import json,sys; [print('*', k['key']) for k in json.load(sys.stdin)]" > allowed_signers
git -c gpg.ssh.allowedSignersFile=./allowed_signers tag -v v0.8.0
# expect: Good "git" signature for * with ED25519-SK key SHA256:duCP4h22hb2oNAZMaFhUlpq0j8+qBbZuaXnS99yUhkY...

Module integrity is independently guaranteed by the Go checksum database:

go mod download github.com/Detective-XH/gopdf@v0.8.0 && go mod verify