Skip to content

PDFluent SDK 1.4.0

Latest
Latest

Choose a tag to compare

View all tags
@jasperdew jasperdew released this 13 Nov 18:31
v1.4.0
eb010f2

PDFluent SDK 1.4.0

Tag: v1.4.0 · Licence: AGPL-3.0 or the PDFluent Commercial Licence

PDFluent 1.4.0 improves default extraction, fixes OCR that returned nothing on some installations, and ships PDF/A conversion as a full feature. It ships for Rust, Python, Node.js, WebAssembly, Java, .NET and C.

Licence and models

The licence is unchanged from 1.3.0: the SDK is available under the GNU AGPL-3.0 or, at your option, the PDFluent Commercial Licence. See https://pdfluent.com/license/ for the commercial terms.

The model files now come from a new models release, models-v2, with a notice for each model. Our layout and formula models are licensed under the same terms as the SDK; the layout models are fine-tuned from IBM's docling-layout-heron-101, which is Apache-2.0. The table model is Microsoft's Table Transformer, unmodified, under its MIT licence. pdfluent models install fetches from models-v2 and checks every file against a pinned digest, as before. Installers from 1.3.0 and earlier keep reading the models-v1 release.

OCR no longer returns nothing (#1136)

OCR could return an empty page without any error. The Tesseract engine asked for word positions through the tsv config file. On an installation whose tessdata folder holds only traineddata files, as our own OCR component bundle does, Tesseract printed plain text, the parser read no words, and the page fell back to a figure placeholder.

1.4.0 requests the word positions with an explicit parameter. Output that is not in the expected form now fails with an error that names the cause and quotes Tesseract's first line and its stderr. A regression test covers an image-only page.

Affected: SDK 1.3.0 and earlier, on installations whose tessdata folder has no configs folder. The fix is in 1.4.0 only; there is no hotfix for earlier releases.

Extraction

quality="auto" follows more of the structure a document has. Scores marked with an asterisk are explained under the list.

  • Headings: heading detection now also follows the document's own size and weight signals, keeps stacked short headings apart, and refuses to call text over a figure a heading. On opendataloader-bench, heading hierarchy (MHS) goes from 0.8678 to 0.8767*.
  • Tables and lists: ruled tables are read from their lines, header rows are recovered, bracketed list labels survive, and a label that wraps over two rows is no longer taken for a section heading. On opendataloader-bench, table structure (TEDS) goes from 0.8695 to 0.9140*.
  • Reading order: a caption beside a picture follows the picture, a column of reference rows is no longer treated as a second text column, and a left sidebar is read first. On opendataloader-bench, reading order (NID) goes from 0.9359 to 0.9463*.
  • Code and forms: monospace listings are written as indented code where the text reads as code, and a selected radio button in a row is marked.
  • Formulas: our own formula model now reads formula-heavy documents by default and writes LaTeX without token spaces, with the equation number kept as a tag. A document is routed to it only when the layout model finds at least three display formulas, at least one per two pages, and the formulas cover at most a tenth of the page on average. Set formula_model=False to switch it off, or formula_latex=True to read every formula. A document the route leaves alone reads exactly as in 1.3.0.
  • OCR: tilted scans are read upright, a scan that carries its scanner's invisible text layer keeps that layer in the auto tier, and Apple Vision and Windows.Media.Ocr are available as an opt-in OCR engine. Tesseract stays the default.
  • Markdown output: link URLs, image placeholders and a final-render choice for figures, with options in Python, Node.js, C, WebAssembly, Java and .NET.
  • Every extraction decision has a rule id in the routing trace, so you can see which rule changed a block.

* opendataloader-bench, all 200 documents, quality="auto" with keep_furniture=True, measured with the benchmark's own harness and evaluator on the release wheel for macOS, with the layout, table and formula models installed. Overall score 0.9282, against 0.9154 for 1.3.0 in the same setup on the same kind of machine (Apple M1 Pro). The benchmark was one of our development targets, so read these as our own measurements and not as an independent ranking. One run each, no confidence interval. Both runs keep page headers and footers, because the benchmark counts them as content. Table scores differ between platforms, so compare Mac with Mac.

Other inputs

The extraction paths now accept more than PDF. Delimited and structured data, RTF, images (through OCR), mail, office files, EPUB and markup documents become the same blocks as PDF pages, so Markdown, JSON and chunking work on them unchanged. Every input is bounded in bytes, nesting depth and output size, and a bound that is hit returns a typed error instead of a partial result. Node.js declares the accepted formats on extractInput.

PDF engine

pdf-syntax, pdf-font, pdf-interpret and the CCITT decoder are levelled with upstream hayro 0.9, and lopdf is merged with upstream 0.45 and 0.46. Rendering runs on the new interpreter device API. Annotations flagged NoView are no longer drawn; clip-only text stays invisible.

PDF/A conversion

PDF/A conversion ships as a full feature in 1.4.0. It converts a document to PDF/A, repairs fonts and widths, embeds subsets, adds the output intent and metadata, and keeps the document's text. The size of the quota it allows itself scales with the document, and a scan is no longer refused for it.

We measured it once on a sealed holdout of 1,000 documents from Zenodo, Wikimedia Commons and arXiv, published between 2007 and 2026. Checked with veraPDF 1.28.2, 993 of the 1,000 converted files conform to PDF/A (99.3%), and one conversion was refused. Real-word text retention at the 5th percentile is 98.4%, so in about one document in twenty, 1.6% or more of the words are lost. Our own target was 99.5% for both, and we did not reach it. The output is 96.6% of the input size in total, and 99.1% for the median document. The measurement used the converter as it stood on 7 November; later changes were not measured again.

Compatibility

The PyMuPDF-compatible layer stays a preview. Its similarity scan and audit records for 1.4.0 are part of the release evidence.

PyMuPDF and MuPDF are trademarks of Artifex Software, Inc. PDFluent is an independent product of Innovation Trigger B.V. and is not affiliated with or endorsed by Artifex.

Fixes

  • ToUnicode bfrange expansion is bounded by source width and by stream and document budgets; a rejected expansion is a validation failure.
  • Hidden optional-content text keeps its position, AnyOff is evaluated correctly, and shared page streams are copied before flattening. PDF/A-1 mask and blend-mode removal reports a fidelity warning.
  • A glyph that ToUnicode maps to U+FFFD is read by its glyph name.
  • The WebAssembly package builds again: the PDF/A quota ceiling no longer overflows a 32-bit address space.
  • Release promotion requires engine, bindings and packaging reviews of the exact promoted commit, each ending with VERDICT: SHIP.

Known issues

  • Page headers and footers on the olmOCR headers and footers category score 0.0013 lower than in 1.3.0 on the auto tier (olmOCR-bench, one run). keep_furniture=True keeps heads and feet on every tier.
  • Pages where display equations cover more than a tenth of the page on average stay outside the formula route on purpose, so they read as in 1.3.0.

Component crates

pdf-engine, pdf-render, pdf-annot, pdf-compliance, pdfluent-forms and pdfluent-sign move to a new major version; pdfluent and all bindings stay compatible with 1.3.

One edge case: a Rust program that also depends directly on a 1.3 version of one of these component crates, and passes types between that crate and pdfluent, moves that component crate to its new major version as well.

Install and upgrade

cargo add pdfluent@1.4.0
pip install pdfluent==1.4.0
npm install @pdfluent/node
npm install @pdfluent/sdk-wasm
dotnet add package PDFluent --version 1.4.0

Java: com.pdfluent:pdfluent 1.4.0 on the PDFluent Maven repository, https://maven.pdfluent.com

Upgrading from 1.3:

  • Re-run pdfluent models install layout tables formula to fetch the models and their notices from models-v2.
  • The formula model is on by default for formula-heavy documents. Pass formula_model=False to keep the 1.3.0 reading.
  • An OCR layer made by 1.3.0 on an installation without a configs folder may be an empty page. Run OCR again on the originals.

Questions and problems: https://github.com/pdfluent/pdfluent-sdk/discussions · Full changelog: https://pdfluent.com/changelog/

Assets 18
Loading