Skip to content

v1.1.0

Latest
Latest

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 21 Aug 13:08
· 6 commits to main since this release
v1.1.0
2d8e244
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 1.1.0 requires python >= 3.11

Note

docTR is now a full document understanding toolkit. On top of text detection and recognition, v1.1.0 adds layout analysis, table structure recognition and a reading-order aware export to Markdown / AsciiDoc / HTML / XML - plus a CLI so you can run the whole pipeline without writing a single line of Python.

What's Changed

Breaking Changes 🛠

  • The minimum supported Python version was raised from 3.10 to 3.11
  • The opencv-python upper bound was relaxed to <6.0.0

New features

  • Layout analysis with the new LW-DETR architecture (lw_detr_s, lw_detr_m) by @felixdittrich92
  • Table structure recognition with TableCenterNet and a StarNet backbone by @felixdittrich92
  • Reading order module and new exporters (Markdown / AsciiDoc / HTML / XML) by @felixdittrich92
  • Vocab whitelisting to constrain the recognition model to a known character set by @felixdittrich92
  • doctr-cli to run end-to-end OCR from your shell by @simont2k
  • Layout masking (ignore_regions) to skip page furniture before detection/recognition by @felixdittrich92
  • preserve_original_coords to map boxes back to the original page after straighten_pages=True by @saad-rd11
  • New dataset classes (LayoutDataset, TableStructureDataset), new metrics (ObjectDetectionMetric, TableCellMetric) and new reference scripts for both tasks

✨ Layout Analysis ✨

Layout analysis localizes and classifies the visual elements of a page - not just the words. The new LW-DETR models are trained on 11 classes (Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, Title) and, unlike most layout detectors, also work on rotated and skewed documents.

layout_example

Left: words found by db_resnet50 + crnn_vgg16_bn. Right: the 18 regions returned by lw_detr_s on the same page.

Available architectures:

Architecture Input shape # params mAP@[.5:.95] AP@[.5] AP@[.75] sec/it (B: 1) - CPU
lw_detr_s (1024, 1024, 3) 15.1 M 66.89 82.83 73.75 0.5
lw_detr_m (1024, 1024, 3) 29.5 M soon soon soon 0.7

Standalone usage

import numpy as np
from doctr.models import layout_predictor
 
model = layout_predictor("lw_detr_s", pretrained=True)
dummy_img = (255 * np.random.rand(800, 600, 3)).astype(np.uint8)
out = model([dummy_img])
# out[0] -> {"class_names": ["Title", "Text", ...], "boxes": array(...), "scores": array(...)}

Like the detection predictor, it accepts assume_straight_pages, preserve_aspect_ratio and symmetric_pad:

from doctr.models import layout_predictor
 
predictor = layout_predictor(
    "lw_detr_s", pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True
)

Inside the OCR pipeline

Pass detect_layout=True to the ocr_predictor (or the kie_predictor) and the detected regions are attached to every page, exported alongside the rest of the page and rendered by .show():

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
 
model = ocr_predictor(pretrained=True, detect_layout=True, layout_arch="lw_detr_s")
doc = DocumentFile.from_images("path/to/your/doc.jpg")
result = model(doc)
 
# Access the detected layout regions of the first page
for region in result.pages[0].layout:
    print(region.type, region.confidence, region.geometry)
# Title 0.97 ((0.13, 0.06), (0.87, 0.11))
# Text 0.95 ((0.11, 0.14), (0.89, 0.42))
# Table 0.93 ((0.12, 0.45), (0.88, 0.79))
 
# The layout is part of the exported representation
export = result.pages[0].export()
print(export["layout"])
 
# Overlay both text and layout regions (use display_layout=False to hide the regions)
result.pages[0].show()

Ignoring regions

masking_example

Layout regions can also be used to remove content before it ever reaches the detection and recognition models. Pass the class names you want to drop to ignore_regions — the corresponding areas are masked out (filled with black), which speeds up inference and keeps page furniture out of your results:

from doctr.models import ocr_predictor
 
# Headers, footers and pictures are masked out before detection & recognition
model = ocr_predictor(pretrained=True, ignore_regions=["Page-header", "Page-footer", "Picture"])

✨ Table Structure Recognition ✨

Table structure recognition parses a table into a machine-understandable representation: it localizes every cell (spatial structure) and recovers the rows and columns each cell spans (logical structure). The new TableCenterNet architecture is built on a lightweight StarNet backbone (7.1 M params).

table_example

The 20 cells predicted by tablecenternet with their logical (row, column) coordinates, and the dense grid to_grid() builds from them.

Architecture Input shape # params Recall Precision F1 Structure acc sec/it (B: 1)
tablecenternet (1024, 1024, 3) 7.1 M 82.31 96.01 88.64 77.53 0.5

Metrics are produced by references/table/evaluate.py using the new TableCellMetric: cell-detection Recall, Precision and F1 (cells matched above an IoU threshold of 0.5), and Structure acc, the share of matched cells whose logical (row/column) coordinates are correctly predicted.

Standalone usage

import numpy as np
from doctr.models import table_predictor
 
model = table_predictor("tablecenternet", pretrained=True)
table_crop = (255 * np.random.rand(800, 600, 3)).astype(np.uint8)
out = model([table_crop])
# out[0] -> {"cells": [{"geometry": ..., "score": ..., "row_start": 0, "row_end": 0,
#            "col_start": 0, "col_end": 0}, ...], "num_rows": ..., "num_cols": ...}

Inside the OCR pipeline

Passing detect_tables=True to the ocr_predictor wires layout detection and table structure recognition together: every region the layout model labels as a Table is cropped and passed to the table model (so detect_tables=True implicitly enables the layout model). Words whose center falls inside a detected cell are regrouped into a structured table available under page.tables - and removed from the regular blocks output, so the same text is never returned twice.

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
 
model = ocr_predictor(pretrained=True, detect_tables=True)
doc = DocumentFile.from_images("invoice_with_table.png")
result = model(doc)
 
page = result.pages[0]
for i, table in enumerate(page.tables):
    print(f"Table {i} ({table.num_rows}x{table.num_cols}):")
    print(table.to_grid())  # dense list of lists -> loads straight into pandas
 
# The remaining (non-table) text is still available as usual
print(page.render())
# Table 0 (3x3):
# [['Item', 'Qty', 'Price'],
#  ['Coffee', '2', '7.00'],
#  ['Tea', '1', '3.50']]
 
import pandas as pd
grid = page.tables[0].to_grid()
df = pd.DataFrame(grid[1:], columns=grid[0])

Tables are included in Page.export() under the tables key, so they survive the JSON export as well.

✨ Reading order & new exporters ✨

markdown_example

Results can now be linearized in reading order and exported to Markdown, AsciiDoc, HTML or XML. The content is ordered column by column, the reading direction is inferred from the recognized text (e.g. right-to-left for Arabic or Hebrew documents), and - when the predictor runs with detect_layout=True - the layout regions are used to render headings, list items and recognized tables, and to place page furniture (headers, footers, footnotes).

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
 
model = ocr_predictor(pretrained=True, detect_layout=True)
result = model(DocumentFile.from_pdf("path/to/your/doc.pdf"))
 
markdown_output = result.export_as_markdown()
asciidoc_output = result.export_as_asciidoc()
html_output = result.export_as_html()
xml_output = result.export_as_xml()          # hOCR
raw_text_output = result.render()            # same as result.export_as("text")
dict_output = result.export()                # same as result.export_as("json")

export_as is a convenience dispatcher over all formats:

result.export_as("markdown")  # or "md"
result.export_as("asciidoc")  # or "adoc"
result.export_as("html")
result.export_as("text")      # same as render()
result.export_as("json")      # same as export()
result.export_as("xml")       # same as export_as_xml()

Sample Markdown output:

# Invoice 2024-042
 
## Billing address
 
Mindee, 54 Rue de Paradis, 75010 Paris
 
| Item | Qty | Price |
| --- | --- | --- |
| Coffee | 2 | 7.00 |
| Tea | 1 | 3.50 |

Every export path shares the same linearization, so render(), export(), export_as_xml() and the Markdown / AsciiDoc / HTML exports all present the content in the same order. The result is memoized on the page, so exporting one page to several formats orders it only once.

If you want the document structure itself in reading order, build the predictor with keep_reading_order=True, which sorts the blocks of every page:

predictor = ocr_predictor(pretrained=True, keep_reading_order=True)

✨ Vocab whitelisting ✨

If you only expect text from one or more known languages, you can now whitelist the corresponding vocabs so that the recognition model can no longer predict any character outside of them.

The whitelist is enforced at the model's final projection layer, before the decoding argmax. Because every logit flows through that projection, the constraint also applies inside the autoregressive decoding loop of SAR, MASTER and PARSeq - a forbidden character can never be produced, not even fed back mid-word. It works with every recognition architecture and with any predictor wrapping one (ocr_predictor, kie_predictor, recognition_predictor).

from doctr.datasets import VOCABS
from doctr.io import DocumentFile
from doctr.models import ocr_predictor
from doctr.models.utils import add_whitelist
 
predictor = ocr_predictor(pretrained=True)
 
# The recognition model can now only predict Polish/German characters
handle = add_whitelist(predictor, [VOCABS["polish"], VOCABS["german"]])
 
input_page = DocumentFile.from_images("path/to/your/image.png")
out = predictor(input_page)
 
# Restore the original, unconstrained decoding
handle.remove()

The returned handle also works as a context manager:

with add_whitelist(predictor, VOCABS["german"]):
    out = predictor(input_page)  # only German characters can be predicted here
# the whitelist is automatically removed outside of the `with` block

Two strategies are available:

  • strategy="mask" (default): forbidden logits are set to -inf, so decoding falls back to the highest-scoring allowed character.
  • strategy="nearest": each forbidden character is first folded onto the closest allowed one (ä -> a, ł -> l), which is handy to normalize accents/diacritics onto a base alphabet.
# Fold any non-ASCII character onto its closest ASCII letter (e.g. é -> e, ł -> l)
handle = add_whitelist(predictor, VOCABS["latin"], strategy="nearest")
out = predictor(input_page)
handle.remove()

The mapping is built by transliteration by default; pass mapping="weights" to derive it from the model's own learned confusions, or a {forbidden_char: allowed_char} dict to override specific characters.

Note

A whitelist can only restrict a model to characters it already knows - characters outside the model's own vocabulary are silently ignored. Make sure the model was trained on a vocab that covers your languages (e.g. a multilingual model).

✨ Command Line Interface ✨

The full OCR pipeline can now be run from your shell on images and PDFs, with the results exported to JSON - no Python required.

doctr-cli --input_path path/to/your/document.pdf --output results.json
cli_example

A few more examples:

# Run OCR on an image
doctr-cli --input_path image.jpg --output ocr_res.json
 
# Pick your architectures and straighten skewed pages
doctr-cli --input_path doc.pdf --det_arch db_mobilenet_v3_large --reco_arch crnn_vgg16_bn --straighten_pages
 
# Rotated documents, with orientation and language detection
doctr-cli --input_path doc.pdf --no-assume_straight_pages --detect_orientation --detect_language

The JSON output follows the usual docTR document model: pages (dimensions & orientation), blocks, lines and words with their confidence scores and bounding boxes.

Documentation: https://mindee.github.io/doctr/latest/using_doctr/using_cli.html

Further improvements

  • Keep the original coordinates when straightening pages - with straighten_pages=True, boxes are remapped onto the original page instead of the rotated one, which is what you want for redaction and annotation:
  from doctr.models import ocr_predictor
 
  model = ocr_predictor(pretrained=True, straighten_pages=True, preserve_original_coords=True)
  • New metrics - a COCO-mAP-like ObjectDetectionMetric for layout evaluation and a TableCellMetric for table structure evaluation
  • Transforms & augmentations now operate on a unified Sample container (image, mask, target), which makes them reusable across detection, layout and table tasks
  • PreProcessor / Resize logic was reworked and rotated post-processing was improved
  • Detection reference scripts can now use the built-in datasets directly (as recognition already could)
  • Reconstitution - synthesize_page got better font scaling, rendering and font fallback
  • Performance - a bfloat16 issue was fixed, MASTER got noticeably faster, and general memory/latency improvements landed across the pipeline
  • Rotated layouts are de-skewed before the reading order is computed
  • CI/CD jobs were refactored with much better caching

What's Changed

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: v1.0.1...v1.1.0

Contributors

badranX, felixdittrich92, and 5 other contributors
Assets 2
Loading