Add pdf-inspector as a pre-OCR classification step in the PDF ingestion pipeline

[heejun32]

What are you trying to do?

I'm ingesting PDF sources (lecture slides, papers, scanned handouts) into Open Notebook. Some are text-based exports (e.g. from PowerPoint/Keynote) and some are actual scans, so the ingestion pipeline needs to decide, per document, whether OCR is required before extracting text.

What feels difficult or missing today?

Today, when Docling is enabled (OPEN_NOTEBOOK_ENABLE_DOCLING=true), scanned PDFs and images go through OCR, but there doesn't seem to be a lightweight pre-check that determines whether a PDF actually needs OCR before the heavier Docling/OCR pipeline runs. For text-based PDFs (a large share of real documents, including most lecture slide exports), this means the OCR/Docling path may still be invoked even though the text layer is already extractable, adding unnecessary latency and resource cost.

What outcome would help?

Text-based PDFs should skip the OCR/Docling path entirely and go straight to fast, local text extraction, while only the pages that genuinely lack a text layer get routed to OCR. Ideally this decision happens automatically and cheaply (on the order of milliseconds per document), without the user needing to know in advance whether a given PDF requires OCR.

How do you handle this today?

No response

Additional context, examples, or possible directions

No response

How would you like to participate?

• I can add use cases or join the product/design discussion.
• I can help explore the technical approach.
• I can help implement an approved work item.
• I can test or validate a proposed solution.

Before posting

• I searched existing Discussions and Issues for related proposals.

[lfnovo]

@heejun32 yes, and the behaviour you describe is real. I checked content-core's routing: with the engine on auto and Docling installed, every PDF goes to Docling, whether or not it already has a text layer. The lightweight PDF extractor is only used when Docling isn't available.

The fix belongs in content-core, the extraction library Open Notebook uses. Open Notebook only passes the engine choice through. I opened lfnovo/content-core#98 with the scope; since you offered to implement it, it's yours there. Two design points to settle in the PR:

1. Does it need pdf-inspector? A heuristic on what pdfplumber already returns (characters per page) might be enough without a new dependency. If pdf-inspector is clearly better on real documents (for example scans with an embedded invisible OCR layer), show it there.
2. Per document or per page? Per document is a fine first step; per-page routing for mixed PDFs can follow.

Once it's released in content-core, Open Notebook picks it up with a dependency bump.

Status: accepted → upstream (lfnovo/content-core#98).