Proposal: ONNX layout detection + .NET data ingestion bridge for PdfPig

[luisquintanilla]

Summary

.NET developers building AI applications need a way to go from PDF to structured, AI-ready content. Python has Docling and PyMuPDF. Node has tools like LiteParse. .NET has few if any equivalents in the space, and PdfPig provides a good foundation to build one on.

I prototyped two new packages on a fork (feature/intelligent-pdf-ingestion branch) that extend PdfPig in two directions:

1. UglyToad.PdfPig.DocumentLayoutAnalysis.Onnx: ML-powered layout detection via ONNX, implemented as a drop-in IPageSegmenter
2. UglyToad.PdfPig.DataIngestion: Connects PdfPig to the .NET data ingestion building blocks (Microsoft.Extensions.DataIngestion), so PdfPig becomes a first-class document reader in .NET AI pipelines

I'd love to get your thoughts on whether you'd be open to accepting either or both upstream before I clean them up into formal PRs.

---

Package 1: DocumentLayoutAnalysis.Onnx

This one extends PdfPig's existing IPageSegmenter interface with ONNX model support. Same input (Word objects), same output (TextBlock results). Instead of heuristics, it runs an ONNX layout detection model to identify regions.

How it works:

1. Render PdfPig Word bounding boxes as a page image (filled rectangles on white background, which is the format layout detection models expect)
2. Run the image through an ONNX model (preprocess → inference → postprocess)
3. Map detected regions back to PdfPig Word objects via bounding box overlap
4. Return TextBlock results, same as DocstrumBoundingBoxes or RecursiveXYCut

Here's what the API looks like:

using var document = PdfDocument.Open("paper.pdf");
var page = document.GetPage(1);

// Create an ONNX-powered segmenter with the Docling RT-DETR model
var model = new RtDetrLayoutModel("path/to/model.onnx");
var segmenter = new OnnxPageSegmenter(model);

// Same API as DocstrumBoundingBoxes or RecursiveXYCut. Just swap the segmenter.
var blocks = segmenter.GetBlocks(page.GetWords());

That's it. Same IPageSegmenter interface, same GetBlocks call.

New types:

Type	What it does
OnnxPageSegmenter	IPageSegmenter implementation. Drop-in replacement for existing segmenters.
ILayoutDetectionModel	Strategy interface for swapping ONNX models without changing pipeline code. Abstracts model-specific preprocessing (image resize, tensor format) and postprocessing (output parsing, coordinate mapping).
RtDetrLayoutModel	Implementation for Docling Layout Heron RT-DETR v2. Supports 17 element types (text, table, section_header, picture, formula, code, list_item, caption, footnote, page_header, page_footer, etc.)
ConfigurableLayoutModel	Generic adapter for custom ONNX models with configurable input dimensions and label mappings
ImagePreprocessing	Static utilities: letterbox/exact resize, HWC→CHW tensor conversion, normalization
DetectionPostprocessing	Static utilities: NMS, IoU, coordinate transforms (model-space → page-space)
LayoutDetection	Record type: BoundingBox (PdfRectangle), Label, ClassId, Confidence
PageImageRenderer	Renders Word bounding boxes as filled rectangles for model input
OnnxSegmenterOptions	Configures confidence threshold, ONNX Runtime SessionOptions, render DPI

Dependencies: ONNX Runtime, SkiaSharp (for page rendering). These are only pulled in by users who reference this package. No impact on the core PdfPig package.

---

Package 2: DataIngestion

.NET recently shipped data ingestion building blocks (Microsoft.Extensions.DataIngestion). These are a set of abstractions for composing AI data pipelines: read documents → process/enrich → chunk → embed → store. The pipeline is built around a few key interfaces:

• IngestionDocumentReader: reads files into a structured IngestionDocument
• IngestionDocumentProcessor: transforms/enriches documents before chunking
• IngestionChunkProcessor<T>: enriches individual chunks after splitting
• IngestionPipeline<T>: chains it all together

The DataIngestion package implements these interfaces for PdfPig. It provides:

Type	What it does
PdfPigReader	IngestionDocumentReader implementation. Reads PDFs into IngestionDocument with per-page sections. Supports three reading modes: TextOnly (native PdfPig extraction), Hybrid (text + page images), and VisionOnly (renders every page as PNG for vision LLM processing). Delegates segmentation to any IPageSegmenter.
VisionOcrEnricher	IngestionDocumentProcessor that sends page images to a vision LLM via IChatClient to extract text. Makes VisionOnly mode work. Useful for scanned documents and image-heavy PDFs.
VisionTableEnricher	IngestionDocumentProcessor that sends table regions to a vision LLM and gets back markdown tables.
ContextualChunkEnricher	IngestionChunkProcessor<string> that generates contextual summaries per chunk via IChatClient for better retrieval.
PageImageRenderer	Renders actual PDF text characters to PNG (unlike the ONNX renderer which draws bounding box rectangles). Used for VisionOnly/Hybrid modes.

Why this matters for PdfPig: it means a developer can go from PdfDocument.Open all the way to a vector store with a composable pipeline. PdfPig becomes a first-class reader in the .NET AI data ingestion ecosystem, similar to the built-in MarkdownReader and MarkItDown readers.

Here's a full pipeline using both packages together:

// 1. ONNX layout detection (DocumentLayoutAnalysis.Onnx package)
var model = new RtDetrLayoutModel("path/to/model.onnx");
var segmenter = new OnnxPageSegmenter(model);

// 2. PdfPig reader with ONNX-powered segmentation (DataIngestion package)
//    The reader delegates layout analysis to the OnnxPageSegmenter,
//    then maps detected regions into IngestionDocument sections.
var pdfReader = new PdfPigReader(segmenter, mode: PdfReadingMode.Hybrid);

// 3. Full ingestion pipeline: read → enrich → chunk → store
using var pipeline = new IngestionPipeline<string>(
    reader: pdfReader,
    chunker: new SemanticSimilarityChunker(embeddingGenerator),
    writer: writer)
{
    DocumentProcessors = {
        new VisionOcrEnricher(chatClient),      // Vision LLM extracts text from page images
        new VisionTableEnricher(chatClient)     // Vision LLM converts tables to markdown
    },
    ChunkProcessors = {
        new ContextualChunkEnricher(chatClient) // LLM adds contextual summaries per chunk
    }
};

await foreach (var result in pipeline.ProcessAsync(directory, "*.pdf")) { ... }

The OnnxPageSegmenter handles layout detection. The PdfPigReader turns that into IngestionDocument sections. The IngestionPipeline enriches, chunks, and writes everything to a vector store. Each piece is swappable.

Dependencies: Microsoft.Extensions.DataIngestion, Microsoft.Extensions.AI, SkiaSharp. Again, separate package. No impact on core PdfPig.

A sample app (PdfAIngest) shows the full end-to-end: Blazor UI + PdfPig in VisionOnly mode + MEDI pipeline + Qdrant vector search.

---

Design decisions

A few things that might be relevant for your review:

• Additive, not breaking. Both packages are new projects. No changes to existing types, APIs, or behavior. Users who don't need these capabilities have zero impact.
• IPageSegmenter is the right interface. The ONNX segmenter extends your existing architecture. Same input, same output. PdfPig already does the hard work of getting accurate Word positions. The ONNX model just adds region classification on top.
• ILayoutDetectionModel as a strategy. Model-specific logic (tensor format, output parsing) is encapsulated behind an interface. The segmenter itself is model-agnostic. Ships with RtDetrLayoutModel (17 element types) and ConfigurableLayoutModel (bring your own).
• Word-level spatial mapping. The ONNX model detects WHERE layout regions are; PdfPig tells us WHAT text is in those regions. Mapping uses bounding box overlap. Works well because PdfPig's Word objects already have accurate spatial coordinates.
• Separate packages = opt-in dependencies. ONNX Runtime and SkiaSharp only come in through the new packages. The DataIngestion package pulls in Microsoft.Extensions.AI and Microsoft.Extensions.DataIngestion. Core PdfPig stays clean.

---

Questions

Before I invest in polishing these into PRs, a few things I'd like to run by you:

1. Would you be open to accepting one or both of these upstream? Both are additive. Separate projects, separate NuGet packages, no changes to existing code.
2. Are the transitive dependencies acceptable? ONNX Runtime + SkiaSharp for the layout package; MEAI + MEDI for the ingestion bridge. Only pulled in if you reference the specific package.
3. Anything you'd want changed in the API? I'm happy to iterate on the shape.
4. Testing preferences? I can add unit and integration tests. Let me know if you have a preferred structure or framework.
5. Would you prefer one PR or two? I could submit the ONNX layout package first (closer to the existing DocumentLayoutAnalysis package) and follow up with DataIngestion, or do both together.

---

Thanks for PdfPig. The IPageSegmenter design made this a natural extension to build. Looking forward to hearing your thoughts.

[BobLd]

@luisquintanilla thanks for posting the above, everything you said makes a lot of sense and is in line with my original idea when I started working on the Document Layout Analysis part. This is really exciting and from my point of view, choosing Onnx makes complete sense, and Data Ingestion is critical.

I believe the best approach to implement that is for you to create your own repos. This is what I did when developing https://github.com/BobLd/PdfPig.Rendering.Skia (and dependencies). This will give you full control over your code, progress and how you ship the packages.

Where PdfPig can help is by making implementing your projects doable (making classes and constructors plublic, adding new interfaces, etc.)

Other people have started looking into it, see BobLd/RapidOcrNet#13

I would really like to update PdfPig so that people can easily integrate it with new models and data ingestion pipelines - keep me posted

[luisquintanilla]

Thanks @BobLd.

I'll take a look at the current implementation and see where there are opportunities to extend current PdfPig primitives to unlock this scenario.

[luisquintanilla]

@BobLd here's the link to the issue proposing extensibility changes to PdfPIg.

#1287

I added some opportunities in RapidOcr project as well but feel free to ignore those. The TextBlock metadata would unblock me right now.