Integrating Docling into a Production RAG Pipeline: Table Handling, Dual Representations, and VLM Fallbacks

[Souheil007]

Hi Docling team and community,

First of all, huge congratulations on Docling!

We recently evaluated several document parsing engines (PyPDF, pdfplumber, Unstructured, Marker, and commercial OCR APIs) for enterprise RAG. Docling's layout analysis and table extraction capabilities stood out in our evaluation for preserving document reading order and structural hierarchy without blindly flattening everything into text.

We wanted to share how we integrated Docling into an open-source RAG pipeline built from first principles, along with a few architectural patterns, benchmark results, and questions for the community.

---

1. How We Use Docling in Our Architecture

Rather than treating Docling's output as flat Markdown strings, we use DoclingDocument.iterate_items() to build a typed abstract syntax tree (DocumentTree).

• Section Headers (SectionHeaderItem): Preserve heading levels (H1, H2, H3) and build an explicit hierarchical breadcrumb trail (Document > Chapter > Sub-section), attached as metadata to every child block.

• Tables (TableItem): Pass tables into a dedicated TableParser that preserves captions, handles multi-row merged headers through forward-fill, and maintains a direct pointer (preceding_prose_id) linking each table to the introductory paragraph that contextualizes it.

• Prose and Code (TextItem, CodeItem): Group content under parent section nodes, keeping reading order clean across multi-column pages.

This gives us a structured representation that downstream chunking and retrieval components can use without losing document context.

---

2. The Dual-Representation Table Pattern for RAG

One major finding from our retrieval experiments: dense embedding models can struggle with raw Markdown pipe tables (| Col1 | Col2 |), especially when column relationships and row context matter.

To address this, we generate two distinct representations from each Docling TableItem.

A. For Embeddings: ChromaDB + BM25

We linearize table rows into key-value sentences:

Section: Financials > Table: Revenue > Row 1:
Region: Americas; 2010: $500M; Margin: 22%

This gives the embedding representation explicit semantic anchors across columns, while also making the text easier to search lexically.

B. For LLM Generation

We pass the original structured table, serialized as Markdown, into the prompt context so the generator can reason about relationships between rows and columns.

This dual-serialization strategy yielded 100% accuracy on our 10-question evaluation suite, with a 90% Top-5 retrieval hit rate.

---

3. Edge-Case Handling: The 95/5 VLM Fallback Pattern

While testing on complex PDF fixtures (sample-tables.pdf), we encountered an interesting edge case in Table 14, a four-column survey containing questions and Yes/No responses.

The extracted columns were horizontally shifted: questions appeared in the last column, while boolean answers appeared in the first.

Since Docling reported the table conversion as SUCCESS without providing the per-table geometric confidence score we needed, we built an automated TableSanityChecker.

It checks for:

• Column alignment skew: For example, short boolean tokens in the first column and long descriptive text in the last column.

• Jagged row lengths: Detecting inconsistent numbers of cells across rows.

• High empty-cell ratios: Identifying potentially degraded table structures.

• Structural health score: Producing a score between 0.0 and 1.0.

The 95/5 Pattern

Our intended architecture is to process approximately 95% of tables locally through Docling and escalate only the roughly 5% that fail the automated sanity checks.

For degraded tables, we route only the relevant page to a lightweight Vision-Language Model, Mistral's pixtral-12b-2409, to reconstruct the table in Markdown.

The goal is to avoid brittle, heuristic-based column rotation and instead use visual information to recover the original structure.

This selective fallback pattern helps keep the normal ingestion path efficient while reserving VLM processing for difficult cases.

---

4. Benchmark and Latency Results

We evaluated our Docling-based pipeline using a 10-question evaluation suite.

Metric | Result -- | -- Answer accuracy | 100.0% (10/10) Top-5 retrieval hit rate | 90.0% (9/10) Median end-to-end latency (P50) | 1,821 ms Full PDF ingestion with RapidOCR | ~60 seconds

The single retrieval non-hit involved an intentional negative hallucination trap that the system correctly refused to answer.

The end-to-end latency includes hybrid search, cross-encoder reranking, and Mistral-7B generation.

These results are specific to our evaluation dataset, but they gave us a useful baseline for investigating document parsing and table-aware retrieval.

---

5. Feedback and Questions for the Docling Maintainers

I'd love to hear the team's thoughts on two questions:

1. Table layout confidence: Are there plans to expose native table layout confidence scores or bounding-box alignment metrics from TableFormer? Such signals could help downstream systems identify structurally degraded tables without relying entirely on custom heuristics.

2. RAG-oriented serialization: Has the team considered providing official helper utilities for dual-representation table serialization, such as key-value linearization for vector stores alongside structured table output for LLM context?

I'd also be interested in hearing how others handle table validation and recovery when the parser reports successful conversion but the resulting structure is semantically incorrect.

---

6. Open-Source Implementation

The complete implementation is open source under the MIT license and includes Docker Compose configuration, latency benchmarks, and architectural documentation.

• Repository: https://github.com/Souheil007/Rag_Full_Version

• Ingestion Architecture Deep Dive: https://github.com/Souheil007/Rag_Full_Version/blob/main/docs/CHUNKING_AND_INGESTION_ARCHITECTURE.md

Thanks again to the Docling team for building and maintaining such a useful tool for document processing!

I'd love to hear your feedback, especially on table confidence signals, structural validation, and RAG-oriented table serialization.

[IlyaKhalafi]

SUCCESS does not establish that a table's structure is semantically correct. The confidence model includes page-level table_score, defaulting to NaN. That is not a calibrated per-table guarantee that columns are correctly associated. Treat unavailable scores as unknown. Aggregate grades can exclude a missing table score.

For dual representation, the existing TableItem exports provide export_to_dataframe(doc=...) and export_to_markdown(doc=...). Build searchable row descriptions from the structured cells while retaining stable table/row references to the original representation. Use cell spans and header flags to expand merged headers. Where TableCell.bbox is present, it also supports geometric alignment checks. Unconditional forward-fill can turn a genuinely empty cell into an invented value. I checked a synthetic merged-header table with docling-core 2.101.1 from PyPI and confirmed that an explicit empty data cell remains empty in both exports. This did not test PDF extraction or your Table 14 fixture.

For fallback selection, I'd evaluate the sanity checker against manually reviewed tables and measure the incorrect tables it misses, alongside the escalation rate. A rectangular grid can still have systematically shifted columns. Keep the original and VLM reconstruction with source-page references, and validate the reconstruction too. The 5% escalation target is a cost target, not evidence of detection quality.

One evaluation adjustment would also help. Since your negative question intentionally has no supporting passage, report retrieval hit rate on answerable questions separately from correct abstention on negative cases. Counting that case as a retrieval miss mixes two different outcomes.

[Souheil007]

1. On table_score: Yeah, Docling saying SUCCESS just means it didn't crash, not that the numbers or columns are right. That’s actually why i stopped relying on its confidence score and started doing our own sanity checks (cross-checking the table numbers against the raw PDF text layer and doing basic column math on total rows).

2. Forward-fill breaking empty cells: Really good point about turning genuine blanks into fake numbers. Right now i only propagate multi-tier headers, not data rows, but using col_span/row_span and bounding box alignment is definitely the right way to do it properly without inventing values.

3. Validating the VLM output: Sure , i'm treating right now vlm as a fix , and treating it as 100% accurate , in fact mistral ocr claim to have most accurate models in ocring , that's why i assumed that but true , a validation is needed for a test set.

4. Cost budget vs detection accuracy: yeah true , the 5% is a cost target , i should be creating a bigger test set to test the whole workflow and evaluate it , my first test set was just a start

5. Negative queries skewing hit rate: I'll update the benchmark to isolate retrieval hit rate strictly to answerable queries, and report a separate kpi for out-of-scope questions.

Thanks a lot for the feedback , i'm curious , have you tried the repo ? did you use it for a use case or your own data ? what are your impressions if you did

[IlyaKhalafi]

In my experience, Docling extracts tables better than other tools I've tried, partly thanks to the edge cases contributors have supported. It still struggles with complex document structures, though. I mostly use VLMs directly, outside Docling, for document understanding now. I haven't used Docling in a while, so my experience may be out of date.