Grounded extraction citations and field confidence

[mvanhorn]

feat: grounded extraction citations and field confidence

Motivation

DocumentExtractor (shipped in PR #2138) returns per-page JSON from a VLM. ExtractedPageData is page_no, extracted_data, raw_text, errors only (docling/datamodel/extraction.py:14-26). Conversion already stores layout provenance on every DocItem (ProvenanceItem: page_no, bbox, charspan) and a document-level ConfidenceReport (layout/ocr/parse; table_score unimplemented). Extraction does not join either.

LlamaExtract exposes cite_sources and confidence_scores on extract: per-field citations (page/text, bbox except Turbo) plus parsing_confidence / extraction_confidence / confidence. RAG threads on r/LangChain ask how to keep provenance without blowing up cost.

This plan adds the same two flags on Docling extract, filled by convert-then-ground: match extracted scalars to DocItem text and copy ProvenanceItem. Do not wait on the VLM to emit citations. Do not wait on PR #4201: that PR is engines, formats, and payload channels, not field citations.

This feature was proposed from code analysis without any hands-on usage.

Research Summary

General Project Dogfooding (Phase 0.5 Step 1e / Phase 1b.5)

• Installed and ran project: NO
• Score: 0/10
• Screenshots: none
• Key observations: bulk skip. Clone inspected at 890dd42.

Feature-Specific Validation (Phase 2d)

• Feature validated: field citations on ExtractedPageData
• Tests run: static read of extraction models, VLM pipeline json.loads, DocumentExtractor.extract signature, LlamaExtract docs
• Key finding: ExtractionVlmPipeline._extract_data (extraction_vlm_pipeline.py:89-98) parses VLM text and constructs ExtractedPageData with no grounding. extract() (document_extractor.py:129-149) has no cite_sources / confidence_scores. Live VLM tests in tests/test_extraction.py skip in CI (IS_CI). Grounding tests must not need a VLM.

Project Context

• Stars: 66923 | Language: Python | License: MIT | Maintainers: @cau-git, @dolfim-ibm, @vagenas, @PeterStaar-IBM
• Philosophy: local MIT IR; extract is beta (docling/.agents/skills/docling/references/extraction.md)

Related Issues & Community Signals

• PHASE_3A CLEAN for this feature. gh search for extraction citation / cite_sources / field grounding returned no issues or PRs.
• PR #4201 OPEN, @cau-git, updated 2026-09-14: API engine, multi-format inputs, payload channels. Adjacent, not this feature. Do not edit that surface.
• PR #2138 MERGED: schema extraction. Precedent that extract PRs land.
• Discussion #2897: provenance source union (audio/video). Not field citations.
• Discussion #216 / PR #4055: forms KV. Blocked as a separate candidate. Do not overlap.
• PROCESS_LEVEL DISCUSSION_PREFERRED.

Social Search Results

Consumed from parent host_handoff. last30days was not run on this host. Path B was not taken.

Source	Evidence	Engagement
r/LangChain provenance	high-recall RAG without losing provenance	6 upvotes, 9 comments
r/LangChain correctness	"is the data actually correct"	4 upvotes, 6 comments
LlamaExtract configuring	cite_sources / confidence_scores flags	competitor docs
LlamaExtract extensions	extract_metadata.field_metadata citations	competitor docs

Competitive Analysis

LlamaExtract: cite_sources: true attaches per-field citations (page/text; bbox on non-Turbo). confidence_scores: true attaches parsing_confidence, extraction_confidence, confidence. extraction_target per_doc / per_page / per_table_row is a different gap (adjacent to #4201; SPECULATIVE here, not this PR).

Docling conversion already has the layout data LlamaExtract has to re-derive from a parse. Extract ignores it.

Maintainer Analysis

• @cau-git: extraction owner, feat(extraction): API engine, multi-format inputs, and payload channels #4201 in flight. This PR must stay off engines/formats/channels. Cite convert-then-ground so it does not collide with VLM payload work.
• @dolfim-ibm / @vagenas: IR / ProvenanceItem. Care that grounding copies existing prov, not a parallel citation type.
• @PeterStaar-IBM: product direction for extract beta.

What Gets Merged vs Rejected

• Merged fast: scoped feat(extraction):, tests that run without GPU, docs/examples, DCO.
• Delayed: feature PRs with no Discussion; overlapping in-flight extraction API (feat(extraction): API engine, multi-format inputs, and payload channels #4201).

Proposed Solution

Add optional grounding on ExtractedPageData:

class FieldCitation(BaseModel):
    field_name: str
    page_no: int
    text: str
    bbox: Optional[Any] = None  # ProvenanceItem.bbox shape; do not fork a new box type
    charspan: Optional[tuple[int, int]] = None
    self_ref: Optional[str] = None  # DocItem.self_ref when matched

class FieldGrounding(BaseModel):
    citations: list[FieldCitation] = []
    confidence: Optional[float] = None  # 0-1

On ExtractedPageData:

• field_citations: Optional[Dict[str, list[FieldCitation]]] = None
• field_confidence: Optional[Dict[str, float]] = None

Flags (default False, LlamaExtract names):

• DocumentExtractor.extract(..., cite_sources: bool = False, confidence_scores: bool = False, converted_document: Optional[DoclingDocument] = None)
• Same kwargs on extract_all.
• Mirror on VlmExtractionPipelineOptions as cite_sources / confidence_scores so an ExtractionFormatOption can pin them. Kwargs override options.

Grounding is a post-step after the VLM pipeline returns pages. extraction_vlm_pipeline.py stays json.loads only. Do not prompt the VLM for citations.

Algorithm in docling/utils/extraction_grounding.py:

1. Require a DoclingDocument. If converted_document is passed, use it. Else if cite_sources or confidence_scores, run DocumentConverter().convert(source).document once per input (same path/stream the extractor already opened). If convert fails, leave citations empty and continue extract success; do not fail the VLM result.
2. Walk extracted_data as a flat dict of scalars (str/int/float/bool). Nested dicts: join keys with .. Lists of scalars: one citation list per key, match each element. Skip dicts/lists of dicts in v1 (no per_table_row).
3. For each scalar, stringify and search DocItem text (reading order). Match order: exact, then casefold+collapsed whitespace. First hit wins. Copy item.prov[0] into FieldCitation (page_no, bbox, charspan) plus self_ref.
4. Confidence: 1.0 exact, 0.7 fuzzy, omit key if no match. If a ConversionResult.confidence page score exists for page_no, optionally min(match_score, page.parse_score) when confidence_scores is on. Do not invent a VLM logit.
5. Restrict candidate items to page_no when ExtractedPageData.page_no is set.

Not in this PR: VLM-emitted citations, per_table_row, whole-document merge (#4201), forms (#4055), word_cells (#3457).

Approach

1. docling/datamodel/extraction.py
   • Add FieldCitation, optional maps on ExtractedPageData. Keep existing fields unchanged so current tests still pass.
2. docling/utils/extraction_grounding.py (new)
   • SPDX MIT header.
   • ground_extracted_page(page: ExtractedPageData, doc: DoclingDocument, *, cite_sources: bool, confidence_scores: bool, page_confidence: float | None = None) -> ExtractedPageData
   • Pure function. Mutate a copy or return a new page model. No gh, no VLM.
3. docling/document_extractor.py

---

Happy to follow maintainer guidance before any PR (CLA noted).

[wittjeff]

Grounding extracted values in the converted document's provenance is a good fit for docling, since conversion already records where every item sits on the page. A few things in the plan would bite in practice, though.

#4201 overlaps more than the plan assumes. It rewrites the three files this touches (datamodel/extraction.py, document_extractor.py, pipeline/extraction_vlm_pipeline.py). It also adds a TEXT channel that extracts from the whole document in one request and returns a single ExtractedPageData with page_no=1. With that in place, step 5 (restrict candidates to page_no) would search only page 1 for text-channel results. The TEXT channel already calls backend.convert() inside the pipeline, so the separate DocumentConverter().convert() in step 1 would convert the same document twice. Building this on top of #4201 once it lands looks simpler than working around it.

Matching values against item text will produce wrong citations.

• If the match is equality against a whole DocItem.text, it will rarely fire: the value 12345 is not the item Invoice No: 12345.
• If it's containment, short values (1, Yes, a date, a total that appears in several places) match the first place they occur in reading order. That citation gets confidence 1.0, and a confidently wrong citation is worse than none.
• With a Pydantic template, values come back typed or normalized: 1234 for 1,234.00, an ISO date for 3 Sept 2026, true for a ticked box. Stringifying those won't find the source text.
• Tables are where most invoice and form values live, and TableItem has no text. Matching would have to walk data.table_cells, which carry their own text and bbox.

The citation would point at the whole item. prov[0].bbox and prov[0].charspan describe the whole item, not the value inside it, and prov[0] is the wrong entry when an item spans two pages. The PDF backend that extraction already opens exposes word cells with bboxes (get_segmented_page().word_cells), without running layout, OCR or table models. Matching against those gives a tighter box at much lower cost than a full conversion, which also matters for the cost concern the proposal starts from. Scanned pages have no text layer either way, so they need OCR or a VLM that returns coordinates.

"Confidence" would mean something else here. 1.0 for an exact match and 0.7 for a fuzzy one is a string-match score, and taking the min with parse_score adds PDF parse quality. Neither says whether the extracted value is correct. Returned as confidence, next to LlamaExtract's flag of the same name, it would read as extraction confidence. Naming it something like grounding_score keeps it honest. A real per-field confidence could come from the model's token log-probabilities: transformers generate can return them (output_scores), and OpenAI-compatible servers usually support logprobs.

A narrower first step could be a pure function over an ExtractionResult plus the page word cells. It would return citations only for unambiguous matches and leave ambiguous values uncited rather than guessing. That can be tested without a VLM, as you noted, and it stays out of #4201's surface.

[mohitghansela]

@mvanhorn @wittjeff I really like the direction of this proposal! Adding native grounding layout metadata directly on ExtractedPageData would be a massive win for RAG tracking accuracy without bloating token processing costs.

However, as @wittjeff noted, we need to ensure this doesn't collide with the incoming TEXT payload channel in #4201. To prevent the double-conversion performance hit and fix the page_no=1 boundary constraint, what if we tweak the approach like this:

1. Decouple Conversion from Grounding: Instead of running an internal DocumentConverter().convert() fallback inside the grounding step, we should strictly require a pre-converted DoclingDocument. If the user calls .extract() without passing it, we can surface a clear warning or return empty citations, ensuring the pipeline remains light and clean.
2. Context-Aware Page Mapping: For single-page chunks or multi-page formats processed via the new TEXT channel, the lookup engine should match against the full DoclingDocument structure globally rather than restricting candidate string arrays strictly to a singular page_no filter.
3. Wait for feat(extraction): API engine, multi-format inputs, and payload channels #4201 to Land: Because feat(extraction): API engine, multi-format inputs, and payload channels #4201 completely rewrites datamodel/extraction.py and document_extractor.py, it makes the most sense to implement this grounding logic as a fast-follow PR immediately after feat(extraction): API engine, multi-format inputs, and payload channels #4201 merges to prevent massive structural git conflicts.

I'd be glad to help sketch out a clean string-matching lookup utility for docling/utils/extraction_grounding.py that accommodates this revised workflow if the core team is aligned!