RAG / embeddings, bounding boxes, output customization and non-text documents — what is supported (and what is the recommended pattern)?

[mannemlokesh-exf]

Hi, and thanks for LangExtract.

Context: we extract structured fields from long, multi-page PDF reports (fund reports with tables) and need the user to click a value and see exactly where it came from on the page. We parse PDFs to Markdown with Docling and send the text to LangExtract (v1.7.0, OpenAI-compatible endpoint). I read the README, the extract() docstring and the source before asking, so I'm sharing what I found first. Please correct me where I'm wrong.

What I checked in v1.7.0

• lx.extract() accepts str (text, or a URL when fetch_urls=True) or an iterable of Document (text). I found no PDF, DOCX, PPTX, XLSX or image handling anywhere in the package.
• An Extraction has extraction_class, extraction_text, char_interval (start_pos/end_pos), alignment_status, extraction_index, group_index, description and attributes. I found no page, bounding-box or coordinate concept.
• Searching the source for embedding / vector / retrieval / rerank / RAG gives no matches. The runtime dependencies contain no vector store or embedding library, and none of the extract() parameters mentions an embedding model, vector store, top-k or retriever.

Questions

1. RAG / semantic search

• Does LangExtract have any built-in retrieval, embedding or vector-search support? If yes, where is it documented, and how is it configured?
• If yes: does it use an in-memory index, and can we plug in our own vector database (e.g. MongoDB Atlas Vector Search, pgvector) and control where the embeddings are stored?
• If no: is the recommended pattern to retrieve the relevant chunks ourselves and call extract() on just those chunks? Is there a supported hook for this (a custom provider, a Document per chunk), and is retrieval something the project wants to support in the future?
• For long documents, is chunking plus extraction_passes the intended approach, and when would you recommend retrieval instead?

2. Source location (bounding boxes)

• Is there any way to get page numbers or bounding boxes (x0, y0, x1, y1) for an extraction, or is char_interval the only location information by design?
• If we feed Markdown generated from a PDF, is the recommended approach to keep our own table of "character range → page and box" and look up char_interval in it? I found the third-party langextract-docling wrapper does this, but it is a PDF-path-only wrapper pinned to langextract==1.6.0.
• When we call extract() with several Documents (e.g. retrieved chunks), are char_interval offsets always relative to that document's own text, and is that guaranteed to stay stable across versions?

3. Customizing the output

• Can we add our own fields to each extraction (e.g. a verbatim evidence quote, a page hint, a period/unit, a model-reported flag) in a supported way? Is attributes the intended mechanism, and does output_schema allow extra per-extraction keys?
• Can we control the exact messages sent to the model (e.g. a real system message), or is the prompt wrapper fixed?
• For OpenAI-compatible providers I see max_output_tokens and top_p being forwarded. Is there a supported way to set a request timeout and retries? The client appears to be created internally with only api_key / base_url / organization.

4. Document types

• Is LangExtract intentionally text-only (strings, URLs of text, Document)? Are PDF, Word, PowerPoint, Excel and scanned images meant to be converted to text before the call? Is there a recommended converter or pipeline, and are there plans for native document support?

Behaviours I ran into (could you confirm whether they are expected?)

a) Crash when every value is empty. If the model returns only empty values (e.g. {"extractions": [{"Auditor": ""}]}, which is what a careful model returns when the field is not in the text it was given), alignment raises and the whole call fails:

File ".../langextract/resolver.py", line ~904, in align_extractions
    self._set_seqs(source_tokens, extraction_tokens)
File ".../langextract/resolver.py", line 568, in _set_seqs
    raise ValueError("Source tokens and extraction tokens cannot be empty.")

Would you prefer empty values to be dropped (treated as "not found")? We currently filter them out ourselves before alignment.

b) MATCH_FUZZY for an exact substring. For values like 92.80 taken from Sales Tax (8%): 92.80, the returned char_interval points at exactly that text, yet alignment_status is MATCH_FUZZY. Is that expected for tokens such as numbers with punctuation?

c) Short values and repeated text. A one-digit value (e.g. 6) appears to align to the first occurrence in the text, which may not be the occurrence the model read. Is there a recommended way to disambiguate (e.g. passing surrounding context, a quote, or an anchor)?

d) Tables. When max_char_buffer splits a wide Markdown table, the header row ends up in a different chunk from its figures. Any guidance on chunking tables?

[donmangudata-ops]

For question 2, char_interval is the only location LangExtract returns, so keeping your own range table is the way to go. Build the text you send and the table in one pass, one segment per source item, then look the interval up with bisect.

import bisect

def locate(segs, start, end):  # segs: sorted (start, end, page, bbox)
    i = max(bisect.bisect_right([s[0] for s in segs], start) - 1, 0)
    return [(s[2], s[3]) for s in segs[i:] if s[0] < end and s[1] > start]

With Docling, each item's prov has page_no and bbox, so segment from there. The box is the whole text block, not just the value. I tested the lookup with langextract 1.7.0 Resolver.align and PyMuPDF text blocks (not Docling): Sales Tax (8%): 92.80 came back as page 2 with a box over that line.

Disclosure: I build commercial data tools in this area (details in my GitHub profile).

[IlyaKhalafi]

For the parts beyond the coordinate mapping already discussed, I checked the 1.7.0 source and ran offline checks against that release.

Retrieval and document formats

LangExtract does not build an embedding index or retrieve documents for you. Its input is text, including text inside Document objects. Convert PDFs, Office files and images upstream, then either extract over the document or retrieve relevant text yourself. A custom model provider changes inference, not retrieval.

For retrieval, pass each chunk as a Document with a unique document_id, and retain its original document ID and source start offset outside LangExtract. Returned intervals are relative to that Document's text. Internal chunking adds its own chunk offset back before returning results, but it cannot recover an offset into a larger document you never supplied. I verified independent offsets for two documents, including a Unicode prefix, and an extraction after several internal chunks. Treat them as Python string positions, not UTF-8 byte offsets. Input contract and offset handling.

Chunking plus additional extraction passes processes the supplied text repeatedly to improve recall. It is not semantic retrieval. Retrieval can reduce the text sent to the model, but it can also omit the passage containing an answer. The built-in chunker is sentence/token based and has no Markdown-table model. For wide tables, create row groups upstream that retain the column headers and identifiers. If you repeat headers or otherwise rewrite chunks, keep a mapping for that rewritten text rather than adding a single offset to every result.

Output fields and model requests

Use Extraction.attributes for units, evidence quotes or model-reported flags. In the raw output, the associated object is named <class>_attributes, for example Amount_attributes. An output_schema constrains that raw envelope; it does not add arbitrary fields to the returned Extraction class. Adding page alongside Amount creates a separate extraction class named page, rather than metadata on the amount. I verified that behavior too. Put those extras inside Amount_attributes. The supported helpers are extraction_item_schema(..., attributes=...) and extractions_schema(...). An evidence quote in attributes is model output, not an independently aligned or verified citation.

The OpenAI provider builds its own formatting system message and puts the rendered extraction prompt in a user message. In 1.7.0, passing system_message, timeout or max_retries through language_model_params does not configure those things. I checked the outgoing parameters with a fake SDK client. The constructor only forwards the key, base URL and organization when creating the SDK client, and infer selects a fixed set of generation parameters. The SDK's own defaults still apply. For explicit transport settings or message control, use a custom provider and pass its model through model= rather than relying on silently ignored kwargs.

The alignment cases

• I reproduced your response containing one empty-string extraction. Returning {"extractions": []} gives no extractions without that failure. A mixed empty/nonempty response and two empty entries did not raise the same error locally, so this is more specific than every missing value causing a crash. Alignment runs per chunk. Your filtering before alignment is useful; asking the model to omit absent fields alone is not a reliable guard.
• The minimal 92.80 extraction from Sales Tax (8%): 92.80 returned MATCH_EXACT locally, so I cannot confirm that example as an unavoidable punctuation limitation. The complete raw model output, tokenizer and resolver_params would help reproduce your MATCH_FUZZY result. The status describes the aligner's matching path, not model confidence.
• A single extracted 6 from Period 6. Result 6. aligned to the first occurrence in my check. The aligner cannot infer which financial row the model intended from the numeral alone. Extract a longer, distinguishing verbatim span or supply a smaller row-specific input. With the longer-span approach, extraction_text is that span, so parse the numeric value from it or carry the value in attributes. Adding a row label only to attributes does not make it an alignment anchor.

These checks used synthetic model responses and a fake SDK client, with no live provider, Docling or PDF processing. I cannot promise future retrieval/native-document support or offset compatibility across releases on behalf of the maintainers. Pinning the version and retaining small alignment regression cases would make this pipeline easier to validate when upgrading.