Multi-agent RAG that reads charts, tables, and figures, not just text.
A multi-agent system that answers questions and extracts structured data from documents where the interesting information is not always in the text layer. Reports, invoices and spec sheets tend to hide their best numbers in charts and tables. Text-only RAG pipelines silently miss those. This project treats figures as first-class citizens: a vision agent opens the actual pixels, reads the chart, and feeds structured findings back into the same grounding and critique loop as every text source.
Built on LangGraph as a real state machine (conditional edges, a genuine correction cycle, checkpointing, human-in-the-loop interrupts), not a linear chain with extra steps.
A supervisor routes each request. The retriever pulls text and figure chunks from Qdrant. If the question depends on visual content, the vision agent reads the referenced figures with a vision-language model and returns structured FigureReading objects. The reasoner drafts an answer with Pydantic-validated output, and a critic grades it against the retrieved sources. An ungrounded draft does not reach the user: the critic sends it back, either to retrieval (sources were missing something, the critique is folded into the next query) or to the reasoner (sources were fine, the draft misused them). The loop is capped, a stubborn failure gets surfaced with low confidence instead of spinning.
Extractions pause at a LangGraph interrupt before finalizing. The draft and the critic's verdict are shown to a human, who approves or edits. Because the graph is checkpointed to SQLite, the pause survives process restarts, which is what makes the pattern usable outside a notebook.
The part I care about most, coming from a computer vision background: the sample quarterly report deliberately states the Q4 revenue figure only inside a bar chart, never in the text. A text-only pipeline answers that question wrong or not at all. Here, retrieval surfaces the figure by its caption, the vision agent reads the bars, and the critic checks the final answer against what the figure actually shows.
Fastest way to see it run: open the Colab notebook above, it goes from a fresh VM to the full eval on a free T4 in about 20 minutes.
Output from a real run (Colab T4, all-local models). The question can only be answered by reading the report text; the system cites the exact source chunk:
$ agentdoc ask "According to the revenue chart, which quarter had the largest growth?"
{
"answer": "Q3",
"citations": [
{
"chunk_id": "quarterly_report-p1-text",
"quote": "Q3 was the strongest quarter in company history at EUR 4.8M,
driven by the rollout of the document automation product line."
}
],
"confidence": "high"
}
grounded=False correction_loops=2
That grounded=False is the critic being deliberately conservative: "largest growth" requires arithmetic over the chart values, so it refuses to certify and attaches its critique instead of hiding it. A make demo target exists to record a GIF of the interactive session (needs vhs and a running Ollama).
Model: llama3.1:8b + qwen2.5vl:7b via ollama, 8 tasks, deterministic string/field matching (see src/agentdoc/eval/). Run on a free Colab T4.
| Configuration | Accuracy | Grounded | Retrieval hit | Avg. correction loops |
|---|---|---|---|---|
| Full graph (with critic) | 88% | 0% | 100% | 2.0 |
| Critic disabled | 88% | n/a | 100% | n/a |
Scoring is deterministic (exact string and field matching, see src/agentdoc/eval/metrics.py), so the numbers are stricter than LLM-judged ones and fully reproducible. The dataset includes a trap question whose answer is not in the documents at all; credit is only given for saying so. The Colab notebook in notebooks/ reproduces the whole table on a free T4 GPU. Rerun with make eval, make eval-no-critic, make update-readme.
Two honest observations from this run. First, the 8B critic never certified a draft as grounded: it treats any claim requiring arithmetic over chart values as unsupported, so every run hit the loop cap and surfaced with the critique attached. Correct behavior, conservative grader. On this small dataset that strictness did not change accuracy, which is exactly the kind of thing an ablation column is for. Second, the one miss (reading the Q4 bar value out of the chart into a numeric answer) is a vision-to-reasoner handoff failure, not retrieval: the figure was retrieved every time. Both are on the improvement list.
Requirements: Python 3.11+, Docker, Ollama.
git clone https://github.com/mtaha-ai/agentic-document-intelligence.git
cd agentic-document-intelligence
pip install -e ".[dev]"
ollama pull llama3.1:8b
ollama pull qwen2.5vl:7b
docker compose up -d # Qdrant
python scripts/make_samples.py
agentdoc ingest examples/sample_docs
agentdoc ask "What revenue value does the bar chart show for Q4 2025?"
agentdoc extract "extract this invoice" --schema invoiceThe Python package and CLI are named agentdoc (package names cannot contain hyphens). The extract command runs until the human review interrupt, prints the draft plus the critic's verdict, and waits for your approval before finalizing.
No Docker? Set vector_store.backend: chroma in configs/default.yaml.
The whole backend swaps with one config field. Everything else, including the vision agent and structured output, stays identical:
llm:
provider: openai # ollama | openai | anthropic | azure | bedrock | vertex
model: gpt-4o-mini
vision_model: gpt-4o-mini| Provider | Text example | Vision example | Install extra |
|---|---|---|---|
| Ollama (default) | llama3.1:8b |
qwen2.5vl:7b |
none |
| OpenAI | gpt-4o-mini |
gpt-4o-mini |
[openai] |
| Anthropic | claude-sonnet-5 |
claude-sonnet-5 |
[anthropic] |
| Azure OpenAI | your deployment | your deployment | [openai] |
| AWS Bedrock | eu.anthropic.claude-... |
same | [aws] |
| Vertex AI | gemini-2.0-flash |
same | [vertex] |
Development and evaluation ran entirely on the free local path. If you use a paid provider, set a spending cap in the provider's console first; the full eval is a few hundred requests, which is well under one euro on gpt-4o-mini but caps are free insurance.
Set LANGSMITH_TRACING=true and an API key in .env (free tier) and every run is traced end to end: routing decision, retrieved chunks, each vision call with its image, critic verdicts and loop iterations. Debugging a multi-agent graph without traces is guesswork, with them it is reading.
Figures are indexed by caption, read on demand. Embedding image content directly (CLIP-style) sounds nicer but makes retrieval quality harder to reason about, and captions plus page context already get the right figure into the top results on realistic documents. The expensive vision call happens only for figures that retrieval actually surfaced.
The critic distinguishes two failure modes. "Sources are missing information" and "the draft misstates fine sources" need different fixes (new retrieval vs. a rewrite). Collapsing them into one retry edge wastes loops on the wrong fix, so the conditional edge is three-way.
Deterministic eval scoring. LLM-as-judge would be friendlier to the numbers and much easier to game by accident. Exact matching is harsh but nobody has to trust a judge prompt.
One process, real interrupts. Human review uses LangGraph's checkpointed interrupt(), not an input() call bolted into a node. The same graph works unattended (eval sets require_human_review: false) and interactive, and a paused extraction can be resumed by a different process later.
Table extraction currently trusts the text layer; for scanned documents it should fall back to the vision agent on a page render. Retrieval is dense-only, and BM25 hybrid would help on exact identifiers like invoice numbers. The figure caption heuristic is a regex, a layout model would do better on messy documents. And the eval set is small; growing it is mechanical work that pays off linearly.
src/agentdoc/
graph.py LangGraph state machine (routing, critic loop, interrupt)
state.py shared graph state
agents/ supervisor, retriever, vision, table, reasoner, critic
providers/ one factory, six backends
retrieval/ BGE embeddings, Qdrant/Chroma stores
ingestion/ PDF parsing with figure extraction, chunking
schemas/ Pydantic contracts (also the structured-output specs)
eval/ dataset, metrics, runner
configs/ YAML config, one file per environment
examples/ sample docs (synthetic) and a walkthrough
notebooks/ Colab notebook reproducing the eval for free
scripts/ sample generator, README updater, demo recorder
tests/ unit tests for the deterministic parts
MIT