FoundryRAG

Structure-Aware Hybrid RAG for Technical Document Intelligence

An offline-first, explainable document question-answering system built with Microsoft Foundry Local, hierarchical document parsing, hybrid retrieval, cross-encoder reranking, bounded follow-up retrieval, and source-level telemetry.

---

Table of Contents

• Overview
• Why This Project Exists
• Measured System Scale
• What Makes FoundryRAG Different
• Engineering Story
• Design Philosophy
• System Architecture
• Document Ingestion Pipeline
• RAG Query Flow
• Explainability and Observability
• Entity Graph
• Evaluation Results
• Measured Latency
• Local-First, Cloud-Ready
• Technology Stack
• Project Structure
• Quick Start
• Current Limitations
• Project Resources
• Author
• License

---

Overview

FoundryRAG is an offline-first document intelligence system designed for technical, industrial, and engineering documents.

It answers questions using content from Markdown, PDF, Word, Excel, and CSV files while running locally through Microsoft Foundry Local. The project does not treat a document as a bag of unrelated text fragments. Every parser produces a hierarchical KnowledgeNode tree containing headings, paragraphs, tables, figures, warnings, notes, and code blocks.

Retrieval combines:

• LLM-based query rewriting
• BM25 sparse retrieval
• Dense vector similarity
• Reciprocal Rank Fusion
• Cross-encoder reranking
• Retrieval grading
• Full parent-section reconstruction
• One bounded follow-up retrieval hop
• Grounded answer generation
• Source-level explainability
• Stage-level latency telemetry

The same retrieval logic is designed to remain stable whether the system runs locally or later connects to Azure AI Search and Azure OpenAI.

---

Project Experience

Core Project Links

Retrieval and Intelligence

Reliability and Explainability

---

Why This Project Exists

Most RAG demonstrations assume that:

• internet access is always available,
• documents can be reduced to fixed-size chunks,
• dense similarity alone is enough,
• the highest-scoring chunks are automatically trustworthy,
• and model output does not need to be inspected.

Those assumptions are weak for industrial and technical environments.

A field engineer, regulated facility, air-gapped network, or maintenance team may need a system that runs locally, preserves document structure, reveals its evidence, and fails visibly instead of fabricating an answer.

FoundryRAG was built around that requirement.

The goal was not to create another chat interface over embeddings. The goal was to build a retrieval pipeline whose internal decisions can be measured, inspected, and improved.

---

Measured System Scale

The following values represent the current indexed corpus and project history.

Metric	Value
Documents ingested	7
Knowledge tree nodes	22,174
Retrieval chunks indexed	6,530
Extracted entities	20
Entity co-occurrence edges	58
Largest tested chunk set	5,772 chunks
Development commits	90+ modular commits

Entity graph extraction is intentionally opt-in because it performs an LLM call per document section. Processing cost therefore scales with structural section count, not only file size.

---

What Makes FoundryRAG Different

Typical Tutorial RAG	FoundryRAG
Fixed-size flat chunks	Hierarchical KnowledgeNode tree
Dense retrieval only	BM25 + dense retrieval + RRF
Raw Top-K results	Cross-encoder reranking
Always trusts retrieved chunks	Relevance grading and deduplication
Matched excerpt in isolation	Full parent-section reconstruction
Single retrieval pass	One bounded follow-up retrieval hop
Tables treated as plain text	Tables and figures preserved atomically
Re-embeds everything	Hash-based incremental ingestion
Hidden retrieval process	Explainability Matrix and telemetry
Silent model degeneration	Repetition-loop detection
Cloud-dependent generation	Local Microsoft Foundry Local execution
No relationship layer	Entity co-occurrence graph

---

Engineering Story

Feature lists show what exists. This section explains why each major feature exists.

Problem	Why It Mattered	Engineering Decision	Result
Flat chunking removed document structure	A table row is meaningless without its header and section	Every parser produces a hierarchical KnowledgeNode tree	Tables, figures, warnings, and sections retain context
Retrieved excerpts were incomplete	A sentence such as “apply after 600 hours” may omit the referenced material or machine	Reconstruct sibling nodes through the parent section	The model receives the complete local section
BM25 and dense indexes were rebuilt repeatedly	Query cost grows quickly with corpus size	Cache the hybrid index and invalidate only when corpus size changes	Retrieval remains in the millisecond range
Small local models entered repetition loops	The model narrated chunks instead of answering	Add repetition-loop detection and move to Phi-4-mini	Invalid output is stopped and reported honestly
Hardcoded synonyms did not generalize	Topic-specific expansion worked only for known words	Replace static expansion with LLM-based rewriting	Query expansion works across arbitrary document domains
One-shot retrieval missed multi-part questions	Some answers require evidence from separate sections	Add one score-triggered follow-up retrieval round	Adaptive retrieval without an open-ended agent loop
Local and cloud embedding dimensions can differ	Mixing 384-dimensional and 1536-dimensional vectors can break similarity search	Validate embedding-model consistency during index construction	Clear configuration error instead of a deep runtime crash
Query rewriting used the full generation budget	A short two-line rewrite took approximately 27 seconds	Add task-specific token limits	Query expansion dropped to approximately 4 seconds
Entity extraction echoed prompt fragments	Large document processing produced repetition instead of JSON	Redesign the prompt, detect loops, and parse tolerant JSON	Entity extraction became reliable on verified documents
Grader threshold was effectively inactive	Irrelevant chunks passed through silently	Correct the rerank threshold and retain keyword fallback	The relevance filter now performs real filtering

---

Design Philosophy

Node tree, not chunk-first

Chunking is a view over the document tree, not the primary representation.

This makes the following possible without format-specific retrieval logic:

• parent-section reconstruction,
• atomic table and figure handling,
• section-aware metadata,
• entity extraction by section,
• and document structure inspection.

Bounded, not open-ended

The follow-up retrieval hop is capped at one additional round.

It is triggered by a cheap score threshold rather than asking another LLM whether the answer is sufficient. This provides adaptivity without uncontrolled agent behavior, tool loops, or unpredictable latency.

Fail honestly, never fabricate

There are no hardcoded fallback answers in the pipeline.

When retrieval is insufficient or generation degenerates, the system returns an explicit failure state and keeps the retrieved references visible.

Measure before optimizing

The telemetry layer records stage-level latency.

This is how the query-rewriting bottleneck was identified and reduced from approximately 27 seconds to approximately 4 seconds.

Local-first, cloud-ready

The stable system runs locally.

Retrieval and generation backends can be replaced later, but parsing, tree construction, reranking, grading, parent reconstruction, and bounded retrieval remain part of the same application logic.

---

System Architecture

flowchart TB
    U["User"] --> UI["FoundryRAG Web Interface"]

    subgraph API["FastAPI Application"]
        CHAT["/chat"]
        UPLOAD["/upload"]
        DOCS["/documents"]
        GRAPHAPI["/graph"]
        SOURCE["/source/{file}"]
        HEALTH["/health"]
    end

    UI --> CHAT
    UI --> UPLOAD
    UI --> DOCS
    UI --> GRAPHAPI
    UI --> SOURCE

    subgraph INGESTION["Structure-Aware Ingestion"]
        PARSERS["Markdown / PDF / DOCX / XLSX / CSV Parsers"]
        TREE["KnowledgeNode Tree"]
        CHUNK["Heading-Aware Chunking"]
        HASH["Hash-Based Deduplication"]
        ENTITY["Optional Entity Extraction"]
    end

    UPLOAD --> PARSERS
    PARSERS --> TREE
    TREE --> CHUNK
    CHUNK --> HASH
    TREE --> ENTITY

    subgraph STORAGE["Local Storage"]
        SQLITE[("SQLite")]
        NODES["knowledge_nodes"]
        CHUNKS["document_chunks"]
        DOCUMENTS["documents"]
        ENTITIES["entities / entity_edges"]
        LOGS["query_log"]
    end

    HASH --> SQLITE
    ENTITY --> SQLITE
    SQLITE --> NODES
    SQLITE --> CHUNKS
    SQLITE --> DOCUMENTS
    SQLITE --> ENTITIES
    SQLITE --> LOGS

    subgraph RETRIEVAL["Retrieval Pipeline"]
        REWRITE["LLM Query Rewriting"]
        BM25["BM25 Sparse Retrieval"]
        DENSE["Dense Vector Retrieval"]
        RRF["Reciprocal Rank Fusion"]
        RERANK["Cross-Encoder Reranking"]
        GRADE["Relevance Grading"]
        COMPRESS["Context Compression"]
        PARENT["Parent-Section Reconstruction"]
        FOLLOWUP["Bounded Follow-Up Retrieval"]
    end

    CHAT --> REWRITE
    REWRITE --> BM25
    REWRITE --> DENSE
    CHUNKS --> BM25
    CHUNKS --> DENSE
    BM25 --> RRF
    DENSE --> RRF
    RRF --> RERANK
    RERANK --> GRADE
    GRADE --> COMPRESS
    COMPRESS --> PARENT
    NODES --> PARENT
    PARENT --> FOLLOWUP

    subgraph LOCAL["Local Execution"]
        EMB["SentenceTransformer Embeddings"]
        FL["Microsoft Foundry Local"]
        PHI["Phi-4-mini"]
    end

    EMB --> DENSE
    FOLLOWUP --> FL
    FL --> PHI

    subgraph OUTPUT["Grounded Output"]
        ANSWER["Final Answer"]
        REFERENCES["Source References"]
        EXPLAIN["Explainability Matrix"]
        TELEMETRY["Stage-Level Telemetry"]
    end

    PHI --> ANSWER
    RERANK --> EXPLAIN
    PARENT --> REFERENCES
    REWRITE --> TELEMETRY
    RRF --> TELEMETRY
    RERANK --> TELEMETRY
    ANSWER --> TELEMETRY

    ANSWER --> UI
    REFERENCES --> UI
    EXPLAIN --> UI
    TELEMETRY --> UI

Loading

---

Document Ingestion Pipeline

flowchart LR
    FILE["Input Document"] --> DETECT{"File Type"}

    DETECT -->|Markdown| MD["Markdown Parser"]
    DETECT -->|PDF| PDF["PDF Parser"]
    DETECT -->|DOCX| DOCX["DOCX Parser"]
    DETECT -->|XLSX / CSV| XLSX["Spreadsheet Parser"]

    MD --> TREE["Unified KnowledgeNode Tree"]
    PDF --> TREE
    DOCX --> TREE
    XLSX --> TREE

    TREE --> TYPES["Typed Nodes<br/>heading / paragraph / table / figure / warning / note / code"]
    TYPES --> ATOMIC["Atomic Content Protection"]
    ATOMIC --> CHUNK["Heading-Boundary Chunking"]
    CHUNK --> HASH{"Document Hash Changed?"}

    HASH -->|No| SKIP["Skip Unchanged Document"]
    HASH -->|Yes| REINDEX["Delete Previous Index Entries"]
    REINDEX --> EMBED["Generate Embeddings"]
    EMBED --> STORE["Store Nodes, Chunks, Metadata"]

    TYPES --> ENABLE{"ENABLE_ENTITY_GRAPH?"}
    ENABLE -->|No| DONE["Ingestion Complete"]
    ENABLE -->|Yes| EXTRACT["Extract Entities Per Section"]
    EXTRACT --> EDGES["Build Co-Occurrence Edges"]
    EDGES --> STORE
    STORE --> DONE

Loading

Parser contract

Every parser returns the same conceptual structure:

KnowledgeNode
├── node_id
├── document_id
├── parent_id
├── node_type
├── heading_path
├── content
├── page_number
├── metadata
└── children

This shared structure prevents downstream retrieval logic from depending on file format.

---

RAG Query Flow

flowchart TB
    Q(["User Query"]) --> RW["1. Query Rewriting<br/>Original + up to 2 rewritten tracks"]

    RW --> EMB["2. Query Embedding"]

    EMB --> MULTI["3. Multi-Track Retrieval"]

    subgraph HYBRID["Hybrid Retrieval"]
        BM["BM25 Sparse Score"]
        DS["Dense Cosine Similarity"]
        FUSION["Reciprocal Rank Fusion"]
        DEDUP["Deduplicate by Chunk ID"]
        BM --> FUSION
        DS --> FUSION
        FUSION --> DEDUP
    end

    MULTI --> BM
    MULTI --> DS
    DEDUP --> RR["4. Cross-Encoder Reranking"]

    RR --> GR["5. Retrieval Grading<br/>Jaccard Dedup + Score / Keyword Filter"]

    GR --> CO["6. Context Compression<br/>Atomic nodes pass through"]
    CO --> PR["7. Full Parent-Section Reconstruction"]

    PR --> BUDGET{"8. Adaptive Context Budget"}

    BUDGET -->|"Top score > 1.2"| TWO["Use 2 Context Blocks"]
    BUDGET -->|"Top score > 0.65"| FOUR["Use 4 Context Blocks"]
    BUDGET -->|"Otherwise"| FIVE["Use 5 Context Blocks"]

    TWO --> CHECK{"Confidence Low?"}
    FOUR --> CHECK
    FIVE --> CHECK

    CHECK -->|No| PACK["9. Package Context"]
    CHECK -->|Yes| SUB["Generate Sub-Queries"]
    SUB --> HOP["One Additional Retrieval Hop"]
    HOP --> MERGE["Merge and Rerank Evidence"]
    MERGE --> PACK

    PACK --> GEN["10. Grounded Generation<br/>Foundry Local + Phi-4-mini"]
    GEN --> LOOP{"Repetition Loop?"}

    LOOP -->|No| OK["Final Answer + References + Telemetry"]
    LOOP -->|Yes| FAIL["Explicit Failure + Retrieved References"]

Loading

Hybrid retrieval

BM25 and dense retrieval solve different problems:

Signal	Strength
BM25	Exact terminology, model numbers, error codes, product names
Dense similarity	Paraphrases, semantic similarity, conceptual matches
Reciprocal Rank Fusion	Combines both rankings without requiring score normalization
Cross-encoder	Evaluates query-document relevance jointly

Bounded adaptive retrieval

The system can perform one additional retrieval hop when the first result set is weak.

It never enters an open-ended reasoning loop.

Maximum retrieval depth = initial retrieval + one follow-up hop

Full parent-section reconstruction

A retrieved node is expanded with its sibling nodes from the same parent section.

This prevents incomplete evidence such as:

"Apply after 600 operating hours."

from reaching the model without the preceding context that explains what should be applied and to which component.

Honest failure handling

If generation fails or enters a repetition pattern:

The system does not fabricate a replacement answer.

Instead, it returns:

• an explicit failure message,
• retrieved source references,
• and the available telemetry.

---

Explainability and Observability

The interface exposes the retrieval process instead of presenting only the final answer.

Explainability Matrix fields

Field	Purpose
Query track	Shows which rewritten query found the evidence
Source file	Identifies the original document
Page number	Enables direct verification
Node type	Distinguishes paragraph, table, warning, figure, and other content
Heading path	Shows the document hierarchy
BM25 score	Indicates lexical relevance
Dense score	Indicates semantic similarity
Fusion rank	Shows the RRF result
Rerank score	Shows cross-encoder relevance
Selected context	Confirms whether the item reached generation
Retrieval hop	Distinguishes initial and follow-up retrieval

Pipeline telemetry fields

Stage	Recorded Information
Query expansion	Generated query tracks and latency
Embedding	Embedding model and latency
Hybrid retrieval	Candidate count and retrieval latency
Reranking	Candidate scores and reranking latency
Grading	Removed and retained candidates
Compression	Context reduction and parent expansion
Follow-up retrieval	Trigger state and hop count
Generation	Model, duration, and failure state

Interface preview

---

Entity Graph

FoundryRAG can extract technical entities from each document section and build co-occurrence edges.

flowchart LR
    SECTION["Document Section"] --> LLM["Local Entity Extraction"]
    LLM --> TERMS["Normalized Technical Terms"]
    TERMS --> UNIQUE["Entity Registry"]
    TERMS --> PAIRS["All Unique Pairs in Section"]
    PAIRS --> EDGES["Weighted Co-Occurrence Edges"]
    UNIQUE --> GRAPH["Interactive Entity Graph"]
    EDGES --> GRAPH

Loading

The graph currently supports:

• document exploration,
• concept discovery,
• section-level relationship inspection,
• and explainability.

It is not currently used as an additional retrieval signal.

---

Evaluation Results

The retrieval pipeline was evaluated against a naive dense retrieval baseline using the same labeled dataset and the same Top-5 setting.

Retrieval quality

Pipeline	Precision@5	Recall@5	MRR
Naive dense retrieval	0.857	0.857	0.857
FoundryRAG advanced pipeline	0.893	1.000	0.905

Relative improvement

Metric	Improvement
Precision@5	+4.2%
Recall@5	+16.7%
MRR	+5.6%

Evaluation assets

The evaluation measures retrieval quality. It does not currently include automated answer faithfulness or generation-quality scoring.

---

Measured Latency

Representative local execution on Apple Silicon M4:

Stage	Latency	Share of Pipeline	Notes
Query expansion	~4.0s	High	LLM-based rewriting with max_tokens=120
Query embedding	~83ms	Low	Local SentenceTransformer
Hybrid retrieval	~64ms	Low	Cached BM25 + vectorized cosine similarity
Cross-encoder reranking	~807ms	Medium	Top candidate relevance scoring
Grade + compress	~5ms	Very low	Deduplication, filtering, and section reconstruction
Token generation	~6.5s	High	Local answer synthesis
Approximate total	~11.5s	—	Depends on model, query, and context size

Optimization result

Before	After	Change
~27s query expansion	~4s query expansion	~6.7× faster

The improvement came from applying task-specific generation limits to short structured calls rather than allowing them to use the full answer-generation budget.

---

Local-First, Cloud-Ready

Current stable mode

Architectural extension path

Local mode

Layer	Implementation
Document parsing	Local Python parsers
Knowledge tree	SQLite
Chunk storage	SQLite
Sparse retrieval	Cached BM25
Dense retrieval	SentenceTransformer
Reranking	Local BGE cross-encoder
Generation	Microsoft Foundry Local
Chat model	Phi-4-mini
Entity graph	Local SQLite
Telemetry	Local query log

After the initial model download, no network connection is required.

Cloud-ready design

The architecture contains integration points for:

• Azure Blob Storage
• Azure AI Search
• Azure OpenAI
• Application Insights

The stable release is local-first. Azure retrieval and generation backends have not been validated end-to-end in the current release.

The following components remain local and unchanged by design:

• document parsing,
• KnowledgeNode tree construction,
• entity graph,
• cross-encoder reranking,
• retrieval grading,
• parent-section reconstruction,
• bounded follow-up retrieval.

flowchart TB
    CORE["Shared FoundryRAG Core<br/>Parsing / Tree / Rerank / Grade / Reconstruct / Follow-Up"]

    CORE --> MODE{"Execution Mode"}

    MODE -->|Local| LOCALRET["SQLite + BM25 + Local Dense Retrieval"]
    MODE -->|Cloud Extension| CLOUDRET["Azure AI Search"]

    LOCALRET --> LOCALGEN["Microsoft Foundry Local"]
    CLOUDRET --> CLOUDGEN["Azure OpenAI"]

    LOCALGEN --> OUTPUT["Same Answer / Reference / Telemetry Contract"]
    CLOUDGEN --> OUTPUT

    CORE --> SQLITE[("Local SQLite<br/>Knowledge Tree + Entity Graph")]
    SQLITE --> OUTPUT

Loading

---

Technology Stack

Core Platform

AI and Retrieval

Document Processing

Frontend and Visualization

Stack Summary

Layer	Technology
Backend	Python, FastAPI, Uvicorn
Local inference	Microsoft Foundry Local
Chat model	Phi-4-mini
Embeddings	all-MiniLM-L6-v2
Sparse retrieval	rank-bm25
Fusion	Reciprocal Rank Fusion
Reranking	bge-reranker-base
Storage	SQLite
PDF parsing	pdfplumber
DOCX parsing	python-docx
Spreadsheet parsing	pandas
Frontend	HTML, CSS, JavaScript
Graph visualization	Vis Network
Evaluation	Precision@K, Recall@K, MRR

---

Project layout

├── api/app.py                FastAPI app: routes, serves the dashboard UI
├── src/
│   ├── config.py              Central config, reads .env, MODE switch
│   ├── db.py                  SQLite schema: knowledge_nodes, document_chunks,
│   │                          documents registry, entities/entity_edges;
│   │                          schema migration on startup
│   ├── chunking.py             chunk_nodes() (tree-aware) + chunk_document() (legacy)
│   ├── graph_builder.py        Entity extraction + co-occurrence graph building
│   ├── parsers/                Pluggable document parsers, all tree-aware
│   │   ├── base.py              KnowledgeNode model, NodeType enum, parser interface
│   │   ├── markdown_parser.py
│   │   ├── pdf_parser.py
│   │   ├── docx_parser.py
│   │   └── xlsx_parser.py
│   ├── retrieval/
│   │   ├── hybrid.py             BM25 + dense fusion (RRF), cached index
│   │   ├── reranker.py           Cross-encoder re-ranking
│   │   ├── grader.py             Retrieval relevance grading + Jaccard dedup
│   │   ├── query_rewriter.py     LLM-based query expansion + sub-query decomposition
│   │   └── compression.py        Sentence-window pruning + full parent-section reconstruction
│   ├── llm_client.py           Foundry Local + Azure OpenAI client wrappers
│   ├── rag_pipeline.py         Orchestrates expansion -> retrieval -> [follow-up hop] -> generation
│   ├── azure_search.py         Azure AI Search index + query helpers (planned, not yet implemented)
│   ├── azure_storage.py        Blob Storage document sync (planned, not yet implemented)
│   └── telemetry.py            Persistent structured query logging
├── scripts/
│   ├── __init__.py
│   ├── ingest.py                Parse + chunk + embed + index, with hash-based dedup
│   ├── sync_azure.py            Push docs to Blob Storage + Azure AI Search (planned)
│   └── run_eval.py              Benchmark harness (planned)
├── static/                     Dashboard UI (chat, latency trace, explainability
│                                panel, entity graph viewer) — Fluent Design theme
├── docs/
│   ├── ROADMAP.md               Full advanced-RAG roadmap and build order
│   ├── sample_docs/             Example knowledge base (multi-format)
│   └── eval_set.json            Labeled Q&A pairs for benchmarking (planned)
├── tests/                       Unit + integration tests (planned)
├── data/                        SQLite DB (gitignored)
├── .env.example                 Template for environment variables
└── requirements.txt

---

Quick Start

Prerequisites

Install Foundry Local

macOS

brew install microsoft/foundrylocal/foundrylocal

Windows

winget install Microsoft.FoundryLocal

Start Foundry Local

foundry service start
foundry model run Phi-4-mini-instruct-generic-gpu:5

Foundry Local prints the service URL and port. The port may change between restarts.

Create the Python environment

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Windows activation:

.venv\Scripts\activate

Configure local mode

MODE=local
FOUNDRY_BASE_URL=http://127.0.0.1:YOUR_PORT/
FOUNDRY_CHAT_MODEL=Phi-4-mini-instruct-generic-gpu:5
ENABLE_ENTITY_GRAPH=false

Ingest documents

python scripts/ingest.py

Enable entity graph extraction:

ENABLE_ENTITY_GRAPH=true python scripts/ingest.py

Entity extraction performs one LLM call per document section and can take a long time for large technical manuals.

Run the application

uvicorn api.app:app --reload

Open:

http://127.0.0.1:8000

After the required models are downloaded, the application can run without internet access.

---

Current Limitations

• The entity graph is currently an exploration and explainability layer, not a retrieval signal.
• Entity extraction can be expensive for documents containing thousands of sections.
• Local generation latency depends on hardware, context size, and model selection.
• The current benchmark measures retrieval quality but does not include automated answer faithfulness scoring.
• Azure retrieval and generation backends are architectural integration points and have not been validated end-to-end in the stable release.
• The local knowledge tree and entity graph remain the source of truth even in the cloud-ready design.

---

Project Resources

Documentation

Project Presentation

Repository

---

Author

Mahmut Esat Kolay

Computer Engineering student focused on artificial intelligence, retrieval systems, local LLM applications, and industrial document intelligence.

Developed as part of the Microsoft Türkiye AI Innovators Summer Internship.

---

License

This project is licensed under the MIT License.

---

FoundryRAG

Structured retrieval. Measurable behavior. Verifiable answers.