Skip to content

Supported File Types

Anjana Tiha edited this page Mar 28, 2026 · 1 revision

Supported File Types

RAG Agent supports 9 document formats across two chunker modules.

Format Reference

Format Extensions Chunking Strategy Library
PDF .pdf Sentence windows per page (5 sentences, page-isolated) PyMuPDF
Word .docx, .doc Paragraph groups (3 paragraphs) + individual table rows python-docx
Excel .xlsx Row → column=value key-value pairs, per sheet openpyxl
Excel (legacy) .xls Row → column=value key-value pairs xlrd
PowerPoint .pptx Text shapes per slide (1 slide per chunk) python-pptx
CSV .csv Row → column=value key-value pairs stdlib csv
Plain text .txt Line-based sliding window (1 line, 0 overlap) stdlib
Markdown .md, .markdown Line-based, headings/bold/italic/code stripped stdlib
HTML .html, .htm Sentence windows (5 sentences), tags stripped BeautifulSoup4

Format Details

PDF

Each page is processed independently to prevent context bleeding between pages. Within each page, 5-sentence sliding windows produce overlapping chunks so that answers that span a sentence boundary are still retrievable.

Word (.docx)

Paragraphs are grouped into windows of 3. Table cells are extracted as individual rows with merged-cell deduplication — merged cells are not repeated across every row they span.

Excel (.xlsx / .xls)

Each row is converted to a human-readable string: Column1=Value1 | Column2=Value2 | .... This format allows the LLM reranker and response model to interpret structured data without needing to parse a table layout.

PowerPoint (.pptx)

All text shapes on each slide are joined into a single chunk. This keeps related bullet points together and preserves slide-level context.

CSV

Same row → key-value strategy as Excel. Column headers from the first row become the keys.

Plain Text

Lines are chunked in windows of 1 line (configurable via TXT_CHUNK_SIZE in config.py). Good for log files, prose, and code.

Markdown

Markdown syntax (headings #, bold **, italic _, code fences, image tags, links) is stripped before chunking so the LLM sees clean prose rather than raw markup.

HTML

HTML tags are removed with BeautifulSoup, then the cleaned text is split into 5-sentence windows. This handles both local HTML files and web pages fetched via URL ingestion.


Chunk Truncation

All chunks are truncated before storage to prevent embedding model overflow:

  • Storage truncation: 300 words or 1,200 characters — whichever is shorter
  • Embedding truncation: 200 words and 1,200 characters — both limits enforced

These limits are set conservatively relative to the BGE base model's 512-token context window.


Adding a New Format

To add support for a new file type:

  1. Add the extension to EXT_TO_TYPE in src/rag/config.py
  2. Add a chunk_<type> function in src/rag/chunkers.py (text-based) or src/rag/binary_chunkers.py (binary)
  3. Add a dispatch case in DocumentLoader._dispatch_chunker()
  4. Write unit and integration tests following the pattern in tests/test_document_loader.py

Clone this wiki locally