Repository navigation
Supported File Types
RAG Agent supports 9 document formats across two chunker modules.
| Format | Extensions | Chunking Strategy | Library |
|---|---|---|---|
.pdf |
Sentence windows per page (5 sentences, page-isolated) | PyMuPDF | |
| Word |
.docx, .doc
|
Paragraph groups (3 paragraphs) + individual table rows | python-docx |
| Excel | .xlsx |
Row → column=value key-value pairs, per sheet |
openpyxl |
| Excel (legacy) | .xls |
Row → column=value key-value pairs |
xlrd |
| PowerPoint | .pptx |
Text shapes per slide (1 slide per chunk) | python-pptx |
| CSV | .csv |
Row → column=value key-value pairs |
stdlib csv |
| Plain text | .txt |
Line-based sliding window (1 line, 0 overlap) | stdlib |
| Markdown |
.md, .markdown
|
Line-based, headings/bold/italic/code stripped | stdlib |
| HTML |
.html, .htm
|
Sentence windows (5 sentences), tags stripped | BeautifulSoup4 |
Each page is processed independently to prevent context bleeding between pages. Within each page, 5-sentence sliding windows produce overlapping chunks so that answers that span a sentence boundary are still retrievable.
Paragraphs are grouped into windows of 3. Table cells are extracted as individual rows with merged-cell deduplication — merged cells are not repeated across every row they span.
Each row is converted to a human-readable string: Column1=Value1 | Column2=Value2 | .... This format allows the LLM reranker and response model to interpret structured data without needing to parse a table layout.
All text shapes on each slide are joined into a single chunk. This keeps related bullet points together and preserves slide-level context.
Same row → key-value strategy as Excel. Column headers from the first row become the keys.
Lines are chunked in windows of 1 line (configurable via TXT_CHUNK_SIZE in config.py). Good for log files, prose, and code.
Markdown syntax (headings #, bold **, italic _, code fences, image tags, links) is stripped before chunking so the LLM sees clean prose rather than raw markup.
HTML tags are removed with BeautifulSoup, then the cleaned text is split into 5-sentence windows. This handles both local HTML files and web pages fetched via URL ingestion.
All chunks are truncated before storage to prevent embedding model overflow:
- Storage truncation: 300 words or 1,200 characters — whichever is shorter
- Embedding truncation: 200 words and 1,200 characters — both limits enforced
These limits are set conservatively relative to the BGE base model's 512-token context window.
To add support for a new file type:
- Add the extension to
EXT_TO_TYPEinsrc/rag/config.py - Add a
chunk_<type>function insrc/rag/chunkers.py(text-based) orsrc/rag/binary_chunkers.py(binary) - Add a dispatch case in
DocumentLoader._dispatch_chunker() - Write unit and integration tests following the pattern in
tests/test_document_loader.py