v1.5.0 — Content-Aware Chunking & PDF Table/Image Extraction

What's New

Content-Aware Chunk Boundaries

• Tables and fenced code blocks are now detected as atomic structural units and kept together in chunks
• Soft cap 500 tokens (prose), hard cap 1000 tokens (structural blocks)
• Oversized tables split at row boundaries with header propagation into each sub-chunk
• Oversized code fences split at line boundaries with opening/closing markers preserved

PDF Table Extraction

• PyMuPDF's find_tables() detects tables in PDFs and renders them as markdown pipe tables
• Replaces previous whitespace-aligned garbled text with clean | col | col | format
• Phantom empty columns (from PyMuPDF layout misalignment) automatically stripped
• Tables positioned correctly among surrounding text via y-coordinate ordering

Image Extraction & Vision Summarization

• Images extracted from PDFs at parse time, filtered (skips icons < 50px and < 5KB)
• [Figure: filename] placeholders inserted in chunk text at correct positions
• During chapter summarization, images sent to the LLM as base64 content blocks
• LLM can now describe diagrams and figures in chapter summaries
• Capped at 5 images per chapter to control costs
• Automatic fallback to text-only if the configured model doesn't support vision

Bug Fixes

• Fixed corpus pipeline crash from parse_pdf() return type change
• Fixed /knowledge → /agentlib-knowledge references in ingestion commands

Testing

• 72 tests passing (up from ~34)
• Verified end-to-end on the CycloneDX Authoritative Guide to SBOM

Full Changelog: v1.4.0...v1.5.0