v1.6.0: Test All Formats with Clear Framework Support Visualization

🎉 Release v1.6.0

📊 Complete Format Testing with Transparency

This release changes the default behavior to test ALL 18 formats while providing clear visualization of which frameworks support which formats.

🔄 Key Changes

1. CI Tests All Formats by Default

• Changed from 'common' tier to 'all' formats
• Benchmarks now test complete framework capabilities
• Framework-specific exclusions remain in place

2. New Format Support Matrix Visualization

• Visual heatmap showing ✓/✗ for each framework-format combination
• Instantly see which formats each framework supports
• New chart: format_support_matrix.png

3. Enhanced Reporting

• Success rate chart now states: "Testing ALL 18 formats"
• Console reports include format exclusion table
• Shows exactly which formats each framework skips

📋 Format Testing Details

Total Formats Tested: 18

• Documents: PDF, DOCX, PPTX, XLSX, XLS, ODT
• Web/Markup: HTML, MD, RST, ORG
• Images: PNG, JPG, JPEG, BMP
• Email: EML, MSG
• Data: CSV, JSON, YAML
• Text: TXT

Framework Exclusions:

• Kreuzberg: No email/data formats (by design)
• Extractous: Skips DOCX, JPG
• Docling: Limited format support
• MarkItDown: Skips DOCX, MD, ODT
• Unstructured: Skips JPEG, JPG, ODT, ORG, RST

🎯 Benefits

1. Complete Testing: See full framework capabilities
2. Transparency: Clear visualization of format support
3. Fair Comparison: Frameworks aren't penalized for unsupported formats
4. User Clarity: Know exactly what's being tested

📈 Expected Results

With all formats tested:

• Success rates will reflect actual framework capabilities
• Format support matrix shows which failures are expected
• Users can make informed decisions based on format needs

🚀 Usage

The new format support matrix will be automatically generated in benchmark reports, providing immediate visibility into framework capabilities across all tested formats.