Skip to content
This repository was archived by the owner on Sep 28, 2025. It is now read-only.

v1.4.0: Tiered Format Support for Nuanced Benchmarking

Choose a tag to compare

View all tags
@Goldziher Goldziher released this 11 Jul 12:47
· 131 commits to main since this release
v1.4.0
This tag was signed with the committer’s verified signature.
Goldziher Na'aman Hirschfeld
GPG key ID: 8C3B71B34E1FAAE0
Verified
Learn about vigilant mode.
e3dc3d9
This commit was signed with the committer’s verified signature.
Goldziher Na'aman Hirschfeld
GPG key ID: 8C3B71B34E1FAAE0
Verified
Learn about vigilant mode.

🎉 Release v1.4.0

🎯 Tiered Format Support

This release introduces a sophisticated 3-tier format support system that provides more nuanced benchmarking options beyond the simple binary choice.

Format Tiers

Tier 1: Universal Support (5/5 frameworks)

  • 7 formats: PDF, PPTX, XLSX, PNG, BMP, HTML, CSV
  • 100% framework support for fair performance comparison
  • Use: --format-tier universal

Tier 2: Common Support (4/5 frameworks)

  • 11 formats: Tier 1 + XLS, MD, JPEG, TXT
  • 80%+ framework support for real-world scenarios
  • Use: --format-tier common (default)

Tier 3: Partial Support (3/5 frameworks)

  • 16 formats: Tier 2 + JPG, EML, MSG, JSON, YAML
  • 60%+ support for specialized format testing
  • Use: --format-tier partial

All Formats

  • 18 formats: Complete capability assessment
  • Use: --format-tier all

🚀 Key Improvements

  • Flexible Benchmarking: Choose the right tier for your use case
  • CI/CD Integration: Manual workflow runs can select format tier
  • Default Strategy: Uses Tier 2 (Common) for balanced benchmarking
  • Legacy Support: --common-formats-only still works (maps to Tier 1)

📊 Expected Success Rates by Tier

  • Tier 1: All frameworks achieve ~100% success
  • Tier 2: Most frameworks achieve 85-95% success
  • Tier 3: Framework-dependent, shows specialized capabilities
  • All: Realistic success rates showing full capabilities

🛠️ Usage Examples

# Fair performance comparison
uv run python -m src.cli benchmark --framework all --format-tier universal

# Real-world benchmarking (default)
uv run python -m src.cli benchmark --framework all --format-tier common

# Test email/data format capabilities
uv run python -m src.cli benchmark --framework all --format-tier partial

# Complete framework evaluation
uv run python -m src.cli benchmark --framework all --format-tier all

📚 Documentation Updates

  • Updated README with tiered format support explanation
  • Enhanced FRAMEWORK_FORMAT_SUPPORT.md with tier definitions
  • Added benchmarking strategy recommendations

This release provides a more sophisticated approach to benchmarking, allowing users to choose between fair comparison, real-world scenarios, and complete capability assessment.

Assets 2
Loading