Skip to content

v0.11.0 - Multi-Model LLM Benchmarking Hub and Observability Dashboard

Choose a tag to compare

View all tags
@Edge-Explorer Edge-Explorer released this 08 Sep 05:55
· 64 commits to main since this release
v0.11.0
fd45b54

Overview

v0.11.0 delivers the Multi-Model LLM Benchmarking Hub, the Observability & Drift Dashboard, and a complete Permissive Licensing Refactor (migrating the PDF backend to Apache-2.0 pypdfium2 and MIT pdfplumber with zero AGPL dependencies).

What's New in v0.11.0

1. Multi-Model LLM Benchmarking Suite (benchmarks/providers/)

  • Modular Provider Architecture: Hierarchical model abstraction supporting distinct subpackages for each major AI provider:
    • benchmarks/providers/openai/: gpt-4o, gpt-4o-mini
    • benchmarks/providers/gemini/: gemini-2.0-flash, gemini-2.0-pro
    • benchmarks/providers/anthropic/: claude-3-5-sonnet, claude-3-opus
    • benchmarks/providers/deepseek/: deepseek-v3, deepseek-r1
    • benchmarks/providers/qwen/: qwen-2.5-72b, qwen-2.5-coder-32b
  • Cost & Latency Evaluation Engine: Standardized metrics calculating extraction throughput, table reconstruction accuracy, RAG faithfulness, and token cost in USD based on official vendor pricing.
  • Dual Execution Modes: Includes an offline deterministic simulation engine (dry_run=True) for zero-cost CI/CD testing alongside live production API key execution.

2. Observability & Drift Dashboard (universal_parser/observability/)

  • MetricsCollector Singleton: Thread-safe telemetry collector tracking per-document duration, page counts, element density, and anomaly signals.
  • Responsive HTML Dashboard: Self-contained dashboard generator (export_dashboard()) visualizing format throughput, average ingestion latency, and anomaly alerts.

3. Permissive Licensing & Engine Refactor

  • Completely eliminated PyMuPDF (AGPL) from the dependency tree.
  • Replaced with pypdfium2 (Google Chrome's C++ PDFium engine, Apache 2.0 / BSD) and pdfplumber (MIT).
  • The entire universal-parser codebase is now 100% permissively licensed (MIT / Apache 2.0 / BSD) for enterprise and commercial use.

Test Suite & Quality Status

  • Total Unit Tests: 60 passed (100% pass rate).
  • Ruff Lint & Format: 0 errors, all checks passing.
  • Peak Memory RSS: 230.57 MB (well under the 250.0 MB RAM budget limit).

Verification Commands

uv run pytest
uv run python benchmarks/run_llm_benchmark.py
uv run python benchmarks/memory_profile.py
uv run ruff check .
Assets 2
Loading