v1.0.2 — Live Demo Links, PyPI Sidebar URLs & Architecture Specifications

Overview

universal-doc-parser v1.0.2 introduces direct Hugging Face Space interactive demo integrations, expands project metadata with PyPI sidebar URLs, and completes the technical documentation suite across docs/.

---

What's New

• Live Demo Integration: Added interactive demo badges and links to README.md and PyPI package metadata ([project.urls]).
• Full Architecture Specification: Populated docs/ARCHITECTURE.md with complete technical breakdowns of the two-pass extraction engine, font-size percentile hierarchy, and adaptive layout fingerprinting.
• Schema Documentation: Populated docs/SCHEMA.md with field-level references, type constraints, and null-safety semantics for all Pydantic models.
• Contributor Guide: Populated docs/ADDING_A_FORMAT.md with step-by-step instructions for implementing new format extractors.
• UI Responsiveness: Implemented payload safeguards in main.py ensuring fast rendering on multi-page reports.

---

Live Demo & Package

• Interactive Web Demo: https://huggingface.co/spaces/Karan6124/universal-doc-parser
• PyPI Package: https://pypi.org/project/universal-doc-parser/1.0.2/

v1.0.1 — Technical Report Rewrite, PyPI Metadata & Documentation Alignment

Overview

universal-doc-parser v1.0.1 aligns repository documentation, PyPI packaging metadata, and technical disclosures. This release repositions the project as a focused technical report, transparently evaluates the competitive landscape alongside Docling, Marker, and Unstructured, and refines PyPI image rendering.

---

What's Changed

• Technical Report Structure: Repositioned the repository documentation from marketing claims to an honest technical report detailing architectural scope, explicit limitations, and planned ablation studies.
• Related Work & Competitive Context: Added in-depth comparison section honestly benchmarking capabilities against Docling, Marker, and Unstructured.
• Differentiator Highlight: Elevated the adaptive layout fingerprinting and template auto-tuning cache as the primary novel contribution for recurring document templates.
• PyPI Installation Fix: Corrected installation commands across all documentation to reference the canonical PyPI package name: universal-doc-parser.
• License Transparency: Disclosed GPL-3.0 copyleft terms for optional Outlook .msg parsing (extract-msg) alongside MIT/Apache-2.0 core dependencies.
• PyPI Banner Rendering: Optimized banner image HTML markup for compatibility with PyPI's warehouse-camo secure proxy.
• Benchmark Disclosure: Clarified live API vs. simulated evaluations, removing unverified simulation figures from published tables.

---

Installation

pip install universal-doc-parser

v1.0.0 - Production Release: Multi-Format Engine, Adaptive Fingerprinting, FastMCP, and PyPI Distribution

Overview

universal-doc-parser v1.0.0 is the first production release of the zero-GPU, CPU-only document ingestion engine built specifically for RAG pipelines and autonomous AI agents. It provides deterministic layout preservation, sub-500ms processing latencies, strictly bounded memory consumption (< 250MB RSS), and 100% commercially permissive licensing across 15+ file formats.

---

What is in v1.0.0

Core Extraction Architecture

• Unified, versioned Pydantic schema across 15+ formats (PDF, DOCX, XLSX, PPTX, HTML, EPUB, CSV, TSV, Parquet, JSON, XML, EML, MSG, MBOX, and legacy OLE binary documents).
• Deterministic multi-column reading order preservation via geometric coordinate clustering.
• Font-size percentile hierarchy computation (P95, P85, P75) for structured heading trees.
• Dual-strategy table extraction (Lattice and Stream) with cell word-density filtering and table height ratio heuristics to prevent side-by-side text block merging.
• Built-in CPU-based OCR fallback via RapidOCR and OpenCV for scanned pages and images with automatic deskew and orientation correction.

Downstream RAG and Knowledge Integration

• Hierarchical chunking (to_chunks) with heading ancestry context breadcrumbs injected into every chunk to eliminate retrieval hallucinations.
• Clean GitHub-Flavored Markdown generation (to_markdown).
• Entity and relationship Knowledge Graph extraction (to_graph).

Adaptive Layout Engine

• 2D spatial histogram discretization (10x10 grid) for content-agnostic document layout fingerprinting.
• Thread-safe LRU template configuration cache with exact SHA-256 and fuzzy Cosine-Jaccard similarity lookups.
• Coordinate-descent heuristic auto-tuner for optimizing extraction parameters on repeating document layouts.

FastMCP Server Integration

• Built-in Model Context Protocol (MCP) server over standard I/O for direct tool integration with Claude Desktop, Cursor, and AI agents (parse_document, list_supported_formats).

Observability and Telemetry

• Real-time ingestion telemetry metrics tracking duration, page volume, element counts, and anomaly signals.
• Standalone self-contained HTML telemetry and drift dashboard export.

Performance and Multi-Model Benchmarks

• Verified < 250MB RSS memory ceiling across 1,000+ page streaming document tests.
• Comprehensive multi-model benchmark suite profiling local CPU parsing against 15 frontier and open-weight LLMs (Google Gemini, OpenAI GPT-4o, Anthropic Claude, DeepSeek, Alibaba Qwen, Meta Llama, Mistral AI, Moonshot Kimi, Zhipu GLM, and Cohere).

---

Installation

Via PyPI

pip install universal-doc-parser

v0.11.0 - Multi-Model LLM Benchmarking Hub and Observability Dashboard

Overview

v0.11.0 delivers the Multi-Model LLM Benchmarking Hub, the Observability & Drift Dashboard, and a complete Permissive Licensing Refactor (migrating the PDF backend to Apache-2.0 pypdfium2 and MIT pdfplumber with zero AGPL dependencies).

---

What's New in v0.11.0

1. Multi-Model LLM Benchmarking Suite (benchmarks/providers/)

• Modular Provider Architecture: Hierarchical model abstraction supporting distinct subpackages for each major AI provider:
  • benchmarks/providers/openai/: gpt-4o, gpt-4o-mini
  • benchmarks/providers/gemini/: gemini-2.0-flash, gemini-2.0-pro
  • benchmarks/providers/anthropic/: claude-3-5-sonnet, claude-3-opus
  • benchmarks/providers/deepseek/: deepseek-v3, deepseek-r1
  • benchmarks/providers/qwen/: qwen-2.5-72b, qwen-2.5-coder-32b
• Cost & Latency Evaluation Engine: Standardized metrics calculating extraction throughput, table reconstruction accuracy, RAG faithfulness, and token cost in USD based on official vendor pricing.
• Dual Execution Modes: Includes an offline deterministic simulation engine (dry_run=True) for zero-cost CI/CD testing alongside live production API key execution.

2. Observability & Drift Dashboard (universal_parser/observability/)

• MetricsCollector Singleton: Thread-safe telemetry collector tracking per-document duration, page counts, element density, and anomaly signals.
• Responsive HTML Dashboard: Self-contained dashboard generator (export_dashboard()) visualizing format throughput, average ingestion latency, and anomaly alerts.

3. Permissive Licensing & Engine Refactor

• Completely eliminated PyMuPDF (AGPL) from the dependency tree.
• Replaced with pypdfium2 (Google Chrome's C++ PDFium engine, Apache 2.0 / BSD) and pdfplumber (MIT).
• The entire universal-parser codebase is now 100% permissively licensed (MIT / Apache 2.0 / BSD) for enterprise and commercial use.

---

Test Suite & Quality Status

• Total Unit Tests: 60 passed (100% pass rate).
• Ruff Lint & Format: 0 errors, all checks passing.
• Peak Memory RSS: 230.57 MB (well under the 250.0 MB RAM budget limit).

---

Verification Commands

uv run pytest
uv run python benchmarks/run_llm_benchmark.py
uv run python benchmarks/memory_profile.py
uv run ruff check .

v0.10.0 - Adaptive Template Fingerprinting, LRU Config Cache and Guardrailed Auto-Tuner

Overview

v0.10.0 introduces the Adaptive Template Intelligence Layer to Universal Parser. This enables the engine to generate content-agnostic structural layout fingerprints, recognize recurring document templates via fuzzy similarity matching, cache per-template parsing configurations, and auto-tune parsing hyperparameters using feedback error signals with strict mathematical guardrails.

---

What's New in v0.10.0

1. Content-Agnostic Structural Layout Fingerprinting

• 2D Spatial Histogram Discretization: Quantizes element bounding boxes $(x_0, y_0, x_1, y_1)$ normalized to $[0.0, 1.0] \times [0.0, 1.0]$ into a fixed $10 \times 10$ occupancy and density matrix.
• Categorical Element Type Distribution: Computes normalized proportions of document element types (headings, paragraphs, tables, figures, code blocks).
• Deterministic SHA-256 Digest: Generates content-independent layout hashes that remain identical even when document text values change (e.g. across monthly invoices or templated filings).

2. Multi-Modal Layout Similarity Engine

• Weighted Cosine-Jaccard Metric: Combines 2D spatial layout alignment ($70%$ weight) and element type categorical distribution ($30%$ weight) into a normalized similarity score $\in [0.0, 1.0]$.
• Enables robust fuzzy template matching for layout variations with minor coordinate shifts or added table rows.

3. Thread-Safe LRU Template Config Cache

• Dual Retrieval: Provides $O(1)$ exact hash lookups with automatic fallback to fuzzy similarity matching ($\ge 0.85$ threshold).
• Least Recently Used (LRU) Eviction: Bounded in-memory capacity management.
• Atomic Disk Persistence: Safe serialization and deserialization to JSON (.save() and .load()).

4. Error Gradient Auto-Tuning with Mathematical Guardrails

• Coordinate Descent Tuning: Adjusts parser hyperparameters based on explicit or automated feedback signals (missed_heading, false_heading, missed_table, false_table, merged_columns, split_columns, ocr_quality).
• Mathematical Guardrails: Parameter clipping constraints that prevent hyperparameters from drifting into degenerate or unstable regimes:
  • column_gap_threshold: $[10.0, 120.0]\text{ pt}$
  • heading_p95_ratio: $[1.05, 3.00]$
  • heading_p85_ratio: $[1.01, 2.50]$
  • min_table_confidence: $[0.30, 0.95]$
  • ocr_dpi: $[100, 300]$

---

Test Suite & Quality Verification

• Total Unit Tests: 58 passed (100% pass rate across all 15 formats and adaptive modules).
• Ruff Lint & Format: 0 errors, all checks passing.
• Memory RSS: Zero memory leaks, strictly within the 250MB RAM budget.

---

Verification Commands

uv run pytest
uv run ruff check .

v0.9.0 - Model Context Protocol (MCP) Interface for AI Agents

What's New in v0.9.0

• Model Context Protocol (MCP) Server: Added universal_parser/mcp/server.py implementing a FastMCP standard server over stdio for direct integration into Claude Desktop, Cursor, and autonomous agent swarms.
• parse_document Tool: Exposes document parsing to AI agents supporting markdown, chunks, and json output formats across 15+ file types.
• list_supported_formats Tool: Exposes dynamic format registry discovery to AI agents.
• All 50 tests passing with 0 Ruff lint errors.

v0.7.0 - Memory Hardening and Performance Benchmarks

What's New in v0.7.0

• Memory Profiling Suite: Added benchmarks/memory_profile.py to mathematically profile and verify CPU memory consumption on large multi-page documents.
• Streaming Verification: Verified peak RSS remains strictly under the 250MB budget limit across 100-page document ingestion and RAG chunk generation.
• Zero Memory Leaks: Audited generator stream pipelines across all document extractors.
• All 46 tests passing with 0 Ruff lint errors.

v0.8.0 - RAG and Knowledge Graph Export Layer

What's New in v0.8.0

• Markdown Export: Added to_markdown() converting any parsed Document across 15+ formats into clean, unified Markdown.
• Hierarchical Token-Aware Chunker: Added to_chunks() preserving active heading hierarchies (e.g. Context: H1 > H2) and page numbers for vector embeddings and RAG pipelines.
• Knowledge Graph Exporter: Added to_graph() generating nodes and edges (CONTAINS, CONTAINS_SECTION, FOLLOWS) for GraphRAG and Neo4j.
• All 46 tests passing with 0 Ruff lint errors.

v0.6.0 - PowerPoint and Legacy Office Binary Support

What's New in v0.6.0

• PowerPoint Support: Added PPTXExtractor for modern .pptx presentations, extracting slide titles as headings, text frames/bullets as list items, and slide tables into TableData grids.
• Legacy Office Binary Formats: Added LegacyOfficeExtractor using xlrd and olefile to parse legacy Microsoft Office 97-2003 binary formats (.doc, .xls, .ppt).
• All 43 tests passing with 0 Ruff lint errors across 15 document formats.

v0.5.0 - Email and Attachment Ingestion

What's New in v0.5.0

• Email Formats Supported: Added MailExtractor for .eml, .msg (Outlook binary), and .mbox mailbox archives.
• Header & Body Extraction: Parses Subject, From, To, and Date metadata alongside HTML/Plaintext body hierarchies.
• Recursive Attachment Ingestion: Automatically saves attached documents (PDFs, spreadsheets, images, CSVs), sniffs their format, and routes them through specialized sub-extractors.
• All 39 tests passing with 0 Ruff lint errors across 12 document formats.