PDF Extraction Library Benchmark

A comprehensive benchmark comparing Python PDF text extraction libraries across accuracy, table extraction, reading order, speed, and memory usage.

Libraries Tested

Library	Version	Type
PyMuPDF/fitz	1.26.5	Python
pdfplumber	0.11.8	Python
pypdf	6.11.0	Python
pdfminer.six	20251107	Python
pdftotext	poppler	CLI (subprocess)
pdftotext -layout	poppler	CLI (subprocess)
LlamaIndex PDFReader	0.x	Python (pypdf wrapper)

Methodology

Test PDFs

• 10 distinct PDFs generated with reportlab, each covering a different topic:
  • AI in Healthcare, Climate & Renewable Energy, Quantum Computing, Supply Chain, Neuroscience, Blockchain/DeFi, Space Exploration, Genomics, Autonomous Vehicles, Large Language Models
• Each PDF contains:
  • Title and multi-paragraph body text
  • A 2-column layout section
  • A 4-column data table (5 rows)
  • Footer on each page
  • 2 pages total
• All content is generated with known ground truth for accurate scoring

Metrics

Metric	Weight	Description
Text accuracy	40%	Character-level match % against ground truth using SequenceMatcher
Table extraction	25%	Rows, columns, headers, and data values correctly extracted (0-100)
Reading order	20%	Multi-column text returned in correct left-to-right sequence (0-100)
Speed	20%	Average milliseconds per page (normalised 0-100, lower is better)
Memory	15%	Peak RAM in MB during extraction (normalised 0-100, lower is better)

Runs

• 8 benchmark rounds total across all 10 datasets
• Results averaged for final ranking

Results

Final Ranking (8 runs, 10 datasets)

Rank	Library	Avg Score	Accuracy	Tables	Reading Order	Consistency
1	PyMuPDF/fitz	44.2	67.8%	100/100	100/100	Consistent
2	pypdf	44.4	67.8%	0/100	100/100	High variance
3	pdfplumber	39.6	25.1%	100/100	0/100	High variance
4	pdftotext-layout	39.1	19.9%	0/100	100/100	Moderate
5	pdftotext	35.2	20.4%	0/100	100/100	Moderate
6	pdfminer.six	33.2	20.4%	0/100	100/100	Very stable
-	LlamaIndex PDFReader	62.0*	67.8%	0/100	100/100	2 runs only

*LlamaIndex PDFReader is a thin wrapper around pypdf — it calls pypdf.PdfReader internally and provides no independent extraction capability.

Key Findings

• PyMuPDF/fitz is the only library that scored 100/100 on table extraction AND reading order across all 10 datasets without a single failure. It is the clear recommendation for any use case requiring reliable extraction from varied PDF layouts.
• pypdf matches PyMuPDF on text accuracy but has zero table extraction capability.
• pdfplumber failed reading order on every single dataset (0/100) — it garbles multi-column text by interleaving characters from adjacent columns.
• pdfminer.six is the most stable library (±2.7 std dev) but consistently ranks last or second-to-last.
• LlamaIndex PDFReader is disqualified as a separate comparison — it is literally pypdf with a Document wrapper.

Recommended Fallback Chain

PyMuPDF (fitz) -> pdfplumber -> pypdf -> pdfminer.six

Usage

Requirements

pip install reportlab pypdf pdfplumber PyMuPDF pdfminer.six
brew install poppler  # for pdftotext CLI

Run the benchmark

python benchmark.py

This will:

1. Generate 5 test PDFs with distinct content
2. Run all libraries against each PDF
3. Print a ranked results table with per-dataset breakdowns

Run the aggregate ranking

python aggregate_ranks.py

Compiles scores from all recorded benchmark runs and prints a cross-run ranking with consistency analysis.

File Structure

pdf-extraction-benchmark/
├── benchmark.py          # Main benchmark script
├── aggregate_ranks.py    # Cross-run aggregation and final ranking
└── README.md

Context

This benchmark was conducted to inform the PDF extraction fallback chain for EssayBot, a document processing system that parses uploaded PDFs and course materials for essay analysis.