feat: add pdf-inspector as an optional PDF parser backend

[Kushpatel49]

Summary

Adds pdf_inspector as an opt-in third backend alongside the existing PyPDF2 (default) and PyMuPDF paths in get_page_tokens and friends. Default behavior is unchanged — users opt in via --pdf-parser pdf_inspector on the CLI or pdf_parser: "pdf_inspector" in config.yaml.

pdf-inspector is a Rust-based PDF parser (with prebuilt Python wheels on PyPI) that emits per-page Markdown with GFM tables, heading tiers, and list markers preserved — signals PageIndex's tree parser and node-summary steps already consume.

Also adds an optional --check-ocr preflight that classifies the PDF with pdf-inspector and refuses image-based files upfront, so users don't spend LLM tokens on a document that needs OCR.

Benchmark

Ran both backends against the eight fixture PDFs in examples/documents/:

Metric	PyPDF2	pdf-inspector	Δ
Total extraction time	15,300 ms	3,340 ms	4.6× faster
Total characters	3,660,778	3,592,631	~equal
GFM tables produced	0	639	∞
ATX headings produced	4	2,028	507×
List markers	193	699	3.6×

Highlights:

• 2023-annual-report.pdf (222 p): 0 → 176 tables. Annual reports are the FinanceBench-shaped case where table structure is load-bearing.
• PRML.pdf (758 p): 0 → 1,212 ATX headings, i.e. real hierarchy signal for the tree.
• q1-fy25-earnings.pdf: 0 → 12 tables (every financial statement).
• Prose-heavy four-lectures.pdf shows no regression.

Changes

• pageindex/utils.py — SUPPORTED_PDF_PARSERS tuple, lazy _load_pdf_inspector(), classify_pdf() helper, and pdf_parser= branches in extract_text_from_pdf, get_text_of_pages, get_page_tokens, get_number_of_pages. get_number_of_pages gracefully falls back to PyPDF2 if pdf-inspector isn't installed so opt-in stays soft.
• pageindex/page_index.py — one-line: threads opt.pdf_parser into get_page_tokens.
• pageindex/config.yaml — new key pdf_parser: "PyPDF2" (default preserved).
• run_pageindex.py — --pdf-parser {PyPDF2,PyMuPDF,pdf_inspector} and --check-ocr flags.
• tests/test_pdf_parser_backend.py — 12 unit tests covering dispatch, unknown-parser rejection, soft fallback when pdf-inspector is missing, and end-to-end use against a fixture PDF. All tests skip cleanly when pdf-inspector isn't installed.

Compatibility

• Not adding pdf-inspector to requirements.txt — it remains truly optional. Users who want it: pip install pdf-inspector.
• Default pdf_parser: "PyPDF2" means zero behavior change for existing users.
• All 18 pre-existing tests still pass alongside the 12 new ones (30/30 green).

Test plan

• python -m pytest tests/ — 30 passed
• run_pageindex.py --help shows new flags
• run_pageindex.py --pdf_path <image-based>.pdf --check-ocr correctly refuses with a helpful message
• Benchmark run against all fixture PDFs
• Reviewer: try pip install pdf-inspector && python run_pageindex.py --pdf_path examples/documents/2023-annual-report-truncated.pdf --pdf-parser pdf_inspector and compare the resulting tree JSON against the PyPDF2 baseline

Notes for reviewer

• pdf-inspector is Rust + wheels on PyPI, so pip install pdf-inspector is a normal-speed install — no cargo required.
• The --check-ocr flag is deliberately its own switch (not implied by --pdf-parser pdf_inspector) so users can pair OCR gating with any backend.
• pdf-inspector does not do OCR. Scanned/image-based PDFs still need an external OCR (MinerU, etc.). The --check-ocr gate just makes that failure mode explicit instead of silent-garbage-in.