Add LiteParse plugin for PDF text extraction

[pirate]

Summary

Adds a new LiteParse plugin that extracts text and metadata from PDF documents using the lit CLI tool by LlamaIndex. The plugin discovers PDFs from other plugin outputs (pdf, responses, staticfile) and processes all found documents, combining results into unified text and JSON outputs.

Key Changes

• Snapshot hook (on_Snapshot__61_liteparse.py): Main extraction logic that:

  • Discovers all PDF files from sibling plugin output directories
  • Runs lit parse on each PDF in both text and JSON formats
  • Combines results with source attribution
  • Generates metadata tracking sources processed
  • Supports configuration via environment variables (timeout, custom args, enable/disable)
  • Handles missing sources gracefully (returns noresults)

• Crawl hook (on_Crawl__43_liteparse_install.finite.bg.py): Binary dependency declaration that:

  • Emits a Binary record for the lit tool
  • Specifies npm installation via @llamaindex/liteparse package
  • Respects LITEPARSE_ENABLED flag and custom LITEPARSE_BINARY paths

• Configuration (config.json): Schema defining:

  • LITEPARSE_ENABLED: Toggle extraction on/off
  • LITEPARSE_BINARY: Custom binary path
  • LITEPARSE_TIMEOUT: Extraction timeout (default 120s)
  • LITEPARSE_ARGS / LITEPARSE_ARGS_EXTRA: Command-line arguments

• Comprehensive test suite (test_liteparse.py): Integration tests covering:

  • Hook script existence and proper structure
  • Crawl hook Binary record emission
  • Binary installation via npm and abx_pkg
  • Configuration disable flag behavior
  • Graceful handling of missing PDF sources
  • Single and multiple PDF extraction with real downloaded PDFs
  • Output validation (content.txt, metadata.json)

Implementation Details

• Uses lit parse command with both text and JSON output formats
• Processes PDFs sequentially with configurable timeout per file
• Combines multiple PDF outputs with source attribution markers (<!-- source: ... -->)
• Metadata includes per-file extraction stats and total sources found
• Supports fallback to TIMEOUT environment variable for timeout configuration
• Tests download real PDFs from the web to validate actual extraction capability

https://claude.ai/code/session_01XyzPfEB3qopffFuRyouaTa

---

Summary by cubic

Adds a LiteParse plugin that uses the lit CLI to extract text and JSON from PDFs found in sibling plugin outputs, writing content.txt, content.json, and metadata.json. Also simplifies binary resolution in tests and fixes shebang checks and forum-dl install issues on Python 3.12.

• New Features

  • Snapshot hook scans pdf, responses, and staticfile for PDFs and runs lit parse in text and JSON modes.
  • Combines per-file outputs with source markers; writes content.json (single JSON or array) and metadata.json.
  • Config via LITEPARSE_ENABLED, LITEPARSE_BINARY, LITEPARSE_TIMEOUT, LITEPARSE_ARGS, LITEPARSE_ARGS_EXTRA; returns noresults when none found and failed if the binary can’t run.
  • Crawl hook emits a Binary for lit with npm install via @llamaindex/liteparse, supports env,npm, and honors custom paths.

• Refactors

  • Tests across defuddle, forumdl, gallerydl, mercury, opendataloader, readability, ytdlp, and liteparse now use Binary.load_or_install() (large boilerplate reduction).
  • LiteParse tests assert exact phrases from real sample PDFs to reduce false positives.
  • Shebang test skips non-script files (__init__, tests, utils) to avoid false failures.
  • forum-dl test installs with --no-deps, --prefer-binary, and pins chardet==5.2.0 to avoid cchardet build failures on Python 3.12.

^{Written for commit a36840f. Summary will update on new commits.}

---

[cubic-dev-ai]

No issues found across 6 files

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_plugin_executables_have_shebang.py">

<violation number="1" location="tests/test_plugin_executables_have_shebang.py:37">
P1: Custom agent: **Test quality checker**

This test now bails out early with `continue`, skipping validations for part of the discovered scripts; that violates the test-quality requirement forbidding skipped/bail-early test paths.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}