thepipe 2.0.0

thepipe 2.0 is out. 🎉 There are two major changes:

• thepipe is now taking an agent forward approach. What I mean by this, is that over time, the project and UX will become more optimized for AI agents, and less so for humans.
• Richer library, fewer dependencies, and fewer bugs: Document Layout Analysis (commonly known as DLA), provenance, gutting the web scraper, and many bug fixes.

The latest and greatest feature, is a very lightweight abstraction for Document Layout Analysis, generalized to work with any vision-language model via an OpenAI-compatible client. Until now, when you scraped a PDF with a VLM, chunk.images was the full page render. That was fine for feeding a multimodal model, but the images were large, they wasted tokens on content already in the scraped text, and even frontier LLMs were unable to interpret finely detailed technical figures within these page images.

Achieve this with

from openai import OpenAI
from thepipe.scraper import scrape_file

chunks = scrape_file(
    filepath="fleet_schedule.pdf",
    openai_client=OpenAI(),
    model="gpt-5.6-luna"
)

c = chunks[0]
print(c.text)                     # markdown, tables included
print(c.metadata["figures"])      # [{"description": "Pie chart of vehicle classes", "bbox": [0.08, 0.61, 0.47, 0.93]}, ...]
c.images[0].save("figure_0.png")  # the cropped pie chart

---

Removing the web scraping was a tough decision, and one that I thought about for about an entire year. The reality is that, while it presents as functional, and may even work for a while, modern web technology largely blocks naive scrapers - for real web scraping you are better off with something like BrowserUse or Firecrawl, which can interact with the page as well - capabilities which i am not interested in building out and maintaining. thepipe is an excellent choice for parsing PDFs and other files for downstream use with large language models, multimodal models, and vector databases. I want to lock in on just that.

Onto the next important change: Provenance. Chunks contain metadata on where they came from. Chunk now has a metadata dict, and every scraper fills it in. Chunkers preserve them (or concatenate/merge them when appropriate)

Some smaller, more boring changes to get out of the way:

• replaced moviepy with ffmpeg. Having the heavy (and rapidly evolving) moviepy dependency gone in the main installation route, makes for a much simpler install, given we only used one feature from moviepy.
• max_input_image_size (also --max-input-image-size on the CLI) caps the pixel size of images sent to the VLM. Useful if you're cost-sensitive; leave it if you're scraping dense scans with high accuracy requirements.
• chunk_by_length had a bug where the recursive call overwrote its own results. Fixed, with a test.
• .csv files detected as text/csv now actually route to the spreadsheet scraper.
• Directory scraping enforces the root boundary; symlinks can't walk out of it (#44, thanks @0xmrma).
• extract() now takes a Pydantic BaseModel schema and uses chat.completions.parse under the hood, so you get validated, typed objects back instead of loosely-shaped JSON. This is a breaking change from the 1.x dict-schema API.
• Fixed an OS command injection vulnerability in scrape_github (CWE-78, #39, thanks @0xmrma). Web scraping is gone now, but if you're pinned to 1.x you should be aware of it.
• Video and audio are now transcribed with Whisper as a whole file rather than in chopped-up segments, which removes the word-splitting artifacts at segment boundaries.
• chunk_by_keywords was setting the wrong path on its output chunks. Fixed.
• PDF pages and directory files are now scraped in parallel (max_workers on scrape_file/scrape_directory), and all VLM calls go through a process-wide BoundedSemaphore so fan-out can't blow past your provider's rate limit. Tune it with LLM_MAX_CONCURRENCY (default 16; 0 disables the cap).
• thepipe now requires Python 3.10+, and all dependencies are bumped to current versions.

The test coverage is more full, and each change has noticeably improved my experience working with my backend, which serves thepipe as a REST API. So, I'm happy with the state of things now, and I hope you all enjoy working with this release as much as I have.

pip install -U thepipe-api

Source is MIT licensed on GitHub. The hosted API at thepi.pe is already on 2.0.0