Skip to content

pulpie v0.0.1

Choose a tag to compare

View all tags
@chonknick chonknick released this 27 Jun 11:56
· 16 commits to main since this release
v0.0.1
0ffa41b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

First public release of pulpie — fast, CPU-friendly web content extraction from HTML using encoder models.

Install

pip install pulpie            # core
pip install pulpie[markdown]  # + markdown output

What it does

pulpie extracts the main content from raw HTML and returns clean HTML or markdown. It classifies HTML blocks with small encoder models (the Orange family), reaching autoregressive-extractor quality at a fraction of the cost.

from pulpie import Extractor

extractor = Extractor()             # downloads pulpie-orange-small (210M) on first use
result = extractor.extract(html)
print(result.markdown)              # clean markdown
print(result.html)                  # clean HTML
print(result.n_main, result.n_other)

MinerU-HTML parity

pulpie's simplify/reconstruct faithfully reproduce MinerU-HTML's simplify_html / map_to_main (Apache-2.0), the segmentation format the Orange models were distilled on. Verified two ways:

  • Offline byte-parity: 65/65 tests reproduce MinerU's exact output across 13 real-page fixtures.

  • End-to-end ROUGE-5 on full WebMainBench English (6,647 pages, orange-small):

    Simplification ROUGE-5 F1
    MinerU-HTML (reference) 0.8625
    pulpie (this release) 0.8625

    Byte-faithful end-to-end (Δ vs MinerU = +0.0000), a +0.14 improvement over the prior simplification.

Model zoo

Name HF repo Size Notes
Orange Small chonkie-ai/pulpie-orange-small 210M Best value; default
Orange Base chonkie-ai/pulpie-orange-base 610M Distilled
Orange Large chonkie-ai/pulpie-orange-large 2.1B Highest quality

Credits

Built on MinerU-HTML / Dripper (Ma et al., 2025) — their simplify_html preprocessing, block-annotation scheme, and the WebMainBench benchmark are foundational. Thank you for releasing your tools and data.

Assets 2
Loading