Skip to content

v0.72.8

Choose a tag to compare

View all tags
@github-actions github-actions released this 26 Oct 08:07
· 390 commits to main since this release
v0.72.8
3307b02

PageSift 0.72.8 Release Notes: Truthful Signals, Complete Allow-Lists, Better Extraction

Release Version: v0.72.8
Release Date: 2026-10-26
License: AGPL-3.0-only (Commercial licensing available)

Overview

0.72.8 closes the fourth-round audit (charset, allow-lists, config, exit codes) and the
fifth-to-eighth-round field audit (extraction quality, metadata fidelity and honest signals).

  • HTTP charset headers that lie no longer produce silent mojibake.
  • --allow-cidr covers every command that fetches, not only scrape.
  • crawl and map no longer report a failed seed as a successful empty result.
  • pagesift loads the platform config file by default, and doctor shows which file it read.
  • Recipe pages keep their ingredients, GitHub code pages keep their code, arXiv pages keep
    their authors, RSS feeds become item lists, and thin or barrier pages say so.

Charset headers are verified against the body

Content-Type and <meta> charset declarations are treated as candidates instead of facts.
After decoding, Pagesift checks whether the declaration actually fits the bytes:

  • UTF-8 declared, GBK body — the declared decode produces replacement characters; Pagesift
    re-detects the body and decodes it as GBK.
  • Latin-1/Windows-1252 declared, UTF-8 body — the declared decode succeeds byte-wise but is
    mojibake; Pagesift recognizes the valid UTF-8 body and decodes it as UTF-8.
  • Small local damage — one or two bad bytes do not switch the whole page; the text is kept
    and a warning is emitted.

Every correction is visible in metadata.pagesift.warnings, for example:

decode: Content-Type declared UTF-8 but decoding produced 70 replacement characters; decoded as GBK instead
decode: Content-Type declared windows-1252 but the body is valid UTF-8; decoded as UTF-8 instead

Correctly declared UTF-8, GBK, Shift_JIS, Big5, EUC-KR and Windows-1252 pages are unchanged.
The UTF-8 fast path remains allocation-free. Encoding names in warnings use the WHATWG
canonical name, so a header saying iso-8859-1 is reported as windows-1252.

--allow-cidr everywhere

The same private-network allow-list is now available on:

scrape  crawl  map  download  session-fetch  collect  serve
  • Applies to HTTP fetching and browser rendering.
  • Repeatable: --allow-cidr 10.0.0.0/8 --allow-cidr 198.18.0.0/15.
  • Invalid values fail during argument parsing (rc=2) instead of being silently dropped:
error: invalid value 'not-a-cidr' for '--allow-cidr <CIDR>': invalid CIDR "not-a-cidr"

mcp has no URL command-line surface and continues to read fetch.allow_cidrs from the config
file.

Honest crawl and map failures

  • map previously returned exit code 0 and printed only the seed URL when the seed fetch failed.
    It now returns the original error (SSRF_BLOCKED, timeout, DNS failure, ...) with empty stdout.
  • crawl now exits with code 1 when it collected zero pages and at least one error. Partial
    success still exits 0, preserving the existing contract for large crawls.
  • crawl no longer sends a short static page through the browser a second time just because the
    text is short. Auto-rendering now requires client code or a substantial HTML document, so the
    seed page is fetched once in the common static case.

Default configuration discovery

Lookup order:

--config  >  PAGESIFT_CONFIG  >  platform config file (only if it exists)  >  built-in defaults

Platform paths:

Platform Path
Linux / macOS $XDG_CONFIG_HOME/pagesift/pagesift.toml, else $HOME/.config/pagesift/pagesift.toml
Windows $XDG_CONFIG_HOME/pagesift/pagesift.toml, else %APPDATA%\pagesift\pagesift.toml, else %USERPROFILE%\.config\pagesift\pagesift.toml

XDG_CONFIG_HOME must be an absolute path. Empty and relative values are treated as unset, so a
broken environment cannot make Pagesift read ./.config/... from the current directory.

PAGESIFT_CONFIG= (empty value) disables all config files — useful for Docker, CI and tests:

PAGESIFT_CONFIG= pagesift doctor   # built-in defaults only

doctor now reports the real source:

ok   config: /home/me/.config/pagesift/pagesift.toml
ok   config: defaults (no config file; create /home/me/.config/pagesift/pagesift.toml to add one)
ok   config: defaults (config files disabled by PAGESIFT_CONFIG=)

Upgrade note

If a pagesift.toml already exists in the platform config directory, commands that previously
used built-in defaults will now load it. Run pagesift doctor after upgrading to confirm the
source. Use --config, PAGESIFT_CONFIG or PAGESIFT_CONFIG= to override discovery explicitly.

Thanks to @w21180239 for proposing default config discovery in PR #3.

Extraction quality and honest signals

Recipe and content asides are kept

Body selection no longer throws away an <aside> that belongs to the article. Content asides
inside <main>/<article> are kept when they are not named like a sidebar and do not consist
mostly of links. Recipe ingredient lists are the common case: the King Arthur chocolate-chip
cookie page now returns the title, all ingredients and the full method instead of a 468-character
introduction.

A preview block (quickview, teaser, excerpt) is no longer mistaken for the article itself.

GitHub code pages keep the code

GitHub's file view renders line numbers and code as two sibling columns. Pagesift now:

  • drops the react-line-numbers column;
  • merges the react-file-line rows into one fenced code block, in order;
  • skips visually-hidden headings such as "Repository navigation".

A 411-line Rust file used to produce 411 standalone numeric lines and an unfenced code blob; it
now produces one code fence and no numeric noise.

Authors and citation metadata

  • Author blocks (authors, author-list) survive link-density pruning, so arXiv abstract pages
    keep the author line in the body.
  • citation_author, citation_title, citation_doi, citation_journal_title,
    citation_conference_title, citation_pdf_url and citation_arxiv_id are read from <meta>.
    Multiple citation_author values are joined with commas into metadata.author.
  • metadata.pagesift.jsonLd now carries every parsed JSON-LD block unchanged, so downstream
    pipelines can consume Article, Recipe, Product and VideoObject data directly.

Barriers and interstitials

Generic German, French and Spanish block pages are classified as ACCESS_BARRIER instead of
being delivered as content (the ZEIT 403 page is the reference case). Age-verification pages are
reported with a new interstitial barrier kind. Known vendor rules are unchanged; this only
fills gaps for sites without a vendor signature.

Thin extraction has a signal

When a 2xx page has a substantial HTML document but less than 500 non-whitespace characters of
extracted text, Pagesift adds:

thin_content: extracted 468 characters from 331 KB of HTML; the page may be a JavaScript shell or the main content was not selected

This distinguishes "the page is genuinely small" from "the content was dropped". empty_content
still covers literal zero.

inspect recognizes more secrets

New secret_indicator rules cover:

  • OpenAI-style keys (sk-…, sk-proj-…, sk-ant-…);
  • JWTs (three base64url segments);
  • api_key = "…" / api-key: "…" assignments.

GitHub tokens, AWS keys, Stripe keys, Slack tokens and private keys were already covered. Values
are still never echoed; findings only contain rule names and byte offsets.

Images keep their title

Markdown images now include the title attribute, which is where some sites put the caption:
![Standards](https://imgs.xkcd.com/comics/standards.png "Fortunately, the charging one …").
Alt text behaviour is unchanged.

RSS, Atom and RDF feeds are parsed, not scraped

Feeds are detected by structure and converted to a Markdown item list:

# xkcd.com

xkcd.com: A webcomic of romance and math humor.

- [Forms of Water](https://xkcd.com/3315/) — Fri, 23 Oct 2026 04:00:00 -0000

Escaped HTML in descriptions is stripped and entities are restored. --only-links returns the
item links. Sitemaps are still handled by the crawl/map path, not by this parser.

Video pages use their embedded metadata

When a video page has no readable body text, the page's JSON-LD VideoObject supplies the
title, description, author and upload date. The result carries an honest warning:

video_metadata: no readable body text; used the page's embedded video metadata

YouTube watch pages now return the video title and description on the HTTP path instead of an
empty document and a bare empty_content warning.

Smaller fixes

  • pagesift diagnose prints the actual version instead of the hard-coded 0.60.

Verification

At the time of writing:

cargo test --workspace                     784 passed, 0 failed

Field probes on the 0.72.8 binary:

Case Before After
King Arthur recipe 468 chars, no ingredients 2,693 chars, 6 teaspoon, title + ingredients + method
GitHub core/src/lib.rs 1,589 lines, 411 numeric lines, no fence 425 lines, 0 numeric lines, one fenced block
arXiv 1706.03762 Vaswani absent, author: null author line in body, 8 authors in metadata
ZEIT 403 block page success + http_error_status ACCESS_BARRIER (generic.denied_de)
xkcd RSS escaped soup clean item list
xkcd #927 image no title title preserved
YouTube watch page 0 chars + empty_content title + description + video_metadata
inspect fixture AWS only AWS, OpenAI key, JWT, api_key assignment
diagnose PageSift 0.60 PageSift 0.72.8
crawl seed (auto engine) 2 requests 1 request

Correctly declared legacy charsets, 4.49 MB pages, PDFs, tables, browser rendering and normal
crawls are unchanged.

Known boundaries

  • A certificate used as both CA and leaf (CaUsedAsEndEntity) is classified as other TLS error; the common expired / hostname-mismatch / untrusted-root cases have distinct codes.
  • map --sitemap only still returns only the seed URL if the sitemap itself fails; that mode
    intentionally skips the seed scrape.
  • mcp remains config-file only.
  • Framework-rendered visible text and server-side i18n are unchanged; the video fallback relies
    on page-declared JSON-LD and does not execute JavaScript.
  • A page that is thin because it really is a short note can still receive thin_content; it is
    a warning, not an error, and never changes the extracted text.

Docker

docker pull ghcr.io/yolow92/pagesift:0.72.8

Contributors

w21180239
Assets 10
Loading