v0.80.3

PageSift 0.80.3 Release Notes: Focused Crawling Checked on Real University Sites

Release Version: v0.80.3
Release Date: 2026-10-28
License: AGPL-3.0-only (Commercial licensing available)

---

Overview

0.80.2 rewrote the focused crawl (crawl --query) and could only check it on synthetic sites.
0.80.3 is the first version checked on real ones: the same 38 university home pages, crawled by
0.80.0 and by this release from the same machine, graded by the same script. The tester's report on
0.80.2 (Waseda, TUM, Manchester, the attachment harvest, the truncated arXiv paper) and the traces
of those real crawls drove every change below; each one is a general rule, and each was kept only
if it held up across the whole set.

On the 32 sites that both versions could reach (two runs of 0.80.3, averaged):

Measure (12 pages per site, adaptive)	0.80.0	0.80.3
Useful pages (international-student or admissions/study pages) ↑	113 of 382 (29.6%)	244 of 369 (66.1%)
International-student pages ↑	24 (6.3%)	143.5 (38.9%)
Median useful pages per site ↑	1.5	8
Sites with no useful page at all ↓	9	0.5

0.80.2 on the same sites (one run, 30 reachable): 195 of 320 useful (60.9%), 112 international.
The synthetic gate (focus_gate, 10 sites) stays at 77.3% (0.80.0: 7.3% on its 8 campuses).

Also in this release: the D-2 follow-up the tester found on the real arXiv paper (a cut file that
lost a whole page of text with no warning), and two crawl defaults that changed for query crawls
(whole-site scope, failed fetches no longer use the page budget).

How the real-site numbers were made. benchmarks/history/0.80.3/: sites.txt (38 home pages),
run.sh (crawl <home> --query "international students admission tuition scholarship visa" --adaptive --limit 12 --out, four sites at a time, robots.txt honoured), grade.py. The grading is
automatic, by URL and title: a page is international when its URL or title names international or
overseas students together with admission, fees, scholarships or visas (or its title names an
international student office); admissions/study when it names admission, applying, fees,
scholarships, degrees or study; anything under news, events, about, research, alumni, giving,
careers or a dated path is useless unless the URL says international. It is a coarse proxy for a
human reading the pages, and it is the same proxy for both versions. Machine: a Linux x86_64 cloud
container, 2026-10-28. Its outbound proxy refuses plain http:// requests, which affects a few links
on every version. Six of the 38 sites were unreachable or blocked for at least one version
(Cloudflare/Imperva challenges, connect timeouts) and are left out of the table. Two runs of the same
build differ by about one point.

---

Focused crawl

Each item is something the real-site traces showed, and the rule that fixes it everywhere.

• Query crawls cover the whole site. Waseda's home page is /top/; its scholarship and
  admissions pages are under /inst/. A path scope taken from the start URL hid them. With a
  query, the default scope is now the whole site (scope: domain); relevance decides where to go.
  An explicit scope (or CLI --scope) still applies; crawls without a query keep auto.
• Failed fetches do not use the page budget. Manchester lost a third of its budget to two
  student-support pages that timed out and a research page behind a Cloudflare challenge. In a
  query crawl, limit now counts pages delivered: errors, error statuses, challenges and
  robots-disallowed links are reported and refunded, up to limit refunds. The queue keeps enough
  links to use the refunds.
• Dead hosts are dropped, live ones are not. A host that times out twice, or answers with a
  challenge once, has its queued pages dropped — unless it has already delivered pages: one
  firewalled page on NTU's main host used to stop the whole crawl after three pages.
• The English site is found where it is declared. ETH's language switch is drawn by script;
  the only static pointer to /en.html is <link rel="alternate" hreflang="en">. For an English
  query, the start page's English alternate is queued as a second home page. English home pages
  ending in .html (/en.html) and on English subdomains (en.snu.ac.kr) are recognised; a
  department subdomain's /en/ is that department's English page, not the site's.
• Sections of the English site count as sections. /en/studies/ is a top-level section of the
  English site, so it gets the section floor, and English-home links are not decayed a level. Once
  the English home is found, other /…/en/ pages stop getting the English-entry floor (a staff
  portal's English page is not an entry point).
• Department home pages share one budget. University home pages link dozens of department
  subdomains, each a different host. All home pages (/, /en/, /index.html) now form one
  cluster, so the existing per-cluster cap and list discount apply to them together.
• Menus are not lists. The list discount (many links into one section on one page) now counts
  only links in the page's main text. Manchester's mega-menu links 58 times into /study/, which
  is where the answers are.
• Relevance thresholds from real pages. On real pages a target page's relevance has a median of
  0.16 (useless pages: 0.07). The adaptive stop's relevance threshold was 0.3, which treated most
  target pages as irrelevant and stopped Manchester after nine pages. Pages under 0.12 are now
  irrelevant (no evidence; a run of them after relevant pages ends the crawl); pages from 0.12 to
  0.3 are neutral; evidence still needs 0.3. Attachments use 0.12 too: TUM's international pages
  now yield 9 PDFs with --attachments (0.80.2: none; the tester ran the same start page).
• Hubs are judged by their text. A page whose text outside link labels is outnumbered by its
  main-content links is a hub; menus no longer make every page a hub.
• Furniture. Login, account, single sign-on, portal, intranet, webmail and LMS hosts and paths
  (my., auth., canvas, moodle…), shops, imprint/data-protection/cookie pages and
  /bin/-style technical paths are furniture; English queries demote paths in another language
  (/ja/, /de/).
• Redirects and duplicates. Same-site http:// links on an https:// page are fetched over
  https, and a same-host redirect from https to http is followed over https unless that would loop
  (the plain-HTTP hop only redirects back, and it fails outright on networks that allow only
  https). Two URLs that land on the same page deliver it once. internal_* and Eloqua elq*
  parameters are stripped as tracking parameters.

Tried and dropped. Weighting query terms by how often they appear in the start page's link
labels (so that "students", which is in half of them, counts less) made no difference beyond run
noise on the real sites (two runs each way) and lowered the synthetic gate; it is not in the
release.

Not solved. ETH, TU Delft and Osaka draw their navigation with JavaScript; a static fetch sees
department links and little else, and the focused crawl does not re-render pages. On TUM the German
pages whose labels contain "international" still win over the English site's sections.

Damaged PDFs: lost page content is named (D-2 follow-up)

The tester's file was the real arXiv 1706.03762 cut in half. arXiv adds its licence notice to page
1 with an incremental update at the end of the file, so the cut keeps the original page-1
dictionary but loses its content stream: the result had all 15 pages, 36,991 of 41,242 characters,
and only the generic pdf_repaired warning. Reproduced here on the downloaded paper cut at 50% to
80%.

A repaired file is now checked page by page: a page whose content stream is gone gets
content_missing: 1 of 15 pages lost their content streams in the damaged file; their text is missing or incomplete (pages 1), and pages whose font definitions are gone get fonts_missing
(the text is there, characters may be decoded wrongly). Intact files skip the check. The audit
log treats content_missing as suspicious and the CLI prints it.

Performance against 0.80.0

Instruction counts (callgrind) on the same machine, 0.80.0 tag against this release; wall-clock
times from different hours of a shared container are not comparable and are not used. Scripts:
benchmarks/history/0.80.1/bench.sh.

Step	0.80.0	0.80.3	Δ
perf_gate (HTML extraction, blog 100 KB + wiki 250 KB)	14,177,064,528	14,170,319,553	−0.05%
pdf_one paper.pdf ×3	2,194,527,360	2,204,204,971	+0.44%
office_bench ascii.csv ×3	4,296,566,432	4,318,536,098	+0.51%
office_bench sjis.csv ×3	1,586,495,037	1,485,456,080	−6.37%
office_bench gbk.csv ×3	1,459,031,365	1,472,941,891	+0.95%
office_bench utf8-cjk.csv ×3	1,176,671,975	1,182,429,852	+0.49%

The Shift_JIS and GBK files were decoded wrongly by 0.80.0 (mojibake; the 0.80.1 fix decodes them
correctly). That fix first cost +2.5% and +23% here, nearly all of it in the charset detector: it
fed every byte of a 64 KiB sample to some thirty candidate encodings. The detector is now fed in
4 KiB blocks and stops once its guess has been the same, with high confidence, three blocks in a
row after the first 16 KiB; a guess that is still changing is fed the whole sample as before.
The same detector decodes HTML pages without a declared charset. The output is byte-for-byte the
same on the benchmark files and on long GBK, Shift_JIS and Windows-1251 test texts.

crawl_gate could not run on 0.80.0 in this container (it needed IPv6, fixed in 0.80.1).

Upgrading

• Query crawls: default scope is the whole site; limit counts delivered pages and failed fetches
  are refunded; hosts that fail repeatedly before delivering anything are dropped.
• CLI crawl --scope defaults to auto, or domain with --query.
• New PDF warnings content_missing and fonts_missing.
• Fetching: same-host https→http redirects are followed over https.
• Charset detection stops early once its guess is stable (same results, less work).
• Library: pagesift_content::Metadata::alternates, Document.alternates (not serialised); the
  request field scope is now Option<CrawlScope>.

---

Join the community on Discord: https://discord.gg/esA3wMZVvA