Repository navigation
v0.1.5
Added
Session::crawlreturnsCrawl, a polite stream ofCrawlPages that obeys
robots.txt and each page's robots rules by default, with five settings:
ignore_robots()turns them off;respect(Respect)adds AI-training
opt-outs (AiTraining);scope(Scope);limit(pages); and
host_limits(HostLimits). By default it sends one request at a time and one
a second per host where the session's ownHostLimitsleave it unset,
pauses a host on 429 or 503 and tries the page up to three times, sends
every request through both the session's ownHostLimitsand the crawl's
pace, keeps the site'sCrawl-delayout of the session's own requests,
waits out a robots.txt outage before it skips a site, reads sitemaps,
follows redirects as links, and drops error pages, files, block pages, and
copies by canonical URL or text, preferring the canonical page over an alias
found first.CrawlStatscounts eachSkipReason.Session::robotsandRobotsread robots.txt by RFC 9309: group matching,
with every group record (includingCrawl-delay,Content-Signal, and
Content-Usage) closing the run ofUser-agentlines, longest match with
*and$, percent-encoding, the 4xx and 5xx rules, up to five redirects,
and a 512 KiB limit.Robotsalso readsContent-Signaland the IETF draft
Content-UsageasPurpose,Crawl-delay, andSitemaplines;
Robots::allows_usetakes the URL.html::textreturns the visible text of a page, one block per line, with
<pre>kept as written and script, style, and head text left out. An
unclosed<head>ends at the first element that cannot be in it.html::Directivesreadsnoindex,nofollow,none, andnoaifrom
<meta name="robots">, a meta tag named for the agent, and
X-Robots-Tag.html::canonicalreturns the page's canonical link.- A Firefox 158 profile,
Browser::Firefox158, from captures of Firefox 158.0
on macOS, Linux, and Windows.
Changed
-
html::links,html::forms, and other readers built on the HTML scanner
now keep text after a comment or a bare<(<a>a < b</a>givesa < b),
skip</followed by a non-letter to the next>, and keep a namespace
prefix in tag names (fb:like), as the HTML standard does. -
A
HostLimits::hostoverride now applies to IPv6 literal hosts; before, it
never matched them. -
leyline-bssl-syscompiles BoringSSL with thecccrate from the source
lists BoringSSL ships, so a build needs only a C and C++ compiler and Git:
no CMake. On Windows, NASM is optional; without it the build links the
NASM objects that ship in the crate. The CMake-only variables
CMAKE_TOOLCHAIN_FILE,CMAKE_GENERATOR, and
LEYLINE_BSSL_COMPILER_EXTERNAL_TOOLCHAINno longer apply; useCC,
CXX,CFLAGS, andCXXFLAGS. The build rebuilds BoringSSL when its
source, the compiler, or the compiler flags change, and reuses the cached
library otherwise, so a restored build cache never links a stale BoringSSL.