Skip to content

v0.1.5

Choose a tag to compare

@github-actions github-actions released this 13 Oct 22:39
· 60 commits to main since this release
f85af55

Added

  • Session::crawl returns Crawl, a polite stream of CrawlPages that obeys
    robots.txt and each page's robots rules by default, with five settings:
    ignore_robots() turns them off; respect(Respect) adds AI-training
    opt-outs (AiTraining); scope(Scope); limit(pages); and
    host_limits(HostLimits). By default it sends one request at a time and one
    a second per host where the session's own HostLimits leave it unset,
    pauses a host on 429 or 503 and tries the page up to three times, sends
    every request through both the session's own HostLimits and the crawl's
    pace, keeps the site's Crawl-delay out of the session's own requests,
    waits out a robots.txt outage before it skips a site, reads sitemaps,
    follows redirects as links, and drops error pages, files, block pages, and
    copies by canonical URL or text, preferring the canonical page over an alias
    found first. CrawlStats counts each SkipReason.
  • Session::robots and Robots read robots.txt by RFC 9309: group matching,
    with every group record (including Crawl-delay, Content-Signal, and
    Content-Usage) closing the run of User-agent lines, longest match with
    * and $, percent-encoding, the 4xx and 5xx rules, up to five redirects,
    and a 512 KiB limit. Robots also reads Content-Signal and the IETF draft
    Content-Usage as Purpose, Crawl-delay, and Sitemap lines;
    Robots::allows_use takes the URL.
  • html::text returns the visible text of a page, one block per line, with
    <pre> kept as written and script, style, and head text left out. An
    unclosed <head> ends at the first element that cannot be in it.
  • html::Directives reads noindex, nofollow, none, and noai from
    <meta name="robots">, a meta tag named for the agent, and
    X-Robots-Tag. html::canonical returns the page's canonical link.
  • A Firefox 158 profile, Browser::Firefox158, from captures of Firefox 158.0
    on macOS, Linux, and Windows.

Changed

  • html::links, html::forms, and other readers built on the HTML scanner
    now keep text after a comment or a bare < (<a>a < b</a> gives a < b),
    skip </ followed by a non-letter to the next >, and keep a namespace
    prefix in tag names (fb:like), as the HTML standard does.

  • A HostLimits::host override now applies to IPv6 literal hosts; before, it
    never matched them.

  • leyline-bssl-sys compiles BoringSSL with the cc crate from the source
    lists BoringSSL ships, so a build needs only a C and C++ compiler and Git:
    no CMake. On Windows, NASM is optional; without it the build links the
    NASM objects that ship in the crate. The CMake-only variables
    CMAKE_TOOLCHAIN_FILE, CMAKE_GENERATOR, and
    LEYLINE_BSSL_COMPILER_EXTERNAL_TOOLCHAIN no longer apply; use CC,
    CXX, CFLAGS, and CXXFLAGS. The build rebuilds BoringSSL when its
    source, the compiler, or the compiler flags change, and reuses the cached
    library otherwise, so a restored build cache never links a stale BoringSSL.