Skip to content

crawlcontract 0.3.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 19 Jul 03:58
v0.3.0
c4a6f4f

crawlcontract 0.3.0 is the first hardened release of the deterministic crawl,
canonical, and indexability contract checker.

Highlights

  • Builds one URL-state graph from HTML, HTTP headers, redirects, robots.txt,
    XML sitemaps, canonicals, internal links, and hreflang.
  • Reports unresolved sitemap, canonical, and internal-link targets, plus redirect
    cycles, with deterministic rule IDs and evidence.
  • Adds standards-aware XML sitemap parsing, multiple static sitemaps, preview-to-
    public origin mapping, and bounded concurrent live fetching.
  • Combines robots directives restrictively and supports wildcard, end-anchored,
    query-aware robots.txt matching with allow-wins ties.
  • Validates policies, snapshot schemas and statistics, output combinations, base
    URLs, and release-diff thresholds.

Downloads

  • x86_64-unknown-linux-gnu: Linux x86-64
  • x86_64-apple-darwin: macOS Intel
  • aarch64-apple-darwin: macOS Apple Silicon
  • x86_64-pc-windows-msvc: Windows x86-64

Each archive contains the executable, README, MIT license, and Apache 2.0
license. Verify downloads with SHA256SUMS:

# Linux
sha256sum --check SHA256SUMS

# macOS
shasum -a 256 --check SHA256SUMS

Install directly from the tagged source with Rust 1.85 or newer:

cargo install --git https://github.com/lame13/crawlcontract --tag v0.3.0

Snapshots currently use schema 1.0; compatibility is not promised before the
first stable release.

See the complete changelog.

Assets 7
Loading