Skip to content

v0.2.0 — identity verification & Apache support

Choose a tag to compare

View all tags
@kinti kinti released this 08 Sep 21:05
· 8 commits to main since this release
v0.2.0
64bd7ad

The headline feature: --verify catches spoofed bot identities.

crawlward --verify /var/log/caddy/*.log

A UA string is free text — anyone can claim to be GPTBot. Now crawlward
checks the source IPs of claimed-bot requests against the vendor's own
published IP prefix lists (OpenAI, Anthropic, Perplexity, Common Crawl and
Mistral publish them; IPv4 + IPv6 CIDR matching). The report now ends with a
verdict per bot: identity consistent, treat as spoofed, or mixed
(usually brand-new vendor ranges, not fraud).

Also in this release

  • Apache combined log support — auto-detected per line, alongside Caddy
    and nginx JSON; the client IP from combined logs feeds --verify
  • Peak requests/hour per bot — an aggressiveness signal you can act on
  • --since / --until date filters and --csv output
  • Registry v3: each bot's verification URL is now data (ranges field),
    with the annotated list in docs/crawlers.md
  • 27 tests; verification logic is tested with injected fetches, so CI stays
    offline

Full changelog: v0.1.0...v0.2.0

Assets 2
Loading