v0.9.3

Fixed

• RAG Sitemap URL Ordering - Global sort by lastmod before applying max_pages limit
  • Previously, URLs were sorted within each sub-sitemap but not globally
  • When max_pages limit was applied, older content could be included instead of newest
  • Now all URLs are sorted by lastmod (newest first) before limiting
  • Ensures max_pages always returns the most recently modified content

Added

• HTTP Response Status Summary - New table showing crawl response distribution
  • Displays status code, description, count, and percentage for each response type
  • Tracks robots.txt blocks, network errors, and all HTTP status codes
  • Printed to stderr after fetch phase completes
  • Helps diagnose crawl issues (404s, rate limiting, etc.)

Changed

• Crawler fetch_page() Return Type - Now returns (url, html, status_code) tuple
  • Previously returned (url, html) or None
  • Now returns status code even on HTTP errors for tracking
  • Returns None only when blocked by robots.txt (no request made)
  • Network errors use status code 0

Full Changelog: v0.9.2...v0.9.3